User Perspectives on Relevance Criteria: A Comparison among Relevant, Partially Relevant, and Not-Relevant Judgments

• Main Authors: Maglaughlin, Kelly L., Sonnenwald, Diane H.
• Language: en
• Published: Wiley Periodicals, Inc. 2002
• Subjects: Information Retrieval; Information Seeking Behaviors
• Online Access: http://hdl.handle.net/10150/105087

This study investigates the use of criteria to assess relevant, partially relevant and not relevant documents. Each study participant identified passages within 20 document representations that were used in making relevance judgments, judged each document representation as a whole to be relevant, partially relevant or not relevant to their information need, and explained their decisions in an interview. Analysis revealed 29 criteria, discussed positively and negatively, used by the participants when selecting passages that contributed or detracted from a document's relevance. These criteria can be grouped into 6 categories: author, abstract, content, full text, journal or publisher and personal. Results indicate that multiple criteria are used when making relevant, partially relevant and not relevant judgments. Additionally, most criteria can have both a positive or negative contribution to the relevance of a document. The criteria most frequently mentioned by study participants in this study was content, followed by criteria concerning the full text document. These findings may have implications for relevance feedback in information retrieval systems, suggesting that users give relevance feedback using multiple criteria and indicate positive and negative criteria contributions. Systems designers may want to focus on supporting content criteria followed by full text criteria as this may provide the greatest cost benefit.