Unit 6 Reading Notes

IIR 8

This chapter discusses the methods to measure the efficiency of the IR.

·          Three things required in a test collection:

¨         A document collection

¨         Queries

¨         A set of relevance judgment

·          Relevance is assessed relative to an information need, not a query. A document is relevant if it addresses the stated information need, not because it just happens to contain all the words in the query. The standard approach to information retrieval system evaluation revolves around the notion of relevant and nonrelevant documents.

·          The two most frequent and basic measures for information retrieval effectiveness are precision and recall. These two quantities trade off against one another.

·          The advantage of system evaluation, as enabled by the standard model of relevant and nonrelevant documents, is that we have a fixed setting in which we can vary IR systems and system parameters to carry out comparative experiments.

·          There are some further methods to measure how satisfied is each user with the results the system gives for each information need that they pose.

Reading: What's the value of TREC: is there a gap to jump or a chasm to bridge?

·          This article discusses the importance of change the concentration from generalization to particularization.

·          In the controlled, laboratory experiment case for core retrieval, the environment consists of documents (D), requests (Q), and relevance assessments (R). D * Q * R tends to be taken as encapsulating the entire information-seeking task (T), rather than just the experimental task (X).

·          TREC needs to engage, more positively and fully, with context and the nature of the whole setup information-seeking task T rather than just the experimental task X.

Reading: Cumulated gain-based evaluation of IR techniques

·          This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position.

·          There are three novel measures are introduces: Direct Cumulated Gain, Discounted Cumulated Gain and Normalized (D)CG Measure.

·          In Direct Cumulated Gain evaluation, the relevance score of each document is somehow used as a gained value measure for its ranked position in the result and the gain is summed progressively from ranked position 1 to n.

·          Discounted Cumulated Gain measure stated that the greater the ranked position of a relevant document, the less valuable it is for the user, because the less likely it is that the user will ever examine the document

·          Normalized (D)CG Measure computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield.

·          The use of the proposed measures are demonstrated in a case study testing runs from the TREC-7 ad hoc track with binary and non-binary relevance judgments.