Low-Cost Evaluation in Information Retrieval

SIGIR 2010 Low-Cost Evaluation in Information Retrieval Page 1 SIGIR 2010 Geneva, Switzerland July 19-23, 2010 Low-Cost Evaluation in Information Retrieval Tutorial of the 33 rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval Proposed by Ben Carterette Evangelos Kanoulas Emine Yilmaz Page 2 Low-Cost Evaluation in IR Ben Carterette Evangelos Kanoulas Emine Yilmaz Evaluating Retrieval Systems Results Judges Search Engines (IR Systems) Page 3 Test Collections • Document Corpus • Topics • Relevance Judgments Topic 1 Topic 2 Topic N Evaluating Retrieval Systems • Evaluation measures – User-oriented • Underlying user model to simulate user’s interaction • Precision@k, nDCG@k, MRR, RBP, ERR [Chapelle et al CIKM09], … – System-oriented • Also often based on a user model [Robertson SIGIR08] • Capture the overall effectiveness of …