Chapter 5 evaluating web retrieval effectiveness

Purpose — The overall quality of an information retrieval system depends on many different aspects of the system and its users' information seeking behaviour, such as the speed of the system, the user interface, the query language and the features provided by the engine. One of the most important aspects is the effectiveness of the retrieval system, i.e. its ability to retrieve items that are relevant to the information need of an end user. This chapter focuses on methods for measuring effectiveness, in particular focusing on recent work that more directly models the utility of an engine to its users. Methodology/approach — We discuss traditional approaches to effectiveness evaluation based on test collections, then transition to approaches based on test collections along with explicit models of user interaction with search results. We contrast this with approaches for which the user is ‘in the loop’, such as user studies and …