Effect of intent descriptions on retrieval evaluation

Test collections play an important role in adhoc and diversity retrieval evaluation. Constructing a test collection for adhoc evaluation involves (1) selecting a set of queries to be judged, (2) selecting an intent (topic) description for that query, and (3) obtaining relevance judgments with respect to the specific intent description for that particular query. Recent work showed that the selection of intents play an important role in the relative performance of retrieval systems for diversity evaluation. However, no previous work has analysed how the choice of a specific intent description may affect adhoc evaluation. We show that intent descriptions have a significant impact in adhoc evaluation and that special care should be given as to how the intent descriptions are selected. We further show that it is better to have very general intent descriptions or no intent descriptions at all when constructing test collections for adhoc …