An Analysis of Systematic Judging Errors in Information Retrieval

摘要

Test collections are powerful mechanisms for evaluation and optimization of information retrieval systems. There is reported evidence that experiment outcomes can be affected by changes in the judge population or in judging guidelines. We examine such effects in a web search setting, comparing the judgments of four groups of judges: NIST Web Track judges, untrained crowd workers and two groups of trained judges of a commercial search engine. Our goal is to identify systematic judging errors by comparing the labels contributed by the different groups. In particular, we focus on detecting systematic differences in judging depending on specific characteristics of the queries and URLs. For example, we ask whether a given population of judges, working under a given set of judging guidelines, are more likely to overrate Wikipedia pages than another group judging under the same instructions. Our approach is to identify judging errors with respect to a consensus set. a judged gold set and a set of user clicks. We further demonstrate how such biases can affect the training of retrieval systems.