Evaluating Success in Search Systems

摘要

Success in improving information retrieval (IR) systems has been an illusive concept forrnmuch of the past half century. From a system-centered perspective, progress inrninformation retrieval has been equated with the improvements in recall and precisionrnbrought about by developments in retrieval algorithms within the IR community. Fromrnthis perspective, progress is mechanistically associated with a better query-to-retrieved-resultsrnmatch. Much of the work undertaken for the TREC Conferences has contributedrnto improvements in IR from this point of view. Useful as it may be, this mechanisticrnapproach to retrieval as a matching process represents only one component of thernentire information retrieval process, and certainly the easiest component to evaluate.rnFrom a human-centered perspective,, success is associated with relevance, a conceptrnso nebulous that a significant number of research studies have tried to pin down itsrnmeaning and identify the factors which contribute to it. . So much paper and so muchrnverbosity about something that seems, at least intuitively and initially, obvious. Nearlyrn10 years ago Saracevic (1996) defined five levels of relevance which are well cited. Yet,rnwe are no closer to an operational definition of user relevance, no closer tornoperationalizing its indicators, and thus no closer to precisely determining how tornmeasure the success of the many IR systems developed for the enterprise or for thernpublic or commercial uses.rnFrom those initial and seminal studies conducted by Saracevic and colleagues (1988)rnin the 1980's to those conducted both pre-Internet and post-Web emergence (c.f.,rnJenson (2005), Toms et al. (2005)), the process for evaluating, analyzing and reportingrnresults from IR system evaluations has remained much the same, namely: have a fewrnparticipants usually students, perform a few searches in a laboratory setting, labourrnover recordings and surveys, and months or years later report the findings, and yes,rnmeasure something! Certainly, our evaluation methodologies and metrics are due for arnreview, discussion, debate and update.rnThis panel will begin by exploring the historical context for IR evaluation, and then focusrnon current and emerging methodologies for collecting, analyzing and reporting findings