Risk-Sensitive Evaluation and Learning to Rankrnusing Multiple Baselines

摘要

A robust retrieval system ensures that user experience isrnnot damaged by the presence of poorly-performing queries.rnSuch robustness can be measured by risk-sensitive evaluationrnmeasures, which assess the extent to which a systemrnperforms worse than a given baseline system. However, usingrna particular, single system as the baseline suu000bers fromrnthe fact that retrieval performance highly varies among IRrnsystems across topics. Thus, a single system would in generalrnfail in providing enough information about the real baselinernperformance for every topic under consideration, andrnhence it would in general fail in measuring the real risk associatedrnwith any given system. Based upon the Chi-squaredrnstatistic, we propose a new measure ZRisk that exhibits morernpromise since it takes into account multiple baselines whenrnmeasuring risk, and a derivative measure called GeoRisk,rnwhich enhances ZRisk by also taking into account the overallrnmagnitude of eu000bectiveness. This paper demonstratesrnthe benefts of ZRisk and GeoRisk upon TREC data, andrnhow to exploit GeoRisk for risk-sensitive learning to rank,rnthereby making use of multiple baselines within the learningrnobjective function to obtain eu000bective yet risk-averse/robustrnranking systems. Experiments using 10,000 topics from thernMSLR learning to rank dataset demonstrate the eu000ecacy ofrnthe proposed Chi-square statistic-based objective function.