Semi-parametric Estimation of the Area Under the Precision-Recall Curve

摘要

Precision and recall are two common metrics used in the evaluation of information retrieval systems. By changing the number of retrieved documents, one can obtain a precision-recall curve. The area under the precision-recall curve (AUCPR) has been suggested as a performance measure for information retrieval systems, in a manner similar to the use of the area under the receiver operating characteristic curve in binary classification. Limited work has been performed in the literature to investigate the bias and variance of AUCPR estimators. The goal of our study was to investigate the bias and variability of a semi-parametric binormal method for estimating the AUCPR, and to compare it to other techniques, such as average precision (AP) and lower trapezoid (LT) approximation. We show how AUCPR can be obtained given the binormal model parameters, and how its variance can be estimated using the delta method. We performed simulation experiments with normal and non-normal data, and investigated the effect of sample size and prevalence. Our results indicated that the semi-parametric binormal approach provided AUCPR estimates with small bias and confidence intervals with acceptable coverage when the sample size was large, and the performance of the binormal model was comparable to or better than alternative methods evaluated in this study when the sample size was small. We conclude that the semi-parametric binormal model can be used to accurately estimate the AUCPR, and that the confidence intervals derived from the model can be at least as accurate as from other alternatives, even for non-normal decision variable distributions.