On the effect of improperness of binormal ROC curves for estimating full area under the curve

摘要

The “binormal” model is commonly used for evaluating diagnostic performance with smooth Receiver Operating Characteristic (ROC) curves. However, one of the artifacts of the binormal model is the non-concave (improper) shape of the ROC curves, which is sometimes evident as a visible and practically unreasonable “hook”. The artificial hook can often be triggered, when the true ROC curve is concave but has high initial slope. In these scenarios it is natural to be concerned with the bias in the estimates of global summary measures, e.g., in the commonly used area under the ROC curve (AUC). The objective of this study is to evaluate the magnitude of said bias as a function of improperness of the fitted binormal ROC curves. The public health relevance of this work stems from the importance of the ROC methodology for various stages of development and regulatory approval of medical diagnostic systems. This work investigates whether the AUC for a visually improper binormal ROC curve provides an acceptable estimate of the full area under an actually concave ROC curve. For this purpose a simulation study was conducted based on a wide range of scenarios described by the concave bigamma ROC curves. The binormal ROC curves were fitted using the least squares approach. Based on the “mean-to-sigma ratio” criteria proposed in the literature, the fitted binormal curves were divided into the three groups based on the magnitude of their visual improperness. In order to assess bias in these groups of curves the binormal estimates of AUCs were compared with the empirical AUCs (which are unbiased for continuous data). Our results indicate that for continuous data the bias of the binormal estimate of AUC was small regardless of the magnitude of improperness of the fitted curve. Thus, if one is interested only in estimating AUC using continuous diagnostic data, the improper shape of the binormal curve can often be unimportant. We used data from a multireader study with 36 ROC curves, to illustrate the differences between the bigamma and binormal AUC estimates for different shapes of binormal ROC curves fitted to pseudo-continuous data from actual diagnostic accuracy studies.