Estimating Sensitivity and Specificity in an ROC Experiment

摘要

Observer studies and clinical studies are used to evaluate imaging technologies and to compare two different technologies. The area under the receiver operating characteristic (ROC) curve is often used as the endpoint. However, in clinical practice, radiologists will operate at a single point on the ROC curve and this will define the sensitivity and specificity of the radiologist using the technology. In an ROC study, sensitivity and specificity are often estimated based on the ratings that the radiologist gave each case, which are needed to generate the ROC curve. Unfortunately, because of intra-reader and inter-reader variability, estimates of sensitivity and specificity based on these ratings can lack accuracy and precision. We demonstrate using an observer study of computer-aided diagnosis in diagnostic mammography that estimates of sensitivity of a given radiologist can be either over or under estimated by as much as 100%. Further, the statistical power of the experiment is substantially reduced because of increased variability between readers. By simply asking the observer explicitly their recommendation (e.g., biopsy on no biopsy), sensitivity and specificity can be measured directly and the power of the study can be maximized.