Using incomplete and imprecise localization data on images to improve estimates of detection accuracy

摘要

We tested new analytic procedures for combining an observer's image-ratings of lesion-likelihood with localization reports that are incomplete (unavailable on images rated as 'normal') and/or imprecise (possibly scored as 'correct' by chance), and for fitting a constrained ROC formulation to the rating data alone. Eight radiologist readers in a previous study had rated the likelihood of nodular lesions on each of 250 chest-film cases (39 with subtle nodules, 36 with 'typical' nodules and 175 normal cases) that were presented in two display modes (original films or on video workstation). Ratings in the four positive categories (2 to 5) were accompanied by reports that grossly localized the suspected nodules into one of 7 film- regions (upper, middle or lower portions of left or right lung field, or retrocardiac), but there was no localization for the cases rated as 'normal' (category 1). In each of 29 sets of data, we estimated the area below the ROC curve (A$-z$/) and its standard error using three different fits: (1) the usual ROC formulation, (2) the constrained ROC formulation and (3) the new procedure that included incomplete and imprecise localization data (I&I). Estimates of A$-z$/ from the usual and constrained ROC fits were quite similar unless the standard ROC exhibited an upward 'hook,' but standard errors of A$-z$/ were always the same or smaller for the constrained ROC fit. The I&I fit that included localization data often estimated A$-z$/ to be either larger or smaller than the usual or constrained ROC fits that considered only the rating data, but its A$-z$/ had substantially smaller standard errors in 28 of the 29 sets of observer data.