Efficiently calculating ROC curves, AUC, and uncertainty from 2AFC studies with finite samples

摘要

Two-alternative forced-choice (2AFC) reader studies are useful for evaluating medical imaging devices because humans can rapidly make direct comparisons with high precision leading to low variability in study results. We propose a method for estimating the receiver operating characteristic (ROC) curve, reader performance (area under the ROC curve. AUC), and uncertainty on AUC from a series of 2AFC trials on a finite data set. Our method greatly reduces the number of 2AFC comparisons required by using an algorithm created for sorting, in this case Merge Sort. By altering the algorithm to work in discrete layers, we can make unbiased estimates as the study proceeds. Because the merging is pre-planned with a tree structure, we can use a Hanley-McNeil approximation to predict the reduction in variance in AUC from performing more 2AFC comparisons. The algorithm is also altered to increase the amount of time between the reader seeing the same image repeatedly thus decreasing potential learning. We compare our method with that of Massanes and Brankov (2016).