Abstract: We review components-of-variance models for theuncertainty in estimates of the area under the ROCcurve, A$-z$/, for the case of classical discriminantswhere we wish the uncertainty to generalize to apopulation of training cases as well as to a populationof testing cases. A key observation from our previouswork facilitates the use of resampling strategies toanalyze a finite data set and classifier in terms ofthe components-of-variance models. In particular, wedemonstrate the use of the statistical bootstrap incombination with a four-term variance model to solvefor the contributions of the uncertainty in A$-z$/ thatresult from a given finite training sample, a givenfinite test sample, and their interaction. At the sametime one obtains an expression from which one canpredict the change in uncertainty in estimates ofA$-z$/ that would result from a given change in thenumber of training samples and change in the number oftest samples. This expression provides a quantitativedesign tool for estimating the size that would berequired in a larger pivotal study from the results ofa smaller pilot study for the purpose of achieving adesired precision in A$-z$/ and the desiredgeneralizability.!20
展开▼
