The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929).

摘要

Discrimination and calibration have been two major components in the evaluation of model performance. Discrimination measures the ability of a model to separate patients with different outcomes. In the case of a binary outcome, good discrimination indicates adequate distinction in the distributions of predicted values, based on the model, between the two classes, defined by the binary outcome. Calibration quantifies how closely the predicted values agree with the observed outcomes. Often, good calibration tends to correspond to good discrimination and vice versa; however, there are exceptions in which a model is strong in one measure and weak in the other [1]. For instance, a model that predicts all positive outcomes to occur with probability 0.51 and all negative outcomes to occur with probability 0.49 has perfect discrimination but bad calibration. If prediction is the goal, it is generally recommended that we should choose the model with good discrimination over the one with good calibration. If a predictive model has poor discrimination, no adjustment or calibration can correct the model. On the other hand, if discrimination is good, but calibration is poor, the model can be re-calibrated without sacrificing the discrimination.