A new scientific accuracy measure for performance evaluation of human-computer diagnostic systems

摘要

This paper first presents a new scientific accuracy measure (denoted by G) for assessing/evaluating the performance of computer medical diagnostic (CMD) systems by incorporating the true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) of human and computer's diagnoses with respect to each other. Based on G, a formula for computing a multi-parameter sensitivity vector S (G), with the assumption that the system parameter percentage variations are small, is then proposed. For a given set of parameter percentage errors, from the expression of S (G), we can compute the error bound of G and assess the reliability of the system with human and/or computer errors being taken into consideration. It has been demonstrated that the new measure G is capable of providing consistent performance evaluation of a CMD system in general. Based on the value of G, a CMD system can be classified as having "good", "fair", or "poor" performance. Even though the proposed basic accuracy measure and its sensitivity study are derived based on the diagnosis using two diagnostic categories (positive and negative) compared by two observers (a human expert and a computer system), however, its methodology can be extended to CMD systems with multiple diagnostic categories and observers. The formulas for measuring the performance of such systems are discussed and present.