The Evaluation of Medical Expert Systems: Experience with the AI/RHEUM Knowledge-Based Consultant System in Rheumatology

摘要

Several issues relating to the evaluation of AI/RHEUM, an artificial intelligence consultant system in rheumatology, are discussed. The clinical knowledge of the consultant system is represented in formal disease criteria in a structured knowledge base. AI/RHEUM has during its development been retrospectively tested against 384 clinical cases selected because they represented diseases in its knowledge base. The system has been challenged with 74 unselected clinical cases admitted to the Arthritis Unit at the University of Missouri-Columbia School of Medicine, and with 59 cases submitted by colleagues at Keio University in Japan. In all three series, the agreement of the AI/RHEUM system with the “gold standard” diagnosis of a consensus of rheumatologist clinicians has exceeded 90%.