Authentic assessments: a method to detect anomalies in assessment response patterns via neural network

摘要

Surveys, questionnaires, and assessments are the only approved means for measuring outcomes, yet there is a paucity of evidence that these tools can perform well as outcomes measures. A major barrier to utilizing these tools as measures of improvement in care is an inability to recognize when the responses are incomplete, inaccurate, or insincere. In this study, we propose a hybrid unsupervised-supervised approach to identify inauthentic assessments via anomalous response patterns. The study utilized 60-question behavioral assessment from a clinical population served by a county-operated public children's behavioral health services (n = 42,945). A novel hybrid unsupervised-supervised approach was developed to identify inauthentic assessment records from highly dimensional assessment data without the need for a priori record labels, which would otherwise require countless hours of record review by highly trained clinical staff. A neural network trained with 75% of the labeled data recognized anomalous response patterns in the test data with 84.5% sensitivity and 97.3% specificity. The model identified 26% of records as potentially inauthentic based on anomalous response patterns. For mental health and behavioral health, this novel method could flag a relatively small proportion of the records for clinical review while allowing records with probable authenticity to bypass review processes, saving time and creating efficiency in care. This method is also a potential tool for improving service quality through the identification of circumstances that have been overlooked but could otherwise be addressed during care. For decades, assessment data has been collected by policy and ignored by practice due to limitations of utility. With the right approaches, we could uncover the patterns of what works for whom that are currently hidden in our existing assessments.