AN ATTRIBUTE AGREEMENT ANALYSIS METHOD FOR HFACS INTER-RATER RELIABILITY ASSESSMENT

摘要

Inter-rater reliability can be regarded as the degree of agreement among raters on a given item or a circumstance. Multiple approaches have been taken to estimate and improve inter-rater reliability of the United States Department of Defense Human Factors Analysis and Classification System used by trained accident investigators. In this study, three trained instructor pilots used the DoD-HFACS to classify 347 U.S. Air Force Accident Investigation Board (AIB) Class-A reports between the years of 2000 and 2013. The overall method consisted of four steps: (1) train on HFACS definitions, (2) verify rating reliability, (3) rate HFACS reports, and (4) random sample to validate ratings reliability. Attribute agreement analysis was used as the method to assess inter-rater reliability. In the final training verification round, within appraiser agreement ranged 85.28% to 93.25%, each appraiser versus the standard ranged 77.91% to 82.82%, between appraisers 72.39%, and all appraisers versus the standard was 67.48%. Corresponding agreement for the random sample of HFACS rated summaries were within appraiser 78.89% to 92.78% and between appraisers 53.33%, which is consistent with prior studies. This pilot study indicates that the train-verify-rate-validate attribute agreement analysis approach has the potential to aid in improving HFACS ratings reliability and contributing to accurately capturing human factors contributions to aircraft mishaps. Additional full-scale studies will be required to verify and fully develop the proposed methodology.