INTER-RATER RELIABILITY OF EXPERT-BASED PERFORMANCE MEASURES

摘要

Human performance measures used in representative settings often rely heavily on process experts who possess the domain knowledge to evaluate operator actions and system slates. This reliance on process experts may lead to inter-rater reliability measurement concerns. We report an empirical study that assessed the inter-rater reliability of two expert-based performance measures employed in a full scope nuclear power plant simulator experiment. The results indicated low inter-rater reliability for measures employing generic cognitive performance anchors that were designed to account for dynamics introduced by operators. In contrast, results indicated high inter-rater reliability for measures employing on pre-defined, specific behavioral anchors that were inflexible to unintended changes to the progression of the scenario. Future research is necessary to develop human performance measures that are both reliable across experts and flexible to dynamics introduced by operators.