On the Equivalence of Multirater Kappas Based on 2-Agreement and 3-Agreement with Binary Scores

摘要

Cohen’s kappa is a popular descriptive statistic for summarizing agreement between the classifications of two raters on a nominal scale. Withm≥3raters there are several views in the literature on how to define agreement. The concept ofg-agreement(g∈{2,3,…,m})refers to the situation in which it is decided that there is agreement ifgout ofmraters assign an object to the same category. Givenm≥2raters we can formulatem−1multirater kappas, one based on 2-agreement, one based on 3-agreement, and so on, and one based onm-agreement. It is shown that if the scale consists of only two categories the multi-rater kappas based on 2-agreement and 3-agreement are identical.