Interrater Agreement Reconsidered: An Alternative to the r_(wg) Indices

摘要

For continuous constructs, the most frequently used index of interrater agreement (r_(wg(1))) can be problematic. Typically, r_(wg(1)) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform null r_(wg(1)) index and discuss alternative methods for measuring interrater agreement. Anew interrater agreement statistic, a_(wg(1)), is proposed. The authors derive the a_(wg(1)) statistic and demonstrate that a_(wg(1)) is an analogue to Cohen's kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniform r_(wg(1)) and a_(wg(1)), and issues such as minimum sample size and practical significance levels are discussed. The authors close with recommendations regarding the use of r_(wg(1))/r_(wg(J)) when a uniform null is assumed, r_(wg(1))/r_(wg(J)) indices that do not assume a uniform null, a_(wg(1))/ a_(wg(J)) indices, and generalizability estimates of interrater agreement.