BARS and Those Mysterious, Missing Middle Anchors

摘要

Purpose: A contributing reason for the common problem of missing middle anchors on behaviorally anchored rating scales (BARS) is the standard deviation (SD) criterion used in scaling phase. An alternative BARS scaling process is proposed based on the a_(wg(1)) index of interrater agreement. Design/Methodology/Approach: Algebraic principles are used to explicate that the SD criterion is analogous to using a r_(wg(1)) interrater agreement statistic with the assumption of a uniform null distribution, and this reliance on r_(wg(1)) decreases the likelihood of anchoring behaviors in the mid-range of the rating scale. Archival data from a law enforcement agency were used to compare the success and failure of anchoring a BARS using a SD criterion versus an a_(wg(1)) criterion. Findings: The a_(wg(1)) criterion was successful at anchoring the full range of the rating scale, but only if the cut-off for anchoring behaviors required a "weak" level of interrater agreement. Implications: The most surprising finding was that the traditional 1.5 SD criterion on a 9-point rating scale is not a particularly stringent agreement requirement for anchoring behaviors. Although we demonstrated the advantages of using a_(wg(1)) to anchor BARS, an equally important conclusion is that incumbents need to be better trained prior to scaling behaviors. Originality/Value: We provide a theoretically defensible approach for anchoring BARS that ameliorates the missing middle anchor problem. Further, the utility of a_(wg(1)) in the BARS context is yet another example of the limitation of r_(wg(1)) when assuming a uniform null distribution.