Murphy and Cleveland (1995) defined rating accuracy as one type of rating quality index, which has been used to evaluate rating quality, rater performance, and even training efficiency. Engelhard (2013) indicated that "accuracy can be defined as the comparison between unknown processes and a defined standard process in order to adjust the unknown process until it matches the standard" (p. 232). In other words, accuracy measures the distance between the operational ratings and the criterion ratings. There are many ways to obtain the criterion ratings. Engelhard (1996) used the criterion ratings defined by a panel of experts. Wolfe and McVay (2012) have defined the true ratings based on average ratings across the operational raters. Rater accuracy within the context of writing assessment can be defined as a latent trait indicating how accurate a rater is in scoring the essays (Engelhard 2013). Other ways to calculate accuracy ratings have been proposed (Cronbach 1955; Sulsky and Balzer 1988; Jones 2007).
展开▼
