Estimation of Decision Consistency Indices for Complex Assessments: Model Based Approaches

摘要

With the implementation of the No Child Left Behind assessment program and the use of proficiency levels as a means of evaluating Annual Yearly Progress, there is a renewed interest in the consistency of classification decisions based on scale scores from achievement test and state-wide proficiency standards. Many of the current methods described in the literature (Huynh, 1976; Hanson and Brennan, 1990; and Livingston and Lewis, 1995) are based on assumptions about the distribution of the conditional errors. Although recent methods (Brennan and Wan, 2004) make no assumptions about the distribution, these methods have one compelling disadvantage: the decision consistency calculated is based on the entire set of data and are not conditional on the location of the cut scores, the student measure and the conditional standard errors of measurement for the students. The decision consistency for a student scoring right at the cut score will be much lower that the decision consistency for a student with a score 5 points above or below that cut score.