Principled assessment as a foundation for standard setting

摘要

This study investigated the impact of using Assessment Engineering (AE) task models as the unit of judgment in a standard setting workshop. The proposed method, or Task Model-based Standard Setting (TMSS), used a procedure similar to that of the Bookmark Standard Setting Procedure; however, task models, rather than items, served as the unit of judgment. The proposed method was compared against the yes/no Angoff method and the Bookmark Standard Setting Procedure in regard to the rigor of the recommended cut-scores, the interpanelist consistency of the cut-scores, and the panelists' understanding of the score scale. The TMSS was found to produce a similar cut-score in regard to rigor and interpanelist consistency as that of the Bookmark; however, panelists were more comfortable with the TMSS procedure and indicated a better understanding of the score scale than the Bookmark group. Panelists indicated the greatest level of comfort/understanding with the yes/no Angoff procedure, which resulted in the least rigorous cut-score. The Angoff group was found to have the lowest level interpanelist consistency comparatively, and did not align with panelists' holistic. Results in this study indicate using task models to set standards, or at least including them in the process, can facilitate panelist understanding of the scale. Furthermore, the TMSS had the strongest collection of validity evidence of the three methods examined in this study.