An Experimental Test of Student Verbal Reports and Teacher Evaluations as a Source of Validity Evidence for Test Development

摘要

The Standards for Educational and Psychological Testing indicate that test instructions, and by extension item objectives, presented to examinees should be sufficiently clear and detailed to help ensure that they respond as developers intend them to respond (Standard 3.20; AERA, APA, & NCME, 1999). The present study investigates the use of verbal reports, one of many sources of evidence for validity arguments, as a way to evaluate the content clarity of 30 items from a large-scale science assessment. Student reports were used to edit items and create a student-modified test form. Evaluations from expert preservice teachers were used to edit the items and create an expert-modified test form. Both experimental forms, along with the original set of 30 items, were then randomly assigned to a sample of 264 examinees. Hierarchical regression analyses indicated that examinee performance on the student-modified and expert-modified forms was similar relative to performance on the original test items. Item statistics indicated that student-modified test items were equally difficult and discriminating as expert-modified test items. The implications of using student and teacher evaluations are discussed for informing test development.