NAEP Validity Studies: A Study of Equating in NAEP

摘要

This study investigates the amount of uncertainty added to NAEP estimates by equating error under both ideal and less than ideal circumstances. For example, circumstances led to a situation in which the 1994 to 1992 reading assessment equating had to be based on a set of common items that was both smaller, and more heavily weighted toward multiple choice, than anticipated. If performance on the two types of items does not change at the same rate over time, such equatings might introduce systematic bias in trends measured from equated scores. Data from past administrations are used to guide simulations of various (better and worse) equating designs, and error due to equating is estimated empirically. The design includes a variety of factors that might affect accuracy of equating, with the levels of each factor based roughly on operational values in the NAEP 1992 and 1994 reading and 1992 mathematics assessments. The purpose is to estimate the approximate additional uncertainty that might be introduced by equating from one assessment wave to the next, and to determine what factors in the equating design contribute most to that uncertainty. The specific factors investigated were number of items in the scale, the proportion of items in the scale taken by each student, the proportion of items in each administration which are common, the proportion of each item type in each scale, the proportion of each item type among common items used for equating, the scale linking strategy (IRT invariance, common item, or multiple group IRT linking), and the change in ability from wave 1 to wave 2.