A generalizability theory approach to standard error estimates for bookmark standard settings

摘要

The bookmark standard-setting procedure is an item response theory-based method that is widely implemented in state testing programs. This study estimates standard errors for cut scores resulting from bookmark standard settings under a generalizability theory model and investigates the effects of different universes of generalization and error sources on standard errors. This study produced several notable results. First, different patterns of variance component estimates are found for different cut scores; therefore, researchers should estimate separate variance components for each cut score and use them to estimate corresponding standard errors. Second, different universes of generalization produce different standard error estimates; thus, policy makers should consider which universe is appropriate for the proposed use of cut scores. Third, participants and groups have nonnegligible effects on several error sources. To decrease the standard errors for cut scores, increasing the number of small groups seems more efficient than increasing the number of participants.