An investigation of testlet-based item response models with a random facets design in generalizability theory.

摘要

Briggs and Wilson (2004, 2007) proposed the Generalizability in Item Response Modeling (GIRM) approach for estimating variance components in Generalizability Theory (GT) using the Rasch model under the single random facet design. Rather than using observed scores as the traditional GT approach does, the GIRM approach uses expected scores. This simulation study further investigates GIRM and extends the work of Briggs and Wilson (2004, 2007) from a single facet design to the balanced random facets px (i : h) design, where p, i, and h represent persons, items, and testlets, respectively. Under the p x (i : h) design, a multidimensional Testlet Response Theory (TRT) model is used to estimate the expected scores within Markov Chain Monte Carlo (MCMC) simulation; therefore, this method is named the TRT approach.;Three different models are employed to generate three types of simulated data, including seven different universes of simulated admissible observations. For the simulated data, the following results were obtained. The estimates of variance components obtained using the TRT approach are generally quite similar to those obtained using the traditional GT approach, with several exceptions occurring in one particular universe for the estimates of the variance component ph. In addition, the estimates of the relative error variance, absolute error variance, generalizability coefficient, and dependability coefficient obtained using the TRT approach are comparable to those obtained using the traditional GT approach.;This study also examines the performance of MCMC for estimating standard errors of estimates (SEEs) of variance components as compared with the other two methods. The results show that MCMC performs very poorly for estimating SEEs while the SEE formula proposed by Searle (1971; referenced in Brennan, 2001) and Brennan's bootstrap procedures (Brennan 2007) occasionally perform well but often produce underestimates of SEEs of variance components.;The variability underlying the simulated data is truly random, whereas for real world applications, the variability underlying the data would not be truly random. Also, with real data, the TRT models considered in this dissertation may not be the best choice. Therefore, the extent to which estimates of variance components based on GT and TRT models are same using real data remains to be examined.