A Multidimensional Scaling Approach to Dimensionality Assessment for Measurement Instruments Modeled by Multidimensional Item Response Theory.

摘要

The statistical assessment of dimensionality provides evidence of the underlying constructs measured by a survey or test instrument. This study focuses on educational measurement, specifically tests comprised of items described as multidimensional. That is, items that require examinee proficiency in multiple content areas and/or multiple cognitive skills for a correct response. Therefore, this study utilized multidimensional item response theory (MIRT) to model the examinee-item interaction. Since MIRT modeling characterizes a test with both examinee and item parameters it provides a basis for the statistical analysis of dimensionality. Specifically, MIRT angular distance (i.e., MIRT inter-item angles) compares the content knowledge and/or cognitive skills targeted by items.;In conjunction with MIRT, this study employed multidimensional scaling (MDS) to statistically analyze the item differences in content knowledge and/or cognitive skills. MDS methodology analyzes the observed patterns of dissimilarity to spatially represent the dataset in a dimensionally accurate geometric space. MDS spatial representations depict the relative positions or grouping of the data in relation to the measured dimensions. MDS evaluates dimensionality with a computed statistical index that numerically expresses the fit of the spatial representation to the data. Since the purpose of this research is to investigate MDS methodology for assessing the dimensionality of tests modeled by MIRT, the study uses a simulation design.;The designed simulation included factors known to affect the accuracy of dimensionality assessment. Specifically, two levels for the simulation factors number of dimensions, number of items per dimension, dimensional structure, MIRT model, and MDS computational algorithm were fully crossed resulting in a total of 32 test conditions. In summary, MDS methods were more accurate in the recovery of two-dimensional (2D) tests. Test simulated as 2D were typically recovered with at least 90 percent accuracy across all factor-levels. MDS methods differed in performance for the recovery of three-dimensional (3D) tests. For the higher dimensionality, accuracy rates between MDS methods differed by as much as 41 percentage points. Overall, the successful recovery of the simulated number of dimensions shows the potential for an MDS approach to dimensionality assessment within the context of MIRT.