Item Calibration: Medium-of-Administration Effect on Computerized Adaptive Scores.

摘要

An important question in the development of item pools for computerized adaptive tests (CATs) is whether data for calibrating items should be collected by a paper-and-pencil (PP) or a computer administration of the items. This study evaluated the effect on adaptive scores of using a PP calibration. The correspondence between adaptive scores obtained with computer-administered items and a PP calibration with adaptive scores obtained with computer-administered items and a computer calibration was evaluated. Forty items from each of four Armed Services Vocational Aptitude Battery (ASVAB) content areas (general science, arithmetic reasoning, work knowledge, and shop information) were administered by computer to one group of Navy recruits and by PP to a second group. These data were used to obtain computer-based and PP-based calibrations of the items. Each calibration was then used to estimate item response theory adaptive scores for a third group of recruits who received the items by computer. The effect of medium of administration was assessed by comparative regression, correlation, and reliability analyses of the scores using the alternative calibrations. Results indicate that, although statistically significant medium effects were found on some content areas, medium of administration did not affect the reliability of the adaptive scores. Although these findings support the use of the PP parameters of the current CAT-ASVAB item pool, it is recommended that further analyses be performed to elucidate the significant effects. Item calibration, Computerized adaptive testing, Computerized ability tests, Computer-administered tests, Calibration mode, Ability tests.