Predictive Validity of Conventional and Adaptive Tests in an Air Force Training Environment

摘要

Conventional ASVAB-7 Arithmetic Reasoning (AR) and Word Knowledge (WK) tests were compared with computer-administered adaptive tests as predictors of performance in an Air Force Jet Engine Mechanic (JEM) training course. All test items were calibrated using a 3-parameter logistic item response model. Adaptive tests were composed of items selected by Bayesian and stratified maximum information (STMI) strategies. Each of 495 JEM trainees was administered three AR tests and three WK tests by computer, including two adaptive tests (Bayesian and STMI) and one conventional ASVAB test in each ability area. Since the two adaptive tests selected items from the same item pools, in each ability area a special item fill-in procedure was used during an examinee's second adaptive test to eliminate repeated presentation of the same items. The conventional ASVAB tests were scored by number-correct, Bayesian, and maximum likelihood methods. Order of test administration was counterbalanced in subgroups of examinees. Validity data were analyzed by multivariate linear-model analyses with testing-strategy/scoring-method (TSSM), item type (AR, WK) and/or order of administration as independent variables, and single-test and composite validities as dependent variables. Additional dependent variables studied included characteristics of ability estimates (distributions, intercorrelations, and score information functions), examinee and computer response times, and the effect of fixed versus variable entry.