Score Quality Issues Related to Individual and Weapon Crew Criterion-Referenced Performance Tests

摘要

Three approaches are described which provide different kinds of information relating to the development of a test and use different amounts of information to arrive at their solutions. All relate to the quality of the score that will result. The binomial model yields the probability that examinees will obtain a certain score given a hypothesized 'true' level of performance. It provides an initial approximation of test length and cutoff scores without test data. The Bayesian model yields the probability that a particular examinee is a member of a certain proficiency group given a specific score. In this case, prior information on typical performance is combined with the binomial model information to relate the observed score to a true performance level. We therefore require information about the examinee population before the scores are observed. The model can improve the classification accuracy of the test and help us set cutoff points and fix test length at a more efficient level. The Rasch model yields the probability that an examinee in a particular skill group will answer a particular item correctly, given the easiness of the item. We need a good deal of information to accomplish the item calibration and person measurement associated with the Rasch model. Here the emphasis is on the particular items that we will rely on for the measurement; the final set is comprised of items which fit the model.