Stability and sensitivity of a model-based person-fit index in detecting item pre-knowledge in computerized adaptive test.

摘要

Item response theory is a modern test theory. It focuses on the performance of each item. Under this framework, the performance of test takers on a test item can be predicted by a set of abilities. The relationship between the test takers' item performances and the set of abilities underlying item performances can be described by a monotonically increasing function called an item characteristic curve. Due to various personal reasons, the performances of the test takers may depart from the response patterns predicted by the underlying test model. In order to calculate the extent of departure of these aberrant response patterns, a number of methods have been developed under the theme "person-fit statistics". The degree of aberration is calculated as an index called person-fit index. Inside the computerized adaptive testing (CAT), test takers with different abilities will answer different numbers of questions and the difficulties of the items administered to them are usually clustered at the abilities of the test takers. Due to this reason, the application of person-fit indices in the computerized adaptive testing environment to measure misfit is difficult.;When the frequent accesses to the item bank has become feasible, test takers may memorize blocks of test items and share these items with future test takers. Individuals with prior knowledge of some items may use that information to get high scores, in the sense that their test scores have been artificially inflated. FLOR is an index of posterior log-odds ratio used for detecting the use of item pre-knowledge. It can be applied both in the fixed item, fixed length test and the CAT environment. It is a model-based index in which aberrant models are defined in the situation of item pre-knowledge. FLOR describes the likelihood that a response pattern arises from the aberrant models.;The present study used the hf plot to access the sensitivity of the person-fit indices. hf plot is a plot of hit rate against false alarm rate. For a higher hit rate, usually a higher false alarm rate is followed. hf plot provides a good tools for comparison between indices by inspection of the speed of rise of the curves. A sensitive index should give a faster rise of the curve. In this study, sensitivity of an index was defined as the speed of rise of the hf plot, which is represented by a parameter hftau estimated from the data obtained from hf plot.;The present study assessed the stability of FLOR over other variables, which were unrelated to item pre-knowledge. It found that FLOR was stable over the discrimination and difficulty parameters of test items. It was also stable over positions of the exposed items in the test and the initial assignment of prior probability of item pre-knowledge. However, the asymptotes (guessing factor) and the probabilities of item exposure did affect the final values of FLOR seriously.;The present study also found that FLOR has a much superior sensitivity over other indices in detecting item pre-knowledge. Concerning about the sensitivity over different abilities of test takers, it was found that the sensitivity of FLOR was the highest among low ability test takers and the weakest among strong ability test takers in the fixed length and fixed items tests. However, the sensitivities of FLOR became the same among different abilities of test takers if items with difficulties matching their abilities were used in the tests. The number of beneficiaries among the test takers did not affect the sensitivity of FLOR. Moreover, in a simulation to test the differentiating power of FLOR, it was found that FLOR could differentiate item pre-knowledge from other reasons of personal misfits (test anxiety, player, random response and challenger) effectively.;After the stability and sensitivity of FLOR were investigated, the application of it in the CAT environment had become the main concern. The present studies found that both the test length and the number of exposed items affect the final value of FLOR. In the fixed length CAT, the FLOR has a much stronger sensitivity than lz and CUSUM in detecting item pre-knowledge. The sensitivity of FLOR in the fixed length CAT was the same as that in the fixed length fixed items test. If the test length could vary, the sensitivity of FLOR in CAT would be slightly weakened. The Adjusted FLOR index could increase the sensitivity. Concerning about the effect of ability on the sensitivity of FLOR in CAT, it was found that the abilities of the test takers in CAT did not affect the sensitivity of FLOR and Adjusted FLOR.