Detection of aberrant response patterns in testing using cumulative sum control schemes.

摘要

Cumulative sum (CUSUM) control schemes are widely used statistical process control (SPC) methods to ensure that a process of interest performs as designed and intended. Composed of three essays, this dissertation applies CUSUM chart methods to detect aberrant response patterns in the context of standardized tests. The tests we use in the studies were either assembled to meet the specifications of a large-scale testing agency, or were assembled randomly with an approach described in literature.;The first essay investigates the likelihood-based PFS, commonly denoted by lz and regarded as one of the best PFSs in the literature. However, we found that the detection performance of the lz statistic was heavily conditional on test characteristics. The simulation results showed that its detection power could severely deteriorate and even become biased (in a hypothesis testing sense) under some specific scenarios. This essay provides an explanation for the potentially poor performance of lz and summarizes the patterns and conditions for which the lz statistic should not be recommended for detecting aberrant behavior.;In the second essay, a new class of cumulative sum (CUSUM) PFSs based on recent work of Zachary G. Stoumbos and some of his coauthors was considered to detect aberrant behavior in the context of linear tests. Extensive Monte Carlo simulations were conducted to compare the detection rates of this new class of CUSUM schemes with those of selected popular PFSs from the literature. We showed that the new class of CUSUM schemes outperforms all of the selected person-fit statistics, for both the true and estimated ability values theta of a test taker.;The third essay extends the above CUSUM PFS model to model-free person-fit detection, based on the Bayes Rule applied to the total number of correct responses from the test. The detection performance of this model-free person-fit CUSUM was then compared with some standard, model-free PFSs from the literature. It was shown that the model-free person-fit CUSUM scheme uniformly and substantially outperforms all considered standard, model-free person-fit statistics. Moreover, we found that the performance of the new class of CUSUM PFSs is "stable" across various scenarios of aberrant behavior.