On the Statistical and Practical Limitations of Thurstonian IRTModels

摘要

Forced-choice questionnaires have been proposed to avoid common response biasestypically associated with rating scale questionnaires. To overcome ipsativityissues of trait scores obtained from classical scoring approaches offorced-choice items, advanced methods from item response theory (IRT) such asthe Thurstonian IRT model have been proposed. For convenient modelspecification, we introduce the thurstonianIRT R package, which uses Mplus,lavaan, and Stan for model estimation. Based on practical considerations, weestablish that items within one block need to be equally keyed to achievesimilar social desirability, which is essential for creating forced-choicequestionnaires that have the potential to resist faking intentions. According toextensive simulations, measuring up to five traits using blocks of only equallykeyed items does not yield sufficiently accurate trait scores and inter-traitcorrelation estimates, neither for frequentist nor for Bayesian estimationmethods. As a result, persons’ trait scores remain partially ipsative and, thus,do not allow for valid comparisons between persons. However, we demonstrate thattrait scores based on only equally keyed blocks can be improved substantially bymeasuring a sizable number of traits. More specifically, in our simulations of30 traits, scores based on only equally keyed blocks were non-ipsative andhighly accurate. We conclude that in high-stakes situations where persons aremotivated to give fake answers, Thurstonian IRT models should only be applied totests measuring a sizable number of traits.