Effects of Unequal Ability Variances on the Performance of Logistic Regression, Mantel-Haenszel, SIBTEST IRT, and IRT Likelihood Ratio for DIF Detection

摘要

Differential item functioning (DIF) of items has become an important issue in test fairness and equity in large-scale assessments. DIF occurs when subgroups of test takers have equal trait levels (on the construct the test is intended to measure, such as ability) but differ in their probabilities of a correct response (Roussos & Stout, 1996). DIF items may threaten the validity of test scores for subgroups (Borsboom, 2006; Maller, 2001) and can mislead researchers about group differences (Woods, 2008). Thus, it is critical to identify DIF items. Numerous parametric and nonparametric methods have been proposed for detecting DIF (Holland & Wainer, 1993; Furlow, Ross, & Gagne, 2009; Rivas, Gabriel, Stark, & Chernyshenko, 2009; Shih & Wang, 2009; Woods, 2009), and a lot of simulation studies have been done to examine the performance of these methods to flag DIF items. However, among these studies, there is little attention to the effects on DIF detection methods of the difference in ability variance between two groups. The empirical study of Bielinski and Davison (1998) and the simulation study of Monahan and Ankenmann (2005) confirmed that the effect of difference in ability variance between reference and focal groups is strong in DIF detection. However, Monahan and Ankenmann's study focused only on the Mantel-Haenszel chi-square test (Monahan & Ankenmann, 2005), whereas Bielinski and Davison's study focused only on the likelihood ratio test (Bielinski & Davison, 1998). Thus, the aim of this study is to examine how the difference in ability variance between reference and focal groups affects four commonly used DIF detection methods: (a) logistic regression modeling (Zumbo, 1999), (b) Mantel-Haenszel chi-square (MH test) (Holland & Thayer, 1988; Mantel & Haenszel, 1959), (c) simultaneous item bias test (SIBTEST; Shealy & Stout, 1993), and (d) likelihood ratio test (Thissen, Steinberg, & Wainer, 1988, 1993).