Using Differential Item Functioning to Test for InterraterReliability in Constructed Response Items

摘要

The purpose of this study was to investigate a new way of evaluating interraterreliability that can allow one to determine if two raters differ with respect totheir rating on a polytomous rating scale or constructed response item.Specifically, differential item functioning (DIF) analyses were used to assessinterrater reliability and compared with traditional interrater reliabilitymeasures. Three different procedures that can be used as measures of interraterreliability were compared: (1) intraclass correlation coefficient (ICC), (2)Cohen’s kappa statistic, and (3) DIF statistic obtained from Poly-SIBTEST. Theresults of this investigation indicated that DIF procedures appear to be apromising alternative to assess the interrater reliability of constructedresponse items, or other polytomous types of items, such as rating scales.Furthermore, using DIF to assess interrater reliability does not require a fullycrossed design and allows one to determine if a rater is either more severe, ormore lenient, in their scoring of each individual polytomous item on a test orrating scale.