The application and empirical comparison of item parameters of Classical Test Theory and Partial Credit Model of Rasch in performance assessments

摘要

This study empirically compares the Classical Test Theory (CTT) and the Partial Credit Model(PCM) of Rasch focusing on the invariance of item parameters. The invariance concept which isthe consequence of the principle of specific objectivity was tested in both CTT and PCM using theresults of learners who wrote the National Senior Certificate (NSC) Mathematics examinations in2010. The difficulty levels of the test items were estimated from the independent samples of learn-ers. The same sample of learners used in the calibration of the difficulty levels of the test items inthe PCM model were also used in the calibration of the difficulty levels of the test items in CTTmodel. The estimates of the difficulty levels of the test items were done using RUMM2030 in thecase of PCM while SAS was used in the case of CTT. RUMM2030 and SAS are both the statisticalsoftwares. The analysis of variance (ANOVA) was used to compare the four different design groupsof test takers. In cases where the ANOVA showed a significant difference between the means of thedesign groups, the Tukeys groupings was used to establish where the difference came from.The research findings were that the test items' difficulty parameter estimates based on the CTT theoretical framework were not invariant across the different independent sample groups. The over-all findings from this study were that the CTT theoretical framework was unable to produce itemdifficulty invariant parameter estimates. The PCM estimates were very stable in the sense that formost of the items, there was no significant difference between the means of at least three designgroups and the one that deviated from the rest did not deviate that much. The item parameters ofthe group that was representative of the population (proportional allocation) and the one where thesame number of learners (50 learners) was taken from different performance categories did not differsignificantly for all the items except for item 6.6 in examination question paper 2. It is apparentthat for the test item parameters to be invariant of the group of test takers in PCM, the group oftest takers must be heterogeneous and each performance category needed to be big enough for the proper calibration of item parameters.The higher values of the estimated item parameters in CTT were consistently found in the samplethat was dominated by the high proficient learners in Mathematics ("bad") and the lowest valueswere consistently calculated in the design group that was dominated by the less proficient learners. This phenomenon was not apparent in the Rasch model.