A comparison between two scoring methods, holistic vs analytic, using two measurement models, the Generalizability Theory and the Many-facet Rasch Measurement, within the context of performance assessment.

摘要

Studies addressing issues related to performance assessment seem to be taking two relatively different directions. One direction is to search for the best scoring method that maximizes reliability and validity of the assessment. This is usually accomplished by comparing two of the most popular scoring methods used with performance assessment, the holistic scoring method vs. the analytic scoring method. Another direction of research dealing with performance assessment is to search for the best measurement model that facilitates better understanding of the data in hand. Two of the most popular measurement models, usually used with performance assessment, are the Generalizability Theory and the Many-facet Rasch Measurement model. Unfortunately, efforts are usually directed to only one of these two directions while the other is ignored. The general objective of this current study was to bring these two lines of research together by studying the interaction between scoring methods and measurement models in terms of their reliability and validity. The goal was to search for the optimum combination between scoring method and measurement model that could be considered a more reliable and valid measure of a student's English as a second language ability. More specifically, the current study had two objectives.; The first objective was to compare the holistic scoring method vs. the analytic scoring method using two measurement models---the Generalizability Theory vs. the Many-facet Rasch Measurement---which resulted in reporting four different scores for each student. Within the Generalizability theory, the analytic scoring method was found to have higher G-coefficient estimates than the holistic scoring method in general. In contrast, the holistic scoring method data was found to have better overall statistical fit indices than the analytic scoring method within the Many-facet Rasch measurement.; The second objective of this study was to identify which of the four scores obtained in the first part of the study was a more valid measure of a student's English as a second language ability based on relationships with other variables approach and using the multiple regression technique. It has been found that students' holistic scores when analyzed by the Many-facet Rasch Measurement have the highest observed R-square value when a multiple regression analysis was conducted using three external variables that were suppose to be related to ESL writing ability as predictors. Another piece of evidence based on the discriminant validity approach suggested that there were no significant differences among the students' four scores in terms of their relationships with students' handwriting quality.