An Application of Generalizability Theory on Writing Assessment: Effects of Marking Components Weighting.

摘要

In writing assessment, there are quite a number of factors influencing the marking stability and the reliability of the assessment such as the attitude towards marking and consistency of markers, the physical environment, the design of the items, and marking rubrics. Even the methods to train markers have effects on the reliability of the assessment. Generalizability Theory was used in this research to analyze the Chinese writing assessment of the Territory-wide System Assessment (TSA) so as to improve the reliability of the assessment.;TSA is a standardized test administered centrally by the Hong Kong Examinations and Assessment Authority every year. The target groups are students from Primary 3, Primary 6 and Secondary 3. TSA focuses on assessing students' basic competency on the three core subjects, Chinese Language, English Language and Mathematics. In contrast to the traditional Chinese writing assessment, there was no requirement on the minimum number of words produced by the student. An analytical approach was adopted to assess students' writing tasks. As a result of this measure, students who did well in some particular marking criteria would end up with a good overall performance.;This study was a post-mortem analysis of the raw scores from a sample of 6,000 students who participated in TSA 2006. As there were three sub-papers, the sample consisted of 2,000 students from each sub-paper. Brennan's GENOVA program (1983) was used to calculate the reliabilities of the assessments.;In the assessment, each student was marked by two raters, assigned at random to the student from a pool of 200 raters. These raters had undergone a series of instructional programs and training prior to the job. Each of the two raters gave seven scores to the script. As there was no minimum number of words as required in the writing assessment, a general belief would be generated that if there was insufficient content (as evidenced by a low score in "content") and poor organization (low score in "organization"), then the student would have written so few words that the chance of making mistakes in "vocabulary" (the 6th score) and in "punctuation" (the 7 th score) would be relatively small. In order to rectify the deficiency in marking, this study used three different methods to apply weights on the "vocabulary" score and on the "punctuation" score. For each method, the GENOVA program was used to calculate the reliability of the assessments. After due comparison, it was found that each of the methods used was able to raise the reliabilities of the assessments under investigation, and the most recommended method was to use students' scores in "content and in structure" as weights.;On the one hand, the study has examined the present mode of marking of the writing assessment in the TSA. This gives opportunity for improving the item-setting and the script-marking procedures of the assessment with a view to raising its reliability and giving valuable feedbacks to teaching and learning. On the other hand, the favourable results of applying weights to sub-scores will serve to provide a good example on improving marking rubrics in large-scale standardized tests of writing assessment in Chinese Language.