Performance assessment: Measurement issues of generalizability, dependability of scoring, and relative information on student performance

摘要

The purpose of this study was to investigate whether the limited number of observations that might be included in a performance assessment would adequately generalize to potential circumstances that would not be observed. Related studies were also conducted to determine how dependably scores are assigned to the measures of students' performance and how different information is provided by paper and pencil test versus performance assessment. A performance assessment was developed in the context of an introductory graduate statistics course and administered to the graduate students along with a paper and pencil test.;A generalizability study was used to estimate the dependability of the performance assessment and to improve the design of the assessment. Dependability of scoring was analyzed through the application of classical test theory and generalizability theory. Correlational and exploratory factor analysis was conducted to determine the relative information provided by two test formats.;This study found that raters do not introduce substantial error into the measurement of performance. Rather, the major source of error is the inconsistency of student performance across tasks, indicating that the number of tasks could be increased to achieve a reliable score for student performance. The correlation between overall scores assigned by two raters and the results of G study suggest that raters are able to consistently evaluate student performance and eventually, the number of raters can be reduced to one and eliminated as a facet in the design of the generalizability study. Relatively high correlation was found between the two measures and there was no evidence of a format factor associated with the use of performance assessment. The factor analytic solution suggests a relationship between factor structure and item discrimination.