Performance Assessment for California Teachers (PACT): An evaluation of inter-rater reliability.

摘要

This research evaluated the inter-rater reliability of the Performance Assessment for California Teachers (PACT). Multiple methods for estimating overall rater consistency include percent agreement and Cohen's Kappa (1960), which yielded discrepancies between rater agreement in terms of whether candidates passed or failed particular PACT rubrics. Thus, inter-rater reliability was examined by categorizing Teaching Events into passing and failing categories, in addition to comparing rater consistency by PACT tasks. One study evaluated teacher candidates' perspectives on their preparedness for assessing student learning, and measured their assessment competence using PACT scores and Collaborative Teacher ratings. Results from the comparison of candidate scores introduced a new set of questions regarding the levels of agreement among PACT raters. So a second study analyzed the inter-rater reliability of PACT. Overall, results suggest the need for further exploration of the consequential validity of PACT, in addition to using multiple methods to estimate its inter-rater reliability so that teaching effectiveness can be fairly identified.