Measuring reading comprehension and mathematics instruction in urban middle schools: A pilot study of the Instructional Quality Assessment (CSE Technical Report 681)

摘要

The quality of reading comprehension and mathematics instruction was explored in five urban middle schools using the Instructional Quality Assessment (IQA) toolkit (N = 34 teachers). The IQA is comprised of protocols for rating observed instruction and the quality of teachers' assignments with student work. The purpose of this research was to investigate the reliability and potential validity of the ratings of these data sources. Commensurate with other research on the quality of middle schools, our results indicated that the quality of instruction varied a great deal within schools and was of a "basic" quality overall. Results indicated a moderate to high level of reliability. Four assignments with student work yielded a stable estimate of quality in both content areas, and when teachers complied with the requirements of the research as few as two observations yielded a stable estimate of teaching quality in both content areas as well. The quality of teachers' observations and assignments were significantly associated in mathematics, but not in reading comprehension. Because of the small sample size it was not possible to apply multi-level models. The relation between the IQA and student achievement on the SAT-10 was explored using linear regression techniques. Results indicated that after controlling for students' prior achievement, socio-economic status (SES), ethnicity, language, and IEP status, the IQA assignment measure in reading comprehension predicted student achievement on the Total Reading, Reading Comprehension, and Vocabulary subscores of the SAT-10. The observation measure in reading comprehension predicted student outcomes on the Reading Comprehension subscore of the SAT-10 only. In mathematics, the quality of teachers' assignments predicted students' achievement on the Procedures subscore of the SAT-10. The quality of observed instruction in mathematics predicted students' achievement on the Procedures and Total Math subscores. Without accounting for clustering within classrooms and schools as multilevel models do, our linear regression analyses may lead to results that appear stronger than they actually are. Nevertheless our analyses indicate the direction of trend in these relationships and raise important questions regarding which data sources may be best (classroom assignments or observations) for measuring specific aspects of instruction and student outcomes. Additional research with larger samples of teachers is needed to make definitive conclusions about the validity of the IQA ratings and under what conditions one might choose to either observe in classrooms or collect assignments with student work. The following are appended: (1) IQA Rubrics; and (2) High and Low Quality Examples of Observed Lessons and Assignments. (Contains 21 tables and 9 footnotes.)