Nonresponse bias in student assessment surveys: A comparison of respondents and non-respondents of the National Survey of Student Engagement at an independent comprehensive Catholic university.

摘要

Based upon the theoretical framework of social foundations inquiry, with engagement as both process and product of the educational process, a quantitative study measured nonresponse bias of first-year undergraduate students in the National Survey of Student Engagement (NSSE) 2004, at one participating NSSE university, an independent comprehensive Catholic university in northeastern Pennsylvania. This assessment tool was developed by higher education professionals to measure engagement as a proxy indicator for undergraduate student learning and development outcomes. Unfortunately, average response rates for the NSSE have been in the low 40 percent range. In such a situation, external validity of the survey results is in question. Therefore, nonresponse bias needs to be measured as an integral part of survey research.;The survey administration was conducted, over approximately a 40-day period, in a multi-stage follow-up process with personal appeals to the subjects. The study tested the research hypotheses by comparing the results of the five NSSE engagement scales and seven student socio-demographic variables using independent sample t-tests and chi-square goodness-of-fit statistical tests. Additional statistical analyses were performed to investigate the comparison of individual survey items that make up the engagement scales, as well as additional items not part of the engagement scales.;Comparative analyses indicated the study respondents were representative of the university non-respondents in terms of race, major of study and first-year grade point average, but not for gender and first-year place of residence. Results for Research Hypothesis One suggest that there is no significant difference between NSSE 2004 first-year respondents and first-year non-respondents on these seven socio-demographic variables. Results for Research Hypothesis Two suggest that there was no significant difference between the two groups on four of the five NSSE National Benchmarks of Effective Education Practice (engagement scales): Level of Academic Challenge, Active and Collaborative Learning, Enriching Educational Experience, and Supportive Campus Environment. However, study results suggest that there was nonresponse bias on the Student-Faculty Interaction Scale and that the non-respondents might be more engaged on the activities questioned by this scale than the respondents. Additional micro-level analyses of individual survey items, both those associated with the engagement scales and those not, indicated a presence of nonresponse bias on numerous items. Several items indicated that non-respondents were more engaged, but most indicated that the non respondents were less engaged than the respondents. Implications of these findings and recommendations are offered for the scholarship of student assessment.