Impact of mixed method designs on knowledge gain, credibility, and utility of program evaluation findings.

摘要

This study attempted to understand the relationship between evaluation approach and the perceived knowledge gain, credibility, and utility of findings. Specifically, the researcher investigated whether or not quantitative, qualitative, and mixed-method evaluations produced different kinds and amounts of knowledge gain, different levels of credibility, or suggested different types of use.;To investigate this question, the researcher selected a group of consumers of evaluations, presented them with three simulated evaluation case summaries, and interviewed them for their reactions. Participants included ten administrators from academic success and student service programs at Arizona State University. The evaluation case summaries were of a student counseling center, a summer transition program, and a supplemental instruction program. Each summary highlighted the evaluation's purposes, research questions, data collection methods, findings, conclusions, and recommendations.;The interview had two phases. During the initial phase, participants responded to questions about each of the summaries separately. For each summary, participants were asked questions regarding perceived knowledge gain, credibility, and utility. During the second phase, participants were asked to rank the summaries on the three dimensions.;Participants ranked the mixed-method summary the highest in knowledge gain because it portrayed the most comprehensive picture of program participants, processes, and outcomes. The mixed-method summary was ranked the highest in credibility because it allowed participants to experience the program through the eyes and voices of the students. Participants also ranked the mixed-method summary the highest in utility. They claimed it prepared them for such internal administrative responsibilities as strategic planning, high stakes decision-making, and programmatic improvement. Participants also anticipated they could take the summary to their superiors to report program outcomes and justify resource requests.;The study makes contributions to evaluation research by suggesting that evaluation approaches produce different kinds and amounts of knowledge gain, produce different levels of credibility, and suggest different kinds of use. These findings extend the current theoretical views regarding evaluation use. The study also makes contributions to evaluation practice in the training of evaluation professionals.