The Influence of program theory-based evaluation on the use of evaluation information: A longitudinal study

摘要

Theory-based evaluation studies have long been promoted as increasing stakeholder knowledge of the program and improving decision-making (Bickman, 1987; Chen, 2004; Suchman, 1967; Weiss, 1998), but such advocacy has little empirical research support. This paper helps to redress this deficiency by reporting on an empirical study which investigated the relationship between the use of program theory as the basis for an evaluation study and the nature of the evaluation information generated and its utility.ud udThe investigation followed three program evaluation studies through their term and focused on the extent of theory use and the nature and influence of the information from each evaluation. A model of factors which it has been suggested influence the use of program evaluation information was developed and investigated (Alkin, 1985; Cousins & Leithwood, 1986, 1993; Cummings, 1997; Leviton & Hughes, 1981; Hudson-Mabbs, 1993; Pawson & Tilley, 1997; Preskill & Torres, 1997). Structural equation modeling was used to test and further explore the model, while qualitative data were used to gain further insight into the study findings.ud udThe investigation was not able to confirm that a greater use of program theory in an evaluation study improves the usefulness of evaluation findings. The study does, however, identify interactions between program theory and other predictor variables, such as the characteristics of the evaluation study, related to increased information use. It also provides further evidence regarding the importance of stakeholder involvement in an evaluation study on the extent to which they use and are influenced by evaluation information. Furthermore, this research provides insight into perceptions developed by many evaluators through experience and practice, such as the vulnerability of an evaluation study to contextual factors which typically are outside the control of the evaluation team.