Comparing Traditional Process Evaluation with the Developmental Evaluation Approach: Experiences with HVAC Programs as Part of a System for Change

摘要

This paper explores potential uses of a developmental evaluation approach for supplementing traditional approaches to evaluating California HVAC programs. The authors consider developmental evaluation’s unique attributes and discuss potential advantages and challenges associated with supplementing current practices with this approach. These discussions are grounded in examples drawn from recent evaluations. Traditional process evaluations ask questions such as: How are program goals reached and what improvements should be made? Traditional impact evaluations seek to document the value and persistence of program impacts attributable to program implementation. Developmental evaluation, in contrast, asks questions such as: To what principles should programs be held accountable, how do programs adapt to contexts in which they operate, and how do programs contribute to sustainable changes? The authors submit that a developmental approach will improve evaluators’ abilities to respond to technological changes and market complexities. In California, the HVAC industry and utilities are attempting to reduce energy use associated with HVAC equipment operation and maintenance through innovative energy efficiency programs and a statewide HVAC strategy. The industry must consider interwoven issues around equipment efficiency, quality installation, quality maintenance, and market demand. Yet current traditional program evaluations do not fully capture the complexity of the situation. These evaluations assess linear progress toward discrete program goals, rather than evaluate program efforts as part of a larger system working to fundamentally shift HVAC practices. Even if energy evaluators cannot fully adopt a developmental approach, aspects of developmental evaluation may provide significant value to the long-term impact of these programs.