Real-Time Evaluation:Blurring the Line between Program, Evaluation and Market Research

摘要

The Northwest Energy Efficiency Alliance has been employing 'real-time evaluation', atype of formative evaluation rarely used in the energy-efficiency industry, to address twocommon situations in which traditional ex-post evaluations cannot provide adequate or timelyinformation: (1) pilot programs when the decision to move ahead with a full-scale program needsto be made in the latter stages of the pilot, before all results are available; and (2) when smallscale,targeted market research is necessary to make a program more effective after it has beenlaunched but the implementation staff either lack the resources or expertise for such work.In the first situation, we have had evaluators work closely with program implementersreviewing materials and providing verbal and written program feedback on very short turnaround.This ensures that our program decision-makers have the maximum amount of evaluationinformation at all times. For market research situations, evaluators are generally more qualifiedto plan and conduct such work than implementers. In addition, it is more cost-efficient to have anevaluator who is already intimately familiar with the project do the work than to hire an outsidecontractor with no project experience.The potentially negative side of real-time evaluation is that such close association withthe program may put the evaluator's credibility or objectivity at risk. This paper will describe thevalue real-time evaluators have brought to Alliance projects, discuss trade-offs betweentraditional and real-time evaluation approaches, and suggest ways to decide when real-timeevaluation makes sense and how it can be used to complement traditional approaches.