The Case for Forward-Looking Energy Program Evaluations

The Case for Forward-Looking Energy Program Evaluations

The Case for Forward-Looking Energy Program Evaluations
We need to make energy program evaluations more relevant for decision-making. Most existing energy efficiency and renewable energy programs have been concerned with confirming the reasonableness of past investments and impacts on utility ratepayers. But there is also growing interest in evaluations of community programs and state public benefit programs, and they need to focus on learning lessons that we can use to make future choices about spending energy dollars. Looking forward, the questions that we can learn from public program evaluations are: (1) How can we most effectively spend public funds? (2) How can we best design public programs to achieve desired economic and environmental outcomes? and (3) How can we best prioritize choices among project and programs to maximize our return on investment? Program evaluations today often fall short of fully addressing these questions.

To make program evaluations most relevant, we need to make them forward-looking.  While program evaluation is necessarily done by looking at the results of past actions (identifying both desired and undesired outcomes), the conclusions to be drawn are about lessons learned that we can apply for the future. The forward-looking evaluation process starts by assessing the results of past program efforts, and the extent to which various expected and unexpected factors supported or worked against the achievement of desired outcomes. We draw lessons for the future when we identify those factors and then identify ways to modify program design and execution accordingly. In that way, we can effectively enable and leverage positive factors in the future, while avoiding and mitigating negative factors. 

Forward looking evaluations must also consider future change and uncertainty. So if we want to apply lessons learned for the future, then we need to consider how the future is likely to be different from the past.  Lacking an effective crystal ball, we can still identify ranges of reasonable expectations concerning the future context for energy programs, including five dimensions of future impact factors:

  • Shifts in populations, including spatial patterns of age and household types;
  • Shifts in the economy, including spatial patterns of industry, occupation, trade and income;
  • Shifts in energy demand and supply technologies, including sources, uses and costs;
  • Shifts in climate, including patterns of heating, cooling and extreme weather events.

Robust and effective program designs can maximize public ROI (return on investment) by recognizing both the direction and likely range of magnitudes associated with future changes in these factors.

An illustrative example, using predictive analytics. Consider the design of green energy programs and their economic implications. The effectiveness of these programs going forward will depend on how they address future energy demand and supply factors. Climate change is projected to shift future cooling and heating requirements, to varying extents, across America.  Of course heating and cooling technologies are quite different in terms of technology options, their energy efficiencies, fuel sources and fuel costs – so climate shifts can change total energy costs in different directions in different areas. Also the expectation of electrifying transportation of goods and people and electrifying more HVAC technology, to name two areas, will impact electrical generation, transmission and distribution in very significant ways.  In addition, future projections show shifts in growth rates for various types of households and business sectors, each of which also has a different pattern of energy intensity per person and per building area. Putting all of these factors together, we can see some very interesting implications for energy incentive and promotion programs, and their economic implications.

This can be illustrated with TREDPLAN Predictive Analytics, a system built to enable forward-looking program and investment planning and decision-making.  In the left-side graphic below, we see a climate change scenario leading some states and regions to see lower net energy costs per manufacturing worker (blue shading) while others will see higher net costs (red shading). This is driven by projections of how heating and cooling degree days will change in the future, along with costs per kWh and per therm.

Now if energy prices rise 10% in the future, as some experts and models predict, then the map changes as shown in the right-side graphic. In that case, some areas (such as western Kentucky and southern Illinois) still remain blue indicating lower annual energy costs. However, others (such as northern Missouri and Central Illinois, as well as parts of Wisconsin), shift from blue to red, indicating higher total annual energy costs for business, with a possible loss of economic competitiveness. The implication, then, is that energy efficiency and renewable energy programs can make the most difference for economic growth and development when targeted to areas and sectors of the economy where they will can have the most positive effect on relative business operating costs.Climate-Energy-Cost-Graphic

More robust program designs can be developed from forward-looking program evaluations. Improved program evaluations can “walk forward” results of past evaluations to see how similar program designs are likely to have different results in the future, by considering shifts in demographics, the economy and climate.  By then considering alternative scenario assumptions about technology adoption affecting energy generation and demand, one can also develop robust and adaptive program designs that stand to generate positive ROI across the widest possible range of future conditions. 

 Note: TREDPLAN is a predictive analytics system developed for program and investment planning. It incorporates a 50 year history and alternative 50 year forecasts of changes in various elements of population, the economy, trade, climate and technology. It includes a calculation engine to estimate implications for various energy, transportation and economic development cost elements. It also includes a visualization system to show relative impacts across space and time. For more information, see or call 617-303-0424 or email .