BPA, in partnership with its public power utility partners, acquires savings from many types of energy efficiency programs and offerings, some of which require site-specific savings estimates. This report documents the first comprehensive impact evaluation of the Site-Specific Savings portfolio.


The Site-Specific Savings portfolio typically accounts for almost half of BPA’s total energy efficiency achievements. However, there have been no previousevaluations of this portfolio.BPA and its customer utilities apply significant resources to estimation of site-specific savings. BPA has undertaken this evaluation to determine what portion of these savings can be verified through the independent data collection and analysis of an appropriately trained team of engineers and technicians.
The evaluation plan established the following objectives:
  1. Estimate first-year kWh savings for the portfolio and for separate domains as needed to understand the savings performance of important portions of the portfolio.
  2. Estimate the lifecycle cost-effectiveness of the portfolio and its domains.
  3. Identify opportunities for improving M&V practices (including data collection and savings estimation) and evaluation methods.


This impact evaluation addressed the capital measures delivered as part of the Site-Specific Savings portfolio during fiscal year (FY) 2012 and 2013 (October 1, 2011 thru September 30, 2013). The BPA reporting system provided data on the projects and measures that comprise this portfolio.
BPA decided to dividethisportfolio into nine domains defined by Option (utilities are Option 1 or 2 for M&V purposes), measure type (Lighting, Non-Lighting, and Energy Smart Reserve Projects), and sector (Industrial and Commercial). These domains are treated as equivalent to atypical “program.” We evaluated a stratified random sample of measures that represented each domain. Overall,this study evaluated less than 3% of measures in the population, but 28% of the savings. Three of the Option 2 utilities chose to increase the sample size so that it would be possible to estimate savings for their service area.Therewas a low non-response rate; we succeeded in evaluating 90% of the sampled measures.
We adhered to the following principles in estimating savings for each sampled measure: (a) Treat all measures consistently; (b) Reuse available data; (c) Focus on the key determinants of Savings; (d) Prioritize items with greatest savings within each measure.Lighting savings were evaluated with a consistent calculation methodology across all sites.Lighting hours of operation were based on data collected from interviews and metering. Non-Lighting and Energy Smart Reserve Power(ESRP)measures were evaluated with best practical algorithms and input data, including,where needed,data from site inspection and efficient-case metering.

Portfolio Savings

The evaluation verified that the savings for the portfolio were nearly the same as the reported savings. As shown in Figure1, the realization rate(RR)1for the entire portfolio is close to one. The reported savings for the entire portfolio falls within the sampling error around the evaluation estimate of savings (RR=0.98 with a relative precision of 3% at a confidence level of 90%2). Realization rates vary across the domains, but in total, the highs and lows balance out and yield a portfolio realization rate near one.
The evaluation also verified costs and benefits associated with the portfolio. Based on the Total Resource Cost Test the evaluation found that thesesavings have a Benefit to Cost ratio of 2.65(lifetime benefits are more than two and a half times the lifetime costs).
figure 1
Figure 1: Realization Rates and Reported Savings by Domain and Overall
Even though the overall realization rate was close to one, there is considerable variation in the realization rates among the sampled measures. Figure2plots the realization rates for each of the 205 measures that we evaluated, with colors and shapes separating the nine domains of the evaluation. A substantial number of measures (approximately 40%) are above and below the dashed lines indicating that their evaluated savings is more than 20% different than the reported savings. However, for the entire portfolio these differences offset each other and result in the portfolio realization rate of 0.98.
figure 2
Figure 2: Distribution of Realization Rates within Each Domain

Lighting Savings

Lighting measures had an overall realization rate of 1. However, we found significant offsetting effects between Option 1 (RR=0.93) and Option 2 (RR=1.08) utilities. As shown in Figure 3, sector made little difference for Option 1, where the evaluation found about 7% less savings for both Commercial and Industrial measures. However, for Option 2, the evaluation found 8.6% more savings for Commercial measures, and 1.9% more for Industrial measures. One Option 2 utility applied realization rates (from a prior evaluation) before reporting savings to BPA. However, this was done almost exclusively for Commercial measures.
Based on the Total Resource Cost Test, the evaluation found that the savings for Lighting measures have a Benefit to Cost ratio of 2.47. This ratio for Option 1-Lighting (2.74) is higher than for Option 2-Lighting (2.23).
figure 3
Figure 3: Lighting Savings (aMW) by Domain

Non-Lighting Savings

For Non-Lighting measures, the overall realization rate is 1.03. Both Option 1 and 2 have realization rates greater than 1, but the Option 2 realization rate of 1.07 is higher than Option 1 RR of 1.02. As shown in Figure 4, sector makes a difference, with reported and evaluated savings much closer for Industrial measure than for commercial. Evaluation found at least 10% more savings for Commercial savings for both Option 1 and 2.

Based on the Total Resource Cost Test the evaluation found that the savings for Non-Lighting measures have a Benefit to Cost ratio of 2.85, somewhat higher than for Lighting measures. This ratio for Option 1 (3.17) is higher than for Option 2 (2.25).

reported and evaluated savings
Figure 4: Non-Lighting Savings (aMW) by Domain

ESRP Savings

Evaluated savings for this program are substantially lower than reported, yet due to its small size, the impact on the portfolio is not substantial. The ESRP realization rate of 0.49 is the lowest among all the domains. The factors leading to this low realization rate include incomplete implementation of measure and downstream reuse of a large portion of the “saved” water. Even with this low realization rate, these savings still have a benefit to cost ratio of 2.76.

Adherence to Protocols

The evaluation also had several findings related to compliance with BPA M&V Protocols and other “Guidelines” that are relevant to this portfolio. Compliance with the BPA M&V protocol selection guide was highest with Option 1 measures and was lowest with Option 2-Commercial measures. Most measures complied with IM documentation requirements, except for Option 1 Lighting, Option 1 Non-lighting, and ESRP invoices. Working savings models are very useful for evaluation and were available for most measures, except for Option 2-Industrial and ESRP measures. Regarding TAP assignment, Option 1 lighting measures did not have TAP codes and Option 2 measures were misclassified more than 40% of the time.


Our most important recommendations are:
  1. Avoid Embedded Realization Rates. Best practice is to apply realization rates to the total savings for a domain or portfolio rather than in the individual measure savings data maintained the reporting system. Do not allow utilities to apply realization rates to their savings estimates prior to reporting savings to BPA.
  2. Enhance the M&V Protocols
    1. Avoid or Improve Simplified Saving Calculators: Some Option 2 projects use “deemed” values or simplified calculators for Non-Lighting measures. These do not provide reliable site-specific estimates of savings. Require that site-specific savings estimates be in accordance with BPA M&V protocols or that simplified calculators are upgraded to conform to the RTF guidelines for Standard Protocols.
    2. Clarify BPA M&V Protocols. The BPA M&V Protocols do not provide clear direction on when and how to compute first-year vs. lifetime savings. They are also not aligned with RTF Guidelines on the definition of current practice baseline. Both issues should be clarified.
  3. Improve Quality Control for ESRP projects. The savings for this domain are being overestimated, although this domain accounts for only 3% of the portfolio. Provide additional quality control review of M&V data collection and modelling for these projects.
  4. Improve Lighting Calculators: The BPA and Option 2 lighting calculators are not consistent and they both lack key features. Improve the BPA Lighting Calculator and require that Option 2 calculators include the same improvements.
  5. Improve and Simplify Program Documentation: Ensure that project documentation includes working M&V models, the M&V protocol used, project invoices, and improved TAP coding. Also, investigate opportunities for reducing redundancy or unnecessary reporting and for developing tools that reduce the reporting effort and facilitate quality control.
  6. Improve Future Evaluations: Align future evaluations with updated M&V protocols, consider faster or real-time evaluation approaches and simplify end-user contact.