TODAY’S STUDY: California’s Zero Emissions Vehicle Program
Process and Outcome Evaluation of the Alternative and Renewable Fuel and Vehicle Technology Program
Lloyd Dixon, Tom LaTourrette, et. al, September 2017 (Rand Corp)
This report presents the results of a process and outcome evaluation of the California Energy Commission’s (CEC’s) Alternative and Renewable Fuel and Vehicle Technology Program (ARFVTP), relying on quantitative and qualitative methods. For the process evaluation, we conclude the following: The process for developing the ARFVTP’s investment plan has been reasonable; CEC has targeted major barriers to market viability across the main fuel-sector types funded by the ARFVTP (hydrogen, electricity, diesel substitutes, gasoline substitutes, and natural gas and biomethane); the ARFVTP has been successful in soliciting proposals; the criteria and weights that the CEC used to evaluate award applications are appropriate, but there are some concerns about the amount of time required to review applications and execute agreements; there are both strengths and weaknesses in how CEC oversees grants once they begin; completing projects on time has been a challenge for many awardees; and awardees are largely achieving their projects’ most-central technical objectives. For the outcomes so far, we conclude that the ARFVTP has made considerable progress reducing many barriers to the market viability of alternative fuels and vehicles. We also find that the ARFVTP seems to have allowed awardees to proceed with projects they would not have undertaken otherwise; after completing their ARFVTP projects, awardees overwhelmingly indicated they had taken steps to continue to develop, market, install, or produce the funded product or technology; and on the workforce side, ARFVTP project grants supported some employment both in California and elsewhere and did result in some hiring for the projects. Stakeholder suggestions on ARFVTP investment priorities going forward vary by fuel type but generally call for continued program support. We also note that insufficient consumer awareness of, and knowledge about, alternative fuels and vehicles was one of the most frequently identified barriers to market viability across all fuel types and that the ARFVTP should not shy away from research, development, and demonstration projects simply because their outcomes can be difficult to quantify.
Over the past decade, through a series of legislative acts and executive orders, California has instituted a comprehensive set of programs and regulations aimed at reducing fossil fuel use and mitigating climate change. The Alternative and Renewable Fuel and Vehicle Technology Program (ARFVTP) was established with Assembly Bill 118 (Núñez) in 2007, as amended by Assembly Bill 109 (Núñez) in 2008. It is managed by the California Energy Commission (CEC) and is a key element of California’s portfolio of climate-related regulations and incentives. In September 2013, the legislature reauthorized the program, extending ARFVTP funding through January 1, 2024 (Assembly Bill 8 [Perea]). The legislature directed CEC to develop a program that provides grants, loans, and other types of financial support
to develop and deploy innovative technologies that transform California’s fuel and vehicle types to help attain the state’s climate change policies. The emphasis of this program shall be to develop and deploy technology and alternative and renewable fuels in the marketplace, without adopting any one preferred fuel or technology.1
The ARFVTP provides about $100 million annually to support projects on research, development, and demonstration (RD&D) and early-stage commercialization for various lowercarbon fuels and vehicle technologies; the ARFVTP spans such options as liquid biofuels, natural gas and biomethane, plug-in hybrid electric vehicles (PHEVs) and battery electric vehicles (BEVs), and hydrogen fuel cell electric vehicles (FCEVs). Overall, CEC will be able to invest a total of $1.5 billion between 2009 and 2024.
Table S.1 shows the number of ARFVTP projects and amounts awarded through January 2015 and invoices paid on these projects through December 2016 by fuel sector, market segment, and technology development phase. All told, the table shows 303 projects, nearly $600 million awarded, and $400 million in invoices paid. Of the 303 projects, the majority are in the electricity fuel sector, involve fueling infrastructure, and are production, distribution, and deployment (PD&D) versus RD&D. The program is still very much in progress: Of the 303 projects funded between 2009 and January 2015, 114 were completed as of September 2016, 155 remained ongoing, 32 had been canceled, and two had not started as of that time.
The legislation that created ARFVTP directed the CEC to regularly evaluate the contribution of funded projects toward promoting a transition to a diverse portfolio of clean alternative fuels, and this report is intended to contribute to that evaluation effort.
To this end, we developed a conceptual framework for the evaluation and the questions to be addressed and then designed and implemented an evaluation, measurement, and verification plan.
The evaluation of the ARFVTP was organized around an outcome–sequence conceptual model (or logic model) for the ARFVTP and included both a process and outcome evaluation.
Within the context of the conceptual model, the following questions are addressed in the process evaluation:
• Has the process for developing the ARFVTP investment plan been reasonable?
• Has the program targeted major barriers?
• Has the ARFVTP been successful in soliciting proposals?
• Has the ARFVTP been successful in evaluating proposals and awarding contracts in a timely manner?
• Has the ARFVTP done an effective job administering the awards?
• Have awardees secured the proposed matching funds?
• Have awardees completed their projects on time?
• To what extent have awardees achieved the main technical objectives of their projects?
Within the context of the conceptual model, the following questions are addressed in the outcome evaluation:
• What has been the impact of the awards on the products and technologies offered by the awardee? (A key aspect of this question is the extent to which the award allowed the awardee to pursue projects that would not have been pursued without the award.)
• What are the market prospects for the products or technologies funded by the award?
• To what extent have funded projects reduced barriers to the market viability of alternative fuels and vehicles? (By market viability, we mean the ability of the private sector to produce and sell alternative fuels and vehicles without government subsidy.) 2
• What has been the impact of the program on employment and workforce capabilities?
This evaluation focuses on the processes and outcomes of the program. It evaluates the processes for developing the investment plan and whether the program is addressing key barriers. However, it does not assess the total funding level for the program or the allocation of resources across fuel types.
To answer the process and outcome evaluation questions, we used both quantitative and qualitative methods:
• Review project documentation to arrive at an independent assessment of the extent to which project technical objectives were achieved.
• Survey ARFVTP awardees, unsuccessful applicants, and potential applicants. • Conduct in-depth reviews of selected ARFVTP projects.
• Verify installation of electric vehicles supply equipment (EVSE) for a selected set of awards.
• Assemble and analyze CEC administrative data on program processes and interview CEC staff on program processes.
• Collect additional information on trends in alternative fuel and vehicle technology markets to inform the outcome evaluation.
A key data source across both evaluations is the web-based survey. The numbers of surveyed organizations and respondents are shown in Table S.2. In addition, 38 in-depth project reviews were completed, and 11 CEC staff were interviewed about CEC processes for developing the investment plans, soliciting applications and making awards, and administering awards.
In this section, we discuss our key conclusions relative to the process and outcome evaluation questions above for the five fuel sectors and the workforce sector.
S.2.1 Conclusions from the Process Evaluation
S.2.1.1 Has the Process for Developing the ARFVTP Investment Plan Been Reasonable?
We find the process for developing the plan has been reasonable.
Key challenges in the ARFVTP investment planning process include keeping sufficiently abreast of technologies and market trends in a wide variety of markets/sectors and managing the inevitable uncertainty that comes from investing in approaches that have not yet reached market viability. To address these challenges, CEC has heavily relied on engaging with industry and other stakeholders to understand changing conditions in the economic and technological environments.3
Opportunities for input come in several forms, including a 25-member advisory board and regular public workshops during which stakeholders and members of the general public can learn about, and provide input into, investment plans either in person or remotely. Recently, CEC has sought to deepen its stakeholder engagement by using “merit review” workshops that drill down into specific investment areas (e.g., biofuels), glean lessons learned from past projects, and hear from panels of experts who provide critical feedback on past investments and insights into future trends.
Our analysis also suggests that stakeholders believe that CEC takes their input seriously; specifically, based on data from our survey, more than 70 percent of those surveyed who participated in ARFVTP workshops and public hearings somewhat or strongly agreed that CEC seriously considers their input.
S.2.1.2 Has the Program Targeted Major Barriers?
CEC has targeted major barriers to market viability across the five main fuel types funded by the program: hydrogen, electricity, diesel substitutes, gasoline substitutes, and natural gas and biomethane. During the project, we identified major barriers to market viability in each of the different market segments: feedstocks for fuel production, fuel production, fueling infrastructure, vehicles, and consumers. We also identified barriers to the development of the workforce needed to staff growing alternative and renewable fuel and vehicle technology (ARFVT) industries. ARFVTP-funded projects have aimed to reduce many, but not all, of these barriers.
Hydrogen projects focused on fueling infrastructure and, with few exceptions, have left barriers related to vehicles and fuel production to other parties. Readiness grants addressed the difficulty of obtaining permits for siting hydrogen-fueling stations.4 Both the readiness grants and grants to the California Fuel Cell Partnership (CFCP) directly targeted limited consumer awareness of hydrogen fuels and vehicles; other projects, by increasing the visibility of hydrogen-fueling stations, indirectly targeted consumer awareness. The program also supported the California Environmental Protection Agency’s Air Resources Board Clean Vehicle Rebate Program (CVRP), which includes rebates for fuel cell electric vehicles (EVs) through a series of interagency rebates.
The electricity projects targeted all the major barriers to market viability that existed when the ARFVTP began. These include both fueling infrastructure and vehicle-related barriers. As with hydrogen, a substantial number of regional readiness grants addressed the difficulty of obtaining permits and included components to promote consumer awareness about EVs and the charging infrastructure.
Feedstock, fuel production, and fueling infrastructure barriers were targeted for both the diesel substitutes and gasoline substitutes fuel types. Vehicle barriers were less important for both fuel types, and the ARFVTP has funded few vehicle projects for these two fuel types.
The natural gas and biomethane projects targeted the widest range of barriers, addressing feedstock, fuel production, fueling infrastructure, vehicle, and consumer barriers.
The workforce development projects generally targeted barriers that a new industry faces in developing its workforce, especially the barriers of limited understanding of industry needs and lack of coordination among stakeholders.
S.2.1.3 Has the ARFVTP Been Successful in Soliciting Proposals? The first step to attracting high-quality applicants is achieving broad-based awareness about a program; in the case of the ARFVTP, there does appear to be a general awareness about the program. A majority of potential applicants participating in this study were aware of the program, and nearly all current and past grantees interviewed said that the program is well known in their industries.
Most project opportunity notices (PONs) have attracted more requests for funds than the total amount of funds available and appear to attract a healthy diversity of applicants, with only a third of applicants applying more than once. Competition for grants is intense, with only 26 percent of applications successful overall. The success rate is modest across all fuel types, varying from 16 percent for hydrogen to 42 percent for natural gas and biomethane.
S.2.1.4 Has the ARFVTP Been Successful in Evaluating Proposals and Awarding Contracts in a Timely Manner?
We found that survey respondents believe the criteria and weights used by CEC to evaluate award applications are appropriate, but there were some concerns. A small number of interviewees (both CEC staff and grantees) expressed concerns about whether the agency has the skill and bandwidth to fully assess the financial and operational capacities of applicants.
A bigger concern was the amount of time required to review applications and execute agreements. Indeed, the mean amount of time between application due date and notice of whether the project was chosen for funding (known as the Notice of Proposed Award [NOPA]) is four months. It takes even longer for CEC to enter into a contract with the awardee: The average time elapsed between the NOPA and an approved contract is nearly eight and a half months. We heard concerns from CEC and grantees alike that the time required to review and award grants is too long given the speed at which market conditions change. CEC must take adequate time to ensure that proposed projects represent worthy uses of public funds and to enter into contracts that protect the public interest, but the long period between proposal submission and project funding for many projects can limit awardees’ abilities to capitalize on technology and market opportunities. However, there is evidence that these time lines have improved as CEC gained more experience running the program.
S.2.1.5 Has the ARFVTP Done an Effective Job Administering the Awards?
We found that there were both strengths and weaknesses in how CEC oversees grants once the grants begin. On the strengths side, the Commission Agreement Managers (CAMs)—the CEC staff who directly oversee individual projects—were generally well regarded by awardees. On the weaknesses side, a small number of awardees expressed concern about CAM turnover. Some awardees cited the burdens associated with performance reporting but recognized the need for public accountability and noted that the process had become easier as they became more accustomed to ARFVTP’s format and requirements.
Awardees were more critical of financial-reporting requirements, which many regarded as unreasonably detailed. Many found CEC’s process for approving changes in project scope or budget allocation burdensome. More than 60 percent of awardees who answered a survey question on the CEC process for approving project modifications either somewhat agreed or strongly agreed that the process was burdensome, and more than one-quarter of those believed that the delays had a substantial impact on administrative cost and project schedule.
CEC often withholds 10 percent of the invoiced amount until all award requirements have been completed. Although this practice helps ensure that awardees provide the work products they promised, just less than 30 percent of awardees responding to the survey said the practice caused their organizations some or substantial financial difficulty.
Despite these concerns, survey respondents and interviewees were generally positive about their experiences with the program. Indeed, a clear majority indicated that they appreciated the program and would apply again if the opportunity or need arose. However, firms involved with developing and producing gasoline substitutes were much more ambivalent about the ARFVTP.
Our evaluation also revealed a need for better document-management practices and a more systematic approach to data collection on the part of CEC. Project documents are difficult to locate, often poorly labeled, and sometimes exist in multiple versions. Such chaotic document management creates impediments for new CAMs inheriting existing projects and independent program evaluators in understanding the history and status of projects. In addition, final reports have no required structure or elements; as a result, it is often difficult to ascertain what the project accomplishments were and how those might compare, or add to, those of other similar projects.
S.2.1.6 Have Awardees Secured the Proposed Matching Funds?
Most, but not all, projects require matching funds (a portion of project funding that comes from a source other than the ARFVTP), and some applicants propose match amounts that exceed the minimum requirement. We found that awardees by and large secure the proposed matching funds. Only 10 percent of completed projects did not achieve the required matching funds. Failure to secure the required matching funds could either mean that the awardee did not complete all the proposed work or that the project ended up costing less than expected.
S.2.1.7 Have Awardees Completed Their Projects on Time?
Completing projects on time has been a challenge for many awardees. Approximately 50 percent of completed projects were completed on or ahead of schedule. Most of the remaining projects were completed seven to 24 months behind schedule, while a small percentage of projects were completed more than two years behind schedule. Delays are greater for ongoing projects, and, if anything, one should expect delays for these projects to lengthen over time because some of the projects have not been active long enough to fall far behind schedule. Delays also tend to be greater for RD&D projects than for PD&D projects.
Delays are most severe for the hydrogen projects. Two of the three projects for which information is available were completed more than two years behind schedule. However, there is some indication that improvements are being made. Hydrogen awardees report that they have been able to reduce the time needed to place a hydrogen-fueling station from 18–24 months to 12–18 months.
Delays in obtaining required permits, the need to change project location, and unexpected problems with the technology were the most common reasons for delay.
S.2.1.8 To What Extent Have Awardees Achieved the Main Technical Objectives of Their Projects?
Awardees are by and large achieving the technical objectives most central to their projects. The RAND team reviewed the project documents to identify the main technical objectives of each funded project.5 We then independently evaluated the extent to which project technical objectives were achieved. Survey respondents were allowed to update these assessments. There was little difference between the RAND and the awardee assessments, except that awardees were able to report the extent achieved in cases for which RAND had insufficient information to make a determination. Using the RAND assessment as updated by survey respondent, we found that only 12 percent of the 364 technical objectives for 106 completed projects (not including workforce development projects and a few multifuel projects) had not been fully achieved, and the percentage does not vary a great deal by fuel type. Eighty-one percent of the technical objectives were fully achieved, and the remaining 7 percent either were in progress or the extent achieved could not be determined. In some cases, a shortfall in one project (e.g., a technical objective to install a specific number of EV charging stations) was offset by surfeits in other projects with a similar technical objective.
As expected, most of the main technical objectives for the ongoing projects remain in progress, but awardees are confident that a high percentage of their technical objectives will be fully achieved. We also found that the success in achieving technical objectives was similar for RD&D projects and for PD&D ones.
For the EV charging station projects, we visited and verified the operability of 326 (about 4 percent) of the more than 8,000 stations funded by the program (more than 6,700 of which have already been installed). The stations tested were spread across 94 different sites. We found that more than 95 percent of these stations were operating properly. We also highlighted a number of installation and operation issues that may help guide future awardees in getting the most benefit from ARFVTP funding.
The technical objectives for three large ARFVTP-funded workforce development projects were in many cases achieved; however, there are important cases in which objectives were not achieved, and lack of data limited our ability to evaluate some efforts. One workforce development project largely completed its technical objectives. In a second case, only about half of the anticipated number of workers were trained. In the third, almost no records of any kind were available, so we could not evaluate whether the technical objectives of the award were accomplished.
S.2.2 Conclusions from the Outcome Evaluation
S.2.2.1 What Has Been the Impact of the Awards on the Products and Technologies Offered by the Awardee?
Critical to the assessment of the ARFVTP’s impact is the extent to which awards have allowed awardees to pursue projects that would not have been otherwise pursued. One way to determine this is to see what those who applied for awards but were turned down— unsuccessful applicants—have done. The advantage of this indicator is that it is based on actual behavior. The disadvantage is that the projects proposed by the unsuccessful applicants may have been more poorly conceived or planned than those funded by the ARFVTP. Based on survey responses, slightly more than one-half of unsuccessful applicants participating in this evaluation had not yet proceeded with the project after their applications were denied. A second approach is to ask awardees what they would have done had they not received the award. Estimates of the percentage of awardees that would not have proceeded without the award are imprecise, but they are consistent with those for the unsuccessful applications. Findings were similar for RD&D and PD&D projects, and there was no strong evidence of differences across fuel types.
These findings tentatively suggest that roughly one-half of the projects funded by the ARFVTP would not have proceeded otherwise. However, the ability to proceed absent ARFVTP funding rests on the availability of other government and private funding sources. If one funding source disappears, other funding sources may be able to fill in the gap. However, if the ARFVTP ends and other funding sources follow suit, then the project may not be able to proceed.
S.2.2.2 What Are the Market Prospects of the Products or Technologies Funded by the Award?
After completing their ARFVTP projects, awardees overwhelmingly indicated they had taken steps to continue to develop, market, install, or produce the product or technology funded by the award. Substantial percentages reported that they had worked on ways to reduce cost and to improve product performance and had attempted to identify potential customers for the product. These findings suggest that awardees believe that the products of their ARFVTPfunded projects have market potential. The growing number of competitors that awardees are facing in their industries (discussed in Section S.3) provides additional evidence of market potential. These findings keep open the possibility that the project outputs will have ongoing impact on barriers to market viability.
S.2.2.3 To What Extent Have Funded Projects Reduced Barriers to the Market Viability of Alternative Fuels and Vehicles?
The ARFVTP has made considerable progress reducing many barriers to the market viability of alternative fuels and vehicles. This is shown in Table S.3, which captures our assessment of the impact of the program on barriers for each of the fuel types; our assessment of impact falls into the following four categories:
• Yes: There is solid evidence the projects funded by the program helped reduce this barrier. The extent to which the barrier has been reduced is not differentiated. Major reductions and reductions that are arguably minor are both marked “Yes.”
• Probable: It is likely the project funded by the program helped reduce this barrier, but there is little empirical evidence to support the case.
• Could not be determined: Because there was either insufficient data on project outcomes or the project team was unable to assess and interpret those outcomes, it could not be determined whether the projects reduced this barrier.
• No: There is no indication the projects funded by the program helped reduce the barrier.
The gray shading in the table identifies barriers that were not major when the ARFVTP began, and “not targeted” identifies major barriers that were not targeted by the ARFVTP. Those marked “yes” and those marked “probable” are bolded to make it easier to see them in the table.
For hydrogen, 48 out of the 60 hydrogen-refueling stations in California have been built with ARFVTP funding, and 48 more ARFVTP-funded stations are under development. Although few projects specifically targeted reducing permitting barriers, others may have reduced them through the experience of building stations. The program also had significant impact on the lack of standards for fueling infrastructure. Consumer rebates through the CVRP for the purchase of FCEVs may have reduced the price of such vehicles, but few vehicles were purchased during the period covered by this evaluation, and the magnitude of any such decline is difficult to determine. A few new hydrogen vehicle models were demonstrated with ARFVTP funding, so, as noted in Table S.3, the program helped reduce this barrier; but the impact is probably minor. Hydrogen readiness projects and funding for the CVRP, as well as the retail fueling stations funded by the ARFVTP, probably contributed to improving consumer awareness, but there is no clear quantifiable evidence that a reduction has occurred.
For electricity, the large number of EV charging stations funded by the program is clear evidence that the program has reduced fuel infrastructure barriers, and the EV readiness grants have helped reduced the difficulty of obtaining permits. About 35 percent of the publicly available charging stations in the state appear to have been partly financed by the program. Vehicle barriers have been reduced by awards to build or expand EV production lines and to develop and demonstrate different types of electrically powered vehicles. It is likely that projects to expand production lines and to design and demonstrate advanced EV components will contribute to reducing vehicle cost, but little data by which to confirm such effects are available. The program may have also helped reduce vehicle prices by funding the CVRP. However, the extent of this reduction is hard to determine, because absent the CVRP, automakers may have had to reduce prices by a similar amount to sell the number of vehicles required by the zero-emission vehicle (ZEV) program. Projects have been funded to improve battery performance, but the study team was not able to determine whether these efforts have paid off in terms of improved vehicle performance.
As with the hydrogen projects, EV readiness grants and the increased visibility of charging stations probably contributed to improving consumer awareness, but there is no clear, quantifiable evidence that a reduction has occurred.
Turning to diesel substitutes, the ARFVTP has made demonstrable progress in reducing barriers related to production capacity and bulk fueling infrastructure. Fifty-four million of the 184- million-gallon-per-year (mgy) biodiesel production capacity in California has been partly financed by the ARFVTP, as has 17.5 million of the 63-mgy renewable diesel capacity. The modest number of bulk fueling infrastructure projects funded by the program has helped reduce barriers in this area to some extent. RD&D projects have sought to demonstrate the feasibility of generating low-cost, low–carbon intensity (CI) feedstocks (algal oil) or producing alternative diesel from different types of feedstocks (e.g., dairy waste and FOG in municipal wastewater); however, there is no indication that the projects have taken significant steps toward commercial viability. Projects to expand production capacity and install the latest technology may well have helped reduce the production costs of alternative diesel, but we were not able to collect any systematic data on production costs.
For gasoline substitutes, demonstrable progress has been made in expanding the number of E85 (a blend of 85 percent ethanol and 15 percent gasoline) fueling stations. The number of stations in California increased from 29 to 96 between 2009 and 2016, 40 of which were funded by the ARFVTP. The program has also expanded the production capacity available in California to produce ethanol from sorghum. Progress in reducing the other barriers targeted is less clear. A demonstration project appears to have improved the ability to process cellulosic feedstocks, but the progress was incremental, and it is unclear whether it will become commercial. While there is some evidence that California ethanol producers have replaced at least some of their corn feedstock with sorghum, we have been unable to determine how much. It does not appear that the projects have made progress in reducing the production costs of advanced ethanol, and it could not be determined whether a project to optimize a medium-duty truck engine to run on E85 made progress in improving vehicle performance.
In the case of natural gas and biomethane, ARFVTP-funded projects have approximately doubled the biomethane production capacity in California. Similarly, 42 percent of new liquid natural gas (LNG) stations and 22 percent of new compressed natural gas (CNG) stations in California added since 2008 were funded by ARFVTP projects. The program also provided substantial funding for subsidies to purchase natural gas vehicles (NGVs), which address the vehicle price barrier.
There was also progress in introducing and/or more efficiently using feedstocks for biomethane production and increased public awareness, although the magnitude of these contributions is unclear. Some vehicle technology development projects helped develop and demonstrate new methane-powered engines, although again the impact is unclear. Finally, there is no evidence the program reduced vehicle production cost, except perhaps indirectly through larger sales that can increase production volumes and result in economies of scale.
S.2.2.4 What Has Been the Impact of the Program on Employment and Workforce Capabilities?
CEC’s ARFVTP project grants did support some employment both in California and elsewhere and did result in some hiring for the projects. Many of the hired workers were retained after grant expirations; however, the magnitude of employment and hiring appeared to be modest. It is likely that the ARFVTP-funded projects have addressed some of the ARFVT industry’s needs. However, without more evaluative information, we cannot determine the extent. We do note that most awardees did not report that skilled labor presented a barrier, and most said that it is getting easier over time to find skilled labor for their projects.
S.3 Observations on ARFVTP Investment Priorities…