Public energy research and development (R&D) is recognized as a key policy tool for transforming the world’s energy system in a cost-effective way. However, managing the uncertainty surrounding technological change is a critical challenge for designing robust and cost-effective energy policies. The design of such policies is particularly important if countries are going to both meet the ambitious greenhouse-gas emissions reductions goals set by the Paris Agreement and achieve the required harmonization with the broader set of objectives dictated by the Sustainable Development Goals. The complexity of informing energy technology policy requires, and is producing, a growing collaboration between different academic disciplines and practitioners. Three analytical components have emerged to support the integration of technological uncertainty into energy policy: expert elicitations, integrated assessment models, and decision frameworks. Here we review efforts to incorporate all three approaches to facilitate public energy R&D decision-making under uncertainty. We highlight emerging insights that are robust across elicitations, models, and frameworks, relating to the allocation of public R&D investments, and identify gaps and challenges that remain.
Predicting the future is extremely difficult1, yet it is nonsensical to ignore the knowledge and understanding we can acquire from experts when making decisions. It is well understood that using experts, and using them wisely, is a key component of evidence-based policymaking2.
In the energy sector, policymakers are faced with a number of near-term decisions, such as balancing technology R&D, performance standards, and subsidies; or designing energy technology R&D portfolios. Designing these policies involves managing the significant uncertainty that exists about how technologies will evolve with and without different policies3. In the wake of the December 2015 Paris Agreement, in which countries agreed to put in place national action plans to reduce greenhouse-gas emissions, and in the context of tight government budgets around the world, designing robust and cost-effective energy innovation policies in the face of technological uncertainty has become more pressing4. The question of how to design portfolios of R&D investments across a range of energy technologies has received much attention, most recently after the launch of the Mission Innovation and Breakthrough Energy Coalition pledges to increase public energy R&D funding and follow-on private investments in energy technologies. The R&D decision literature typically approaches this problem in two stages: first considering how R&D investments affect future technology costs5, then considering how future technology costs affect energy and climate policies (for example, EMF28; Ref. 6).
One method for informing policy decisions is to perform expert elicitations (a structured process for eliciting subjective probability distributions from subject-matter experts7). Over the past 10 years, a number of researchers have performed expert elicitations to characterize the uncertainty around future cost and performance of various energy technologies, often subject to assumptions about public R&D investments. This information, however, is just one of the pieces of analysis needed to inform the design of optimal energy R&D portfolios. Supporting R&D policy decisions requires knowledge in a multiplicity of domains, ranging from forecasts for multiple technologies, the economics and engineering of integrating technologies in the energy system, the relationship between economic activities, emissions and the climate, and the role of risk and hedging strategies. For this reason, there has been an effort to integrate the data from expert elicitations about the future of energy technologies into integrated assessment models and decision-making frameworks to provide results that can serve to support energy R&D decisions. Until recently, these three domains, illustrated by the analytical components in Fig. 1, had largely resided in different literatures.
The components in Fig. 1 have been used to inform a range of questions, including both energy infrastructure planning and energy technology R&D policy. Expert elicitations have been used to calibrate deterministic analysis in both climate–economy and energy–economy models (both of which are referred to as IAMs henceforth) through the use of means or medians. (Box 1 includes a description of IAMs and a discussion of the challenges associated with their use.) The greatest value of expert elicitations is that they allow researchers and policymakers to explicitly use probabilistic data to characterize uncertainties in their analysis, permitting the design of policies capable of hedging against particular risks8,9,10. Having an explicit characterization of uncertainty is particularly important when the problem or question under consideration includes non-linearities. Non-linearity may characterize preferences, as in the case of risk aversion, where a decision-maker is willing to give up something, such as money or enjoyment, to avoid some level of risk. Such preferences are represented by concave utility functions in economic models11. Non-linearities can also reside in the underlying problem itself. In the case of energy R&D investments, there may be non-linearities in the impact of R&D investments on energy technology costs, in the competition and complementarity among technologies in the market, and in irreversibilities or tipping points that might characterize environmental damages. A third type of non-linearity emerges when decision-makers can make additional decisions in the future once uncertain outcomes have been revealed. As has been demonstrated12, when future options are available, near-term decisions that avoid lock-ins or irreversible effects are more welfare-enhancing than those that naively maximize the expected net present value. Collectively, these non-linearities make it important to consider the full distribution of outcomes from decisions, including the tails and not just average values. This applies to multiple types of energy planning problems; the literature, however, has focused this kind of analysis on R&D problems. While in some cases the full stochastic treatment does not yield significantly different results from a simple best guess13,
An important distinction we make in Fig. 1 is to classify integrated uncertainty modelling efforts into Learn-then-Act (comprising sensitivity analysis and uncertainty analysis) and Act–Learn–Act (which are often simply called Act-then-Learn in the literature and comprise one-stage and multi-stage decision-making under uncertainty, DMUU). Learn-then-Act frameworks combine elicitations with IAMs; Act–Learn–Act frameworks also integrate decision theory models (sometimes within IAMs themselves18, sometimes in a multi-model framework8).
The differences between Learn-then-Act and Act–Learn–Act frameworks are related to when decisions take place in relation to when uncertainty is resolved. The key uncertainty in the papers we review is technological outcomes. Learn-then-Act frameworks take near-term actions as given (in this case a particular level of public energy R&D investment), and explicitly model decisions which take place only after uncertainty has been realized (deployment decisions which take place within IAMs). Act–Learn–Act frameworks are more realistic, explicitly modelling near-term decisions that must be made before uncertainty is resolved; the full probability distribution is taken into account when choosing an alternative. In all of the papers reviewed, the near-term decision is investment in public energy R&D. The remaining later-stage decisions, including deployment, take place within the IAMs.
In this Review, we discuss work combining expert elicitations on energy technologies with IAMs and frameworks for decision-making under uncertainty to support public energy technology R&D policy decisions in the face of technological uncertainty. The insights stemming from the types of integrated quantitative analysis depicted in Fig. 1 and described in this Review are, of course, just one input to public policy. There are many other factors, such as politics, stakeholder values, and the interactions between policies in the energy sector and other sectors that require government investment, to name a few. Nevertheless, insights about optimal public energy R&D portfolios arising from integrated quantitative analysis can usefully support decisions by clarifying the presence of non-linearities and by providing clarity about assumptions, goals and trade-offs of different investments. The aims of this Review are three-fold: to summarize and categorize recent research incorporating expert elicitations and IAMs in decision-making frameworks; to highlight the most relevant insights coming from this literature for the design of public energy R&D portfolios; and to identify gaps and key research areas going forward.
Expert insights about the future of energy technologies
Before delving into the integrated approach presented in Fig. 1, we review the literature on energy technology expert elicitations.
Verdolini et al.19 provide a review of 29 expert elicitation studies performed between 2007 and 2012, covering eleven energy technology categories. The range of studies reviewed there (plus two additional recent wind studies, making a total of 31 papers) are summarized in Table 1 and the key challenges and gaps associated with energy technology expert elicitations are summarized in Box 2. It is important to note that none of the elicitations explored alternative scenarios of private-sector R&D investment due to a large extent to the lack of information regarding current private energy R&D investments by technology.
The analysis of 31 papers included in Table 1 (Refs 3,19) provides robust evidence of the significant uncertainty surrounding future energy technology costs and performance. The uncertainty characterized by elicitations goes beyond the uncertainty estimated from forecasting methods based on historical data (for example, learning or experience curves, as a function of deployment20 or of time19). For example, using 20-year windows of historical data on solar photovoltaic (PV) module costs from Fraunhofer21, the average annual cost reduction ranges between 2.7% and 4.6%, depending on the start date of the window; the average annual cost reduction over a 25-year period (1980–2015) is 2.8%. For comparison, using individual expert answers from the Harvard solar PV expert elicitation22 on module costs, implied average annual cost reductions between 2010 and 2030 show a much wider range, from 0.1% to 4.7%. This range of uncertainty does not seem to depend on the level of R&D investment5,23. It is also worth noting that most of the elicitations sought to isolate the impact of R&D, recognizing that there are factors beyond R&D, such as input prices, learning-by-doing and economies of scale that have driven previous cost reductions. This has, for example, been demonstrated for solar PV20.
According to the studies in Table 1 and the related discussion in Box 2, on average, with significant variation across technologies, experts expect that higher levels of R&D would lead to cost reductions of over 50% between 2010 and 2030. Looking at the tails of the distributions, however, it is possible to detect non-zero probabilities of more significant breakthroughs leading to sustained cost reductions on the order of 10% per year until 20303. Examining only the medians or means would result in ignoring low-probability, high-impact events in decision-making. Among the studies, results vary by technology in terms of forecast rates of cost improvements; however, the uncertainty within and between studies dominates.
When comparing the rate of change in the future cost of technologies implied by the expert elicitation studies of projections based on historical data, it was found19 that the technologies differ in the degree to which the experts see the future as being similar to the past. For example, the median expert forecasts for nuclear power and biofuels are similar to historical data, while expert forecasts for bioelectricity are relatively pessimistic. This is consistent with the possibility that expert judgments represent a semi-independent source of information: experts are influenced by historical trends as well as by their personal experiences and beliefs about future technological developments.
There are many challenges and gaps associated with energy technology expert elicitations (see Box 2). Nevertheless, elicitations provide a systematic way, using the best available knowledge, to provide information on future technological change unavailable from other types of analyses.
Modelling and managing technological uncertainty
It is not enough to have an understanding of the uncertainty surrounding the future cost and performance of energy technologies. The design of efficient and cost-effective energy policies requires understanding the broader social impact of changes in these technologies, including the impact of interactions across technologies and between R&D and other policies. A large body of literature has explored the importance of technological costs on future mitigation pathways under uncertainty without using inputs from expert elicitations (see for example refs 24,
Table 2 classifies modelling efforts into Learn-then-Act and Act–Learn–Act, categories introduced in Fig. 1. Sensitivity analysis, which takes the energy R&D investment decision as given, addresses technological uncertainty by experimenting over a range of technological scenarios. The result of sensitivity analysis is a range over the standard outputs of IAMs (for example, investment levels and climate mitigation costs). Most sensitivity analysis efforts do not rely on expert elicitations, but Table 1 includes some sensitivity analysis studies that do.
Uncertainty analyses also take the energy R&D investment decision as given, but move a step further, using probability distributions of future technology costs and performances as inputs, and providing outputs from IAMs as probability distributions. The most well-known method of uncertainty analysis is Monte Carlo analysis.
The last two columns of Table 1 move into the realm of DMUU, which provides insight into near-term strategies given the current state of information. In the 1-stage DMUU approach, as illustrated in Fig. 1, key decisions (in this case R&D investment decisions) are made before uncertainty is resolved. Multi-stage DMUU approaches, on the other hand, allow for recourse — key decisions are revisited after uncertainty is revealed, including future climate policies or results from the previous energy R&D investments. Thus, multi-stage DMUU models can identify near-term decisions or policies with option value12. Among multi-stage DMUU frameworks, it is most common to have two stages, but some models include a larger number of downstream decisions. The two most common methods for implementing multi-stage models are dynamic programming27 and stochastic programming. Dynamic programming is increasingly being implemented with high-dimensionality solution techniques, such as approximate dynamic programming (ADP)28. In general, computational challenges are difficult in all multi-stage frameworks due to the curse of dimensionality: while stochastic programming needs to manage the number of future uncertain states, ADP models need to manage the number of state variables. Some have argued that the addition of decision stages beyond two may not provide considerable insight25,29. In all DMUU studies here, IAM-based deployment decisions take place after uncertainty is resolved. DMUU differs from sensitivity and uncertainty analysis, as the uncertainty explicitly feeds back into the near-term decision.
Sensitivity to energy technology costs. While expert elicitations have been used to calibrate IAMs, this does not fully utilize the probabilistic data assembled in expert elicitations. In Table 1, we include sensitivity analysis studies that use the range of values elicited from experts to analyse the impact of technology scenarios on model outputs. Typically, sensitivity analyses vary technology input parameters one at a time to assess the range of societal outcomes associated with extreme technology assumptions. This helps to identify critical uncertainties and to bound results. Pugh et al.30 performed an informal expert elicitation over various technical outcomes across a wide range of technologies, and used the results to assess emissions reductions and return on investment. Ricci et al.31 and Iyer et al.32 each perform a simple sensitivity analysis to bound possible future costs and efficiency of a single technology (carbon capture and storage (CCS) and nuclear, respectively) and its economic implications. Barron and McJeon33 investigate how much technologies would need to improve to significantly reduce the cost of climate-change mitigation. Olaleye and Baker34 perform a large scale, multi-parameter scenario analysis to identify technologies that are complements or substitutes at a societal level.
In recent years, there has been an effort to go beyond conventional sensitivity analysis by allowing various input parameters to change together, thus capturing interaction effects. Such approaches can use elicited probability distributions for random draws35 and are referred to as global sensitivity analysis. Bosetti et al.36 perform a multi-model global sensitivity analysis of energy technology costs, drawing on three elicitation studies over five technologies. Their results, robust to the choice of the IAM, indicate that when there is no climate policy, among the technologies considered, nuclear cost assumptions are the biggest technology cost determinants of projected societal energy costs and baseline emissions, complementing findings about the role of nuclear with climate policy37. Sensitivity analysis is more computationally tractable than the methods we discuss later, and is appropriate for identifying areas of interest that could be the subject of additional research. It has descriptive value and its results can be easily presented to policymakers or other users, usually in the form of scenarios representing particular narratives. Sensitivity analysis represents a first step towards understanding the resilience of a particular policy. It has the weakness, however, of not taking into account many of the non-linearities mentioned above. Thus, researchers and analysts should consider going beyond sensitivity analysis, even if this presents challenges to communication, because “decision-makers are often far better served by examining the diversity of opinion”38 and thinking about the robustness of policy approaches to the uncertainty.
Towards understanding uncertainty. Expert elicitation data can also be used in uncertainty analysis, most commonly Monte-Carlo analyses. Monte-Carlo analyses use random draws from probability distributions as inputs to models, such as IAMs, resulting in probability distributions over the outputs of interest. Gillenwater39 uses an elicitation on wind to estimate the effect of policies on the probability of private sector deployment investment. A similar, but more sophisticated method, Latin Hypercube Sampling (LHS)40, can reduce the number of needed simulations while preserving the probabilistic interpretation of the model outputs. Chan and Anadon41 account for the dependency between R&D-induced energy technology improvements across a set of 25 technologies (some of which are interrelated) using LHS. The results include probability distributions of future CO2 emissions, oil imports, consumer and producer surplus and other metrics under different R&D scenarios, with and without specific demand-side policies.
Another method of uncertainty analysis is to propagate probability distributions through a model. Olaleye42 uses large-scale scenario analysis to estimate probability distributions over welfare for different R&D portfolios, by associating each technological outcome with a conditional probability derived from elicitations. Lemoine and McJeon43 combine energy technology elicitation results with an IAM to investigate the probability distributions of climate damages and mitigation costs. Several elicitation studies use deterministic runs of an IAM combined with elicited probabilities of particular technology costs to derive conditional probability distributions over the impacts of R&D on the marginal abatement cost curve, with a focus on the different ways in which different technologies impact the curve44,
Nemet and Baker47 combine the results of a solar expert elicitation with a techno–economic model that allows them to derive probability distributions over the cost of solar conditional on both technology R&D and technology subsidies. Similarly, a set of interrelated papers propagate elicited distributions of the outcomes of R&D in CCS technology through different techno-economic models to produce probability distributions of the future technology costs and overall CO2-avoidance for various R&D investment and deployment policies48,
Uncertainty analysis forces researchers and policymakers to think about probabilities, with the added benefits (and added complexities) that this entails51. However, similarly to sensitivity analysis, uncertainty analysis by itself does not directly inform near-term decision-making, such as the optimal amount and allocation of R&D investments.
Modelling insights from DMUU frameworks. The studies presented in Table 3 and discussed in Box 3 show that there are trade-offs involved in selecting DMUU strategies, involving the number of technologies that are explicitly modelled as uncertain, the number of decision stages, the representation of R&D, and use of reduced-form decision models. Baker and Solak52 consider only three technologies, but solve a technologically-detailed stochastic programming version of the IAM DICE, where investments into R&D impact the marginal abatement cost curve and are the first stage decision. All papers with 4 or more technologies feed outputs from IAMs into reduced-form decision models. Chan and Anadon41 include continuous public R&D levels for 6 technology areas that affect the cost and performance of 25 individual technologies, but only one-stage DMUU. Six papers in Table 3 use two-stage DMUU8,42,53,
It is important for researchers and users of information to be aware of the crucial design decisions being made when conducting this analysis and to understand what trade-offs are being made and why. Key design decisions are the number of technologies that are modelled with R&D dependent uncertainty, the number of decision stages, whether or not R&D funding is continuous, and the decision framework.
Another question for researchers and policymakers is whether one piece of the analytical framework depicted in Fig. 1 is driving the results. Is it enough to look at the results of expert elicitations to determine which R&D investments are best to achieve a particular policy goal? Or, on the other hand, do the built-in assumptions of IAMs swamp the assumptions stemming from expert elicitations? We find evidence that both components influence results, along with the decision framework. A comparative analysis8 reported in Fig. 2 shows that the expert elicitation data matters: the IAM and decision frameworks were identical yet the R&D portfolio results vary by elicitations. Fig. 3 shows that the inclusion of more technologies (for example, some studies include vehicles and some do not) also has an impact on results about optimal energy R&D portfolios. Figures 2 and 4 both show that the decision framework matters. Comparing the results of Baker et al.8 who use the UMass elicitations with the McJeon study53 in Fig. 2 shows that while both studies employ the same IAM and elicitation data, they have different results. Santen and Anadon17 also find that the decision framework matters: the results presented in Fig. 4 show that deterministic, Monte-Carlo, and 5-stage DMUU lead to different levels of R&D versus technology deployment investments. Finally, Barron57 shows that the choice of the IAM matters, with GCAM and MESSAGE providing different optimal R&D portfolios using the same elicitations and decision frameworks.
One important dimension that emerges when thinking about designing optimal energy R&D portfolios is to what extent they depend on the context, including other policies or policy goals. In particular, climate policy stringency and the availability of resources for R&D investments and for deployment of energy technologies are key factors prevalent in policy debates in the energy sector.
We now review the insights for policy that emerge from the literature introduced above. We organize these insights into three areas: the impact of climate change targets on optimal R&D portfolios; the impact of R&D budget constraints on the allocation of R&D investments; and a comparison between funding allocated to public R&D and to technology deployment.
Optimal R&D portfolios under different climate targets. Figure 2 summarizes a range of studies investigating the extent to which different climate constraints affect optimal R&D portfolios. Results are organized into groups with common elicitations, IAMs, and decision methods. Of the six groups analysed, which come from 4 major studies41,53,56,59, four groups (Groups 1–4) compare a baseline ‘no policy’ scenario with either a lenient and stringent climate stabilization scenario or with just a stringent climate stabilization scenario (the Fig. 2 caption provides information about the specific studies), Group 5 compares a lenient and a stringent climate stabilization scenario and Group 6 presents the optimal R&D allocation under a lenient climate stabilization policy. Although the climate scenarios are not entirely comparable, we group them into the broad categories of stringent and lenient climate target scenarios; details and assumptions are reported in Table 3.
There are five technology-specific results that emerge from the analysis presented in Fig. 2. First, solar gets the smallest share of R&D and its role slightly decreases as the target gets more stringent. This is mainly due to the different magnitude of solar R&D investments compared to the other type of technologies. Second, the stricter the climate constraint, the greater the share of CCS in energy R&D portfolios, with the exception of one study8 using the UMass elicitations. CCS is critical for negative emissions (bioenergy plus CCS), which in turn is an increasingly important strategy for stricter climate targets. Third, stricter climate constraints lead to an equal or smaller share of nuclear in energy R&D portfolios, with the exception of one study8 using elicitations from Ref. 22. This appears to be due to large nuclear investments under low-stringency targets, with less room to grow as targets get more stringent. Fourth, all studies that include vehicles find that a significant share of the overall budget is devoted to advanced vehicles R&D, ranging from 30–80% of the total R&D investments; this fraction increases with stricter climate constraints. Fifth, R&D funding for utility-scale energy storage, a technology area included in only one study, increases with stricter climate constraints.
It is important to note that this is a summary of emerging insights, and many gaps and challenges exist, such as model biases and missing or poorly modelled technologies, making it difficult to extract recommendations about the precise level of R&D investments into specific technologies. However, one general conclusion does appear: technologies which provide flexibility see higher R&D investments as climate stringency increases. CCS allows for ex post emission decreases from fossil plants; storage allows higher penetration of intermittent technologies; and vehicles allow abatement to reach into the transportation sector. The fact that we do not see this pattern for biofuels reflects elicitation results, particularly in the US studies, which may in turn reflect high existing levels of R&D.
Optimal R&D portfolios for different R&D budget constraints. Another pressing question in the minds of policymakers, particularly in times of financial crisis, is how R&D investment portfolios should change for different public energy R&D budget constraints. Figure 3 summarizes the results of seven papers22,41,42,52,54,56,58 that consider multiple public energy R&D budget constraints.
Results show that with tighter R&D budgets, portfolios become less diverse, with some technologies not funded at all. Among the four studies that include vehicles22,41,42,53 there is a decrease in the share of vehicles R&D with greater R&D budgets. This is likely because vehicles R&D (in particular batteries) get a large share to begin with in constrained budgets; and because the marginal improvement seen in the elicitations above and beyond those levels is smaller than that of other technologies, especially CCS. Among the three studies that include bioenergy41,42,56 the fraction of R&D devoted to total bioenergy (the combination of biofuels and bioelectricity) stays relatively constant, driven to some extent by assumptions about the availability and cost of biomass.
Solar plays an interesting role in these portfolios. First, we note that, as the R&D budget constraint is relaxed, and consequently the number of technologies considered grows, the share of R&D for solar decreases precipitously (more so than for nuclear, CCS, or vehicles). Moreover, the share of solar is non-monotonic in a number of studies, more prominently than any of the other technologies. This may indicate that solar plays the role of a kind of ‘filler’ technology in these portfolios. The problem of finding an optimal portfolio subject to a constraint is what is known as a ‘knapsack problem’. It is well known that knapsack problems can lead to non-monotonic results as the constraint (in this case the total energy R&D budget) relaxes. Solar R&D plays this role since, especially in comparison to CCS and nuclear, it can be productive even at lab scales. On the other hand, in the absence of cost-effective storage technologies, solar PV has a limited impact on the overall economy, due to structural assumptions about intermittency in many of the IAMs.
The main disagreement in the results in Fig. 3 is for the fraction of R&D devoted to nuclear versus CCS. In two of the papers42,54 nuclear increases at the expense of CCS as the R&D budget grows; while for the other 4 papers41,52,53,58 that include both technologies this trend is reversed. Nuclear and CCS tend to be strong substitutes; thus, depending on the specific IAM assumptions, different technologies take a larger share. Both of these technologies are associated with significant socio-technical issues that are difficult to model, such as public acceptance. Thus, factors embedded in IAMs beyond future cost (for example, limits on the rate of deployment) are likely to play a role on whether one or the other technology receives the largest share of R&D portfolios.
We can also ask to what degree R&D expenditures are justified: what amount is the right amount? This question has been asked in the theoretical literature using hypothetical probability distributions or the comparative statics of risk (see Ref. 60 for a review). Some of the papers reviewed here have taken on this question using elicitation-based distributions. McJeon53 finds that the savings in abatement costs are 2–3 orders of magnitude higher than the R&D investment costs for all portfolios considered; Anadon, Chan, and Lee22 find that current US public energy R&D budgets could be increased by at least an order of magnitude and still be cost-effective in terms of consumer and producer surplus; there are similar results in other papers, especially when opportunity costs are low, potential benefits are high, and a limited set of projects is available. Most papers, however, find that the optimal investment in R&D includes only a subset of the total available projects. Thus, the opportunity cost of R&D investment does play a role. A few papers have found that R&D is more valuable in second-best worlds8,52,61; that under-investment may be more costly than over-investment52; and that the optimal investment decreases with increasing discount rates and opportunity costs, and appears to be non-monotonic in risk, first increasing then decreasing as risk increases52,54.
R&D versus deployment expenditures. A key debate in innovation and climate policy is the question of how to balance public support for R&D as opposed to technology deployment. A recent study62 highlighted this as one of the hardest questions in crafting comprehensive energy policy. In Fig. 4 we highlight the ratio of (mostly public sector) R&D investments compared to the long term (mostly private sector) investments in deployment resulting in some of the elicitation-based studies. Fig. 4 shows the percentage of R&D investment to the total investment (including R&D and all deployment costs, such as capacity investments and government subsides) between 2010 and 2040; this proportion ranges between 0.1–9.2%.
While the innovation literature has highlighted the need for both technology-push and market-pull technology policies63, and the energy economics literature has discussed the need for two instruments to address two market failures of environmental and knowledge externalities64, we are not aware of any attempt to obtain an estimate of what the optimal range of public R&D is when compared to total deployment costs, given our current knowledge. While the range, including uncertainty, spans over one order of magnitude, no study found that the fraction of R&D should be more than 9.2% of the total investment, with the median result around 3.5%. For reference, this ratio for solar PV in the United States in 2015 was roughly 1.1%. This estimate was calculated65 using a federal R&D investment in solar PV in 2015 of US$231 million, and a rough deployment investment of US$20.6bn, estimated using a value of total installed solar capacity of 7.3 GW, split between 4.3 GW of utility-scale solar and 3 GW of non-utility-scale solar66, and assuming average solar costs in 2015 of 2 US$ W–1 for utility-scale solar and 4 US$ W–1 for non-utility-scale solar67. The question of R&D versus deployment is a ripe area for future research, since only limited studies were available for this analysis.
In addition to shedding light on these three areas, the existing literature also includes a few studies that investigate different questions. Ziolkowska68 combines a biomass and bioenergy elicitation with a multi-objective decision framework, considering economic, environmental and social policy objectives in evaluating different biomass feedstocks to inform energy policy. Bistline61 investigates the optimal electricity capacity investment portfolio under uncertainty about technology prices derived from elicitations. Using a two-stage stochastic programme, the paper finds that natural gas prices and the stringency of climate policy are the main risks associated with capacity expansion planning.
Gaps and challenges in combining the three approaches
We have discussed the insights that can be gained about public energy technology R&D policy through an integrated analytic approach that combines expert elicitation, IAMs, and DMUU frameworks. We now address some gaps in knowledge and methodology and potential challenges to this combined approach.
First, the combined approach is necessarily limited by the availability of expert elicitations and the limitations of IAMs. The set of technologies that can be included in a combined framework is limited to those for which elicitations have been performed and which IAMs represent in some detail. In addition, the representation of the energy system in IAMs can strongly influence results. For example, the result that solar PV plays a small role in many R&D portfolios may be related to how IAMs model the challenges of grid integration. Recent work69 has revisited the representation of wind and solar across multiple models, leading to higher shares of these technologies in more recent versions of the models. It will be important to test expert elicitation data on this new generation of IAMs. Beyond these gaps, the limitations of IAMs (Box 1) and expert elicitations (Box 2) affect the combined analysis, and thus must be considered when interpreting results.
A second issue relates more generally to all optimization frameworks used in policy assessments: how, and by whom, is the objective function defined? The papers reviewed here are primarily aimed at a high level, relevant to national governments or international agencies. However, much decision-making takes place at lower levels, such as sub-national agencies. Moreover, it has long been acknowledged that the problem of identifying an objective function, even in the case of a single decision-maker, is quite challenging70. This becomes even more challenging when the problem is in the public realm and consists of multiple policy goals71, stakeholders72, and long timeframes. The role of optimization and decision frameworks in this case is not to provide a single optimal solution, but rather to shed light on the impact of decisions, uncertainty, and preferences in a quantitative framework; ultimately, decision-makers are faced with value judgments over particular optimization criteria. There are considerable gaps in the objectives that have been explored by this approach in the literature so far. Most of the papers employing DMUU either minimize the cost of achieving a climate goal (such as a particular stabilization scenario) or minimize the combined cost of abatement and damages. A few papers explore other objectives, such as maximizing consumer and producer surplus, reducing carbon price, or reducing CO2 emissions in the US22. Ziolkowska68 uses a multi-criteria approach, including criteria such as economic efficiency, reducing GHGs and water use, and supporting local communities. The wide range of possible policy objectives (which might vary based on the geographical and socio-economic context of the decision-maker) has yet to be explored using these frameworks. Some objectives of interest may be general economic development, reduction of inequality or poverty, energy access, and many others. To truly address this gap, researchers must collaborate with policymakers to support decisions in specific contexts. Nevertheless, there is the danger of ‘lamp-posting’ — of only addressing questions that current models are set up to answer.
A third area that could help address some of the limitations of the combined approach is to perform structured multi-model comparisons; including sets of IAMs with different assumptions about technology diffusion, built-in objectives, and decision frameworks, as well as multiple elicitation studies to parameterize technological uncertainty performed using different methods. Structured multi-model comparisons have shown great promise in increasing the robustness of insights in the context of IAMs and the Energy Modeling Forum (EMF) studies6,69,73, and could help increase the robustness of existing insights from the integrated approach presented here, or provide new policy insights.
A fourth area in need of additional attention by the research community is that of understanding the private sector response to government policy, both in terms of R&D investments and the challenges to the commercialization of technologies74, and on the interaction between innovation in different technologies through spillovers. While some headway has been made using patent studies75, little is known about private R&D spending, its effectiveness, and its relation to public R&D and other policies76. Thus, this important piece of the puzzle often gets left out of technology policy analyses. Similarly, the challenges related to spillovers — between locations and between energy R&D and other areas — are not included in the reviewed papers due to the limited availability of data and analysis. A particular concern is a lack of representation of R&D in emerging economies, and related international spillovers. Most broadly, it is important to note that these important mechanisms, including private R&D response and first phase deployment challenges, have not been considered in expert elicitations nor are they well-represented in IAMs. These mechanisms have an important role, and thus are ripe for more research.
Current decision-making on public energy R&D investments is generally opaque: it is unclear to external analysts, stakeholders, or taxpayers where assumptions about future costs and R&D effectiveness come from, and what goal or goals are being prioritized22. The type of analytical approach presented here can serve as a complement to inform and scrutinize public decisions, since the source of the technology assumptions (the names of the experts) is clear, and the models and optimization criteria are specified. However, this approach is only useful to the degree that policymakers buy in to the results and insights. This is a major challenge, not just for this combined framework, but for all approaches that explicitly include uncertainty in the analysis. And thus, a fifth and final area for future work is in uncertainty communication, including two-way communication with policymakers through a process of research codesign, so that frameworks, such as those we have reviewed here, are informed by both the insights of policymakers and responsiveness to their needs.
Governments are increasingly turning their attention to questions about energy technology R&D portfolios and energy innovation policy. The US. Department of Energy recently created the Office of Technology Transitions, responsible for developing and overseeing delivery of strategic vision77 and goals around technology commercialization, including considerations about R&D portfolios. Various global efforts are emerging aimed at improving energy technologies, including Mission Innovation, the Breakthrough Energy Coalition, and the United Nations Framework Convention on Climate Change Green Climate Fund.
Decisions about how to best support innovation in energy technologies are hard for a number of reasons, but among the most important is the deep uncertainty about the outcomes of R&D and other policies, the interactions of technologies, and the science of climate change. The approach we review in this paper — integrating expert elicitations, IAMs, and decision frameworks — is complex. Yet the transition pathway ahead of us demands radical changes in policies and in the economy, as well as strategies to enable such radical changes, in all their complexities. Thus, governments have an imperative to acquire, integrate, and use the best available information in order to support this important decision-making process. Many governments are developing capabilities in IAMs, allowing them to use this set of tools to inform decisions (for example, the US Department of Energy78, and the UK Department of Business, Energy and Industrial Strategy79,80). It would be worthwhile for governments to develop capabilities similar to those presented in this paper to understand the full ramifications of expert elicitations alongside other data and DMUU frameworks81.
Of course, expert elicitations, IAMs, and decision frameworks, while crucial, are themselves only part of the full range of available information and analytic tools that can be used to inform policy decisions. An open question for the future is how to go beyond the already complex and integrative approach presented here, as additional data becomes available. This additional information could come from sources as diverse as prediction markets, increased computing power, and hybrid data–human-based forecasting methods. The challenge rests in incorporating this body of information while all the while maintaining a firm control on the various components of the analysis and ensuring a rigorous validation of the whole process, through methods such as repeated elicitations of experts, controlled comparison of elicitation methods, open-source code for integrated assessment models and frameworks, randomized control trials, and open data sets. Nonetheless, we hope this Review has shown that expert knowledge reflecting uncertainty can be profitably integrated with sophisticated modelling methods to inform energy technology R&D policy.
How to cite this article: Anadón, L. D., Baker, E. & Bosetti, V. Integrating uncertainty into public energy research and development decisions. Nat. Energy 2, 17071 (2017).
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
We are grateful to Nidhi Santen for useful insights and feedback on the description on ADP, to Franklyn Kanyako for assistance with the figures, and to Patricia McLaughlin for her work proofreading it. L.D.A. is grateful to the Climate Change Initiative of the Doris Duke Charitable Foundation and the Science, Technology and Public Policy programme at the Harvard Kennedy School for supporting early work on this paper and to the EU H2020 research and innovation programme under the Grant Agreement No 730403 (INNOPATHS). V.B. is grateful to funding from the European Research Council under the European Community’s Programme ‘Ideas’ — Call identifier: ERC-2013-StG / ERC grant agreement number 336703 — project RISICO ‘RISk and uncertainty in developing and Implementing Climate change pOlicies’. We also acknowledge financial support from the United Nations Environment Program through the Green Growth Knowledge Platform (GGKP) Research Committee on Technology and Innovation for the preparation of GGKP Working paper 01-2016, a report that helped us identify the need for this Review. E.B. acknowledges the NSF-sponsored IGERT: Offshore Wind Energy Engineering, Environmental Science, and Policy (NSF DGE 1068864).