Review Article

Integrating uncertainty into public energy research and development decisions

  • Nature Energy volume 2, Article number: 17071 (2017)
  • doi:10.1038/nenergy.2017.71
  • Download Citation
Published online:


Public energy research and development (R&D) is recognized as a key policy tool for transforming the world’s energy system in a cost-effective way. However, managing the uncertainty surrounding technological change is a critical challenge for designing robust and cost-effective energy policies. The design of such policies is particularly important if countries are going to both meet the ambitious greenhouse-gas emissions reductions goals set by the Paris Agreement and achieve the required harmonization with the broader set of objectives dictated by the Sustainable Development Goals. The complexity of informing energy technology policy requires, and is producing, a growing collaboration between different academic disciplines and practitioners. Three analytical components have emerged to support the integration of technological uncertainty into energy policy: expert elicitations, integrated assessment models, and decision frameworks. Here we review efforts to incorporate all three approaches to facilitate public energy R&D decision-making under uncertainty. We highlight emerging insights that are robust across elicitations, models, and frameworks, relating to the allocation of public R&D investments, and identify gaps and challenges that remain.

Correction online 22 November 2017

Predicting the future is extremely difficult1, yet it is nonsensical to ignore the knowledge and understanding we can acquire from experts when making decisions. It is well understood that using experts, and using them wisely, is a key component of evidence-based policymaking2.

In the energy sector, policymakers are faced with a number of near-term decisions, such as balancing technology R&D, performance standards, and subsidies; or designing energy technology R&D portfolios. Designing these policies involves managing the significant uncertainty that exists about how technologies will evolve with and without different policies3. In the wake of the December 2015 Paris Agreement, in which countries agreed to put in place national action plans to reduce greenhouse-gas emissions, and in the context of tight government budgets around the world, designing robust and cost-effective energy innovation policies in the face of technological uncertainty has become more pressing4. The question of how to design portfolios of R&D investments across a range of energy technologies has received much attention, most recently after the launch of the Mission Innovation and Breakthrough Energy Coalition pledges to increase public energy R&D funding and follow-on private investments in energy technologies. The R&D decision literature typically approaches this problem in two stages: first considering how R&D investments affect future technology costs5, then considering how future technology costs affect energy and climate policies (for example, EMF28; Ref. 6).

One method for informing policy decisions is to perform expert elicitations (a structured process for eliciting subjective probability distributions from subject-matter experts7). Over the past 10 years, a number of researchers have performed expert elicitations to characterize the uncertainty around future cost and performance of various energy technologies, often subject to assumptions about public R&D investments. This information, however, is just one of the pieces of analysis needed to inform the design of optimal energy R&D portfolios. Supporting R&D policy decisions requires knowledge in a multiplicity of domains, ranging from forecasts for multiple technologies, the economics and engineering of integrating technologies in the energy system, the relationship between economic activities, emissions and the climate, and the role of risk and hedging strategies. For this reason, there has been an effort to integrate the data from expert elicitations about the future of energy technologies into integrated assessment models and decision-making frameworks to provide results that can serve to support energy R&D decisions. Until recently, these three domains, illustrated by the analytical components in Fig. 1, had largely resided in different literatures.

Figure 1: Analytic approach to supporting energy R&D decisions using expert elicitations.
Figure 1

a, Analytical components in this Review are expert elicitations and IAMs, illustrated here by the two different schematic diagrams. b, Two main types of decision-making under uncertainty (DMUU) frameworks to support policy are illustrated here: Learn-then-Act frameworks, in which we include sensitivity and uncertainty analysis; and Act-then-Learn-then-Act (Act–Learn–Act) frameworks, where we include 1-stage and multi-stage DMUU. c, Illustrated here are the types of outputs that emerge from the different types of analysis: deterministic scenarios of future CO2 emissions under different R&D levels, probabilistic outcomes of future CO2 emissions under different R&D levels, and optimal energy R&D portfolios for 3 technologies under two different climate policies in terms of the level of stringency. More details can be found in the text.

The components in Fig. 1 have been used to inform a range of questions, including both energy infrastructure planning and energy technology R&D policy. Expert elicitations have been used to calibrate deterministic analysis in both climate–economy and energy–economy models (both of which are referred to as IAMs henceforth) through the use of means or medians. (Box 1 includes a description of IAMs and a discussion of the challenges associated with their use.) The greatest value of expert elicitations is that they allow researchers and policymakers to explicitly use probabilistic data to characterize uncertainties in their analysis, permitting the design of policies capable of hedging against particular risks8,9,10. Having an explicit characterization of uncertainty is particularly important when the problem or question under consideration includes non-linearities. Non-linearity may characterize preferences, as in the case of risk aversion, where a decision-maker is willing to give up something, such as money or enjoyment, to avoid some level of risk. Such preferences are represented by concave utility functions in economic models11. Non-linearities can also reside in the underlying problem itself. In the case of energy R&D investments, there may be non-linearities in the impact of R&D investments on energy technology costs, in the competition and complementarity among technologies in the market, and in irreversibilities or tipping points that might characterize environmental damages. A third type of non-linearity emerges when decision-makers can make additional decisions in the future once uncertain outcomes have been revealed. As has been demonstrated12, when future options are available, near-term decisions that avoid lock-ins or irreversible effects are more welfare-enhancing than those that naively maximize the expected net present value. Collectively, these non-linearities make it important to consider the full distribution of outcomes from decisions, including the tails and not just average values. This applies to multiple types of energy planning problems; the literature, however, has focused this kind of analysis on R&D problems. While in some cases the full stochastic treatment does not yield significantly different results from a simple best guess13,​14,​15, at other times16,17 it does; unfortunately it is difficult to know a priori when it is important.

Box 1: Integrated assessment models.

In this work we use the term IAMs to jointly refer to energy–economy models (EEMs) and climate–economy models (CEMs), which are described here.

Energy–economy models. Typically bottom-up models that represent the energy sector with a fine level of detail on individual technologies. They may represent a particular geographic area or region (although EEMs covering the whole globe exist) and they are typically used to estimate how changes in input parameters representing resources prices, technologies characteristics, climate or energy policies, and energy demand all shape energy use, investments in the energy system, and the energy system’s impacts on environmental or economic metrics. Technologies are deployed on a least-cost basis, given assumptions about the rate at which the lowest cost options can penetrate the market. EEMs also do not require assumptions about the economic cost of environmental impacts (for example, climate damages) or impacts that the energy sectors may have on other sectors, such as trade. These simplifications come at the cost of not representing interactions between the energy sector and other sectors in the economy that may be affected by fuel and power prices; that is, EEMs do not allow modelers to calculate the impact on gross domestic product of various factors. An example of a widely used EEM by policymakers and analysts in academia and the non-governmental organization community is MARKAL107.

Climate–economy models. These integrate different scientific domains in a unique framework in order to model the implications and feedbacks between systems108 (a common example being the energy system, the climate system and the earth system). CEMs may represent the energy sector at different levels of detail (typically lower than energy system models) but they also represent the interaction between this sector and other economics sectors (for example, oil extraction) or even other systems (for example, forests). CEMs, similarly to EEMs provide to the climate modelling community emissions scenarios of greenhouse gases, and to the impacts community projections of socio-economic states. CEMs are used to study the implication of climate policies on the energy systems, given assumptions about technology costs and penetration potentials and are conditional on other socio-economic assumptions. A subset of CEMs model the feedback effects of climate impacts on the economy, thus studying the optimal level of admissible climate change, given the balance of mitigation costs and benefits from avoided climate change. As large uncertainties affect this feedback link, several criticisms have been made of CEMs employed to perform this type of analysis109. However, most of the analyses presented in this Review abstract from the climate damage feedback and assume a given climate policy, rather than endogenously calculating it. Some prevalent CEMs include DICE110, GCAM111 or WITCH112.

Both categories of model face the challenges of calibrating the dynamic evolution of a large array of technological parameters and of the penetration of different technologies, which is affected by model structure. Therefore, coupling these types of models with expert judgments as input for these key parametric assumptions about technology costs and performance and their associated uncertainties is an important step towards more informative, robust and transparent investigations. Insights robust to differences in model structure can only be addressed through multi-model comparisons. Further investigations in the area of technology diffusion also based on historical evidence can play an important role too.

An important distinction we make in Fig. 1 is to classify integrated uncertainty modelling efforts into Learn-then-Act (comprising sensitivity analysis and uncertainty analysis) and Act–Learn–Act (which are often simply called Act-then-Learn in the literature and comprise one-stage and multi-stage decision-making under uncertainty, DMUU). Learn-then-Act frameworks combine elicitations with IAMs; Act–Learn–Act frameworks also integrate decision theory models (sometimes within IAMs themselves18, sometimes in a multi-model framework8).

The differences between Learn-then-Act and Act–Learn–Act frameworks are related to when decisions take place in relation to when uncertainty is resolved. The key uncertainty in the papers we review is technological outcomes. Learn-then-Act frameworks take near-term actions as given (in this case a particular level of public energy R&D investment), and explicitly model decisions which take place only after uncertainty has been realized (deployment decisions which take place within IAMs). Act–Learn–Act frameworks are more realistic, explicitly modelling near-term decisions that must be made before uncertainty is resolved; the full probability distribution is taken into account when choosing an alternative. In all of the papers reviewed, the near-term decision is investment in public energy R&D. The remaining later-stage decisions, including deployment, take place within the IAMs.

In this Review, we discuss work combining expert elicitations on energy technologies with IAMs and frameworks for decision-making under uncertainty to support public energy technology R&D policy decisions in the face of technological uncertainty. The insights stemming from the types of integrated quantitative analysis depicted in Fig. 1 and described in this Review are, of course, just one input to public policy. There are many other factors, such as politics, stakeholder values, and the interactions between policies in the energy sector and other sectors that require government investment, to name a few. Nevertheless, insights about optimal public energy R&D portfolios arising from integrated quantitative analysis can usefully support decisions by clarifying the presence of non-linearities and by providing clarity about assumptions, goals and trade-offs of different investments. The aims of this Review are three-fold: to summarize and categorize recent research incorporating expert elicitations and IAMs in decision-making frameworks; to highlight the most relevant insights coming from this literature for the design of public energy R&D portfolios; and to identify gaps and key research areas going forward.

Expert insights about the future of energy technologies

Before delving into the integrated approach presented in Fig. 1, we review the literature on energy technology expert elicitations.

Verdolini et al.19 provide a review of 29 expert elicitation studies performed between 2007 and 2012, covering eleven energy technology categories. The range of studies reviewed there (plus two additional recent wind studies, making a total of 31 papers) are summarized in Table 1 and the key challenges and gaps associated with energy technology expert elicitations are summarized in Box 2. It is important to note that none of the elicitations explored alternative scenarios of private-sector R&D investment due to a large extent to the lack of information regarding current private energy R&D investments by technology.

Table 1: Summary of energy technology expert elicitation studies.
Box 2: Overview of energy elicitations, challenges and gaps.

Table 2 provides an overview of the energy technology expert elicitations to date. Studies are organized by the research institution that was home to the research, whether or not studies had explicit questions about the future of technologies contingent on public R&D, the year of publication, the geographic coverage, and the year that was the focus of questions about the future of technologies. We then discuss the challenges associated with expert elicitations and research gaps in the area of energy technology expert elicitations that will improve the robustness and scope of the insights of future analysis.

In spite of the fact that expert elicitations overcome some of the challenges of using learning curves or deterministic estimates to forecast future energy technology costs and characteristics, expert elicitations (which are time and resource intensive7) have other limitations. First, expert elicitations are subject to a number of biases, such as availability, anchoring, and overconfidence113 and thus should only be used when there is no other good source of information. In the case of energy technology expert elicitations, we argue that this condition is met: while past aggregate data can give insights on overall effectiveness of public R&D, it has so far not provided probabilistic information about the impact of R&D on particular technologies or programmes. This problem is particularly acute when the technology of interest is novel and there is little historical data; more traditional methods, based on historical data, might be fine with more mature technologies. Indeed, Wiser et al.98 find strong agreement between experience curves and expert forecasts for onshore wind, but not for much-newer offshore wind. In addition, expert elicitations should only be used when there are knowledgeable experts that can provide insights about questions that involve “matters of fact” (Ref. 7). Some technologies may be too early in their development for experts to reasonably project costs.

Second, expert elicitation studies may be particularly difficult to compare with each other. The studies listed in Table 2 include different questions asked through different survey modes to different types of experts. For example, the studies vary considerably in their level of disaggregation, ranging from studies that question experts on the levelized cost of electricity for an entire technology category (for example, for CCS in the National Research Council study in Ref. 88), to those that ask for a probability distribution over specific technical parameters (for example, for sorbent concentrations in Amine-based CCS systems as in the CMU study85).

Third, it is often difficult to validate forecasts from expert elicitations with realized data. Energy technology expert elicitations look into the future, with endpoints typically between 2022 and 2050. Nevertheless, there is some evidence that some technologies have evolved on the faster end of what experts predicted. For example, the prices in solar PV electricity in Barbose et al.114 are lower than all but the most optimistic experts in the meta-analysis of solar elicitations by Verdolini et al.23. A similar preliminary finding on the faster rate of decrease of solar PV in particular periods is shown in Verdolini et al.19). Even once sufficient time has passed, it is difficult to evaluate the accuracy of elicitations that are conditional on things like R&D investments since the conditions may have changed. For the same reason, it is difficult to provide feedback to experts about their projections to help them adjust and improve their forecast. This is obviously an issue affecting all distant future elicitations.

Fourth, there is continuing controversy on the best way to use multiple, conflicting expert judgments (see for example a discussion on whether it is best to aggregate — average in some fashion — elicitations or not in Refs 38,115). Whether or not multiple expert judgments are combined (and how) can impact the ultimate results. The studies reviewed here used multiple methods, ranging from simple linear averaging41, to scenarios selecting optimistic, median, and pessimistic experts22.

Finally, there are gaps in the technologies that have been covered by elicitation studies. The bulk of the attention has gone to the electricity sector, with biofuels being the notable exception. To the best of our knowledge, no elicitations have been published on residential or industrial efficiency technologies. And, within the electricity sector, the supply-side has gotten much more attention than the demand-side, with the three studies on electric or hybrid vehicles as the main exception. Even within the supply side some technologies have received less attention than others. For example, there are two studies on wind97,98, two on non-CCS fossil technology88,96, one on advanced storage22, and none in nuclear fusion, geoengineering, biopower with CCS, or storage potentials. In addition, available expert elicitation studies include experts from the US and the EU, which means that wider regional coverage, including developing countries and especially China and India, is a really important gap.

The analysis of 31 papers included in Table 1 (Refs 3,19) provides robust evidence of the significant uncertainty surrounding future energy technology costs and performance. The uncertainty characterized by elicitations goes beyond the uncertainty estimated from forecasting methods based on historical data (for example, learning or experience curves, as a function of deployment20 or of time19). For example, using 20-year windows of historical data on solar photovoltaic (PV) module costs from Fraunhofer21, the average annual cost reduction ranges between 2.7% and 4.6%, depending on the start date of the window; the average annual cost reduction over a 25-year period (1980–2015) is 2.8%. For comparison, using individual expert answers from the Harvard solar PV expert elicitation22 on module costs, implied average annual cost reductions between 2010 and 2030 show a much wider range, from 0.1% to 4.7%. This range of uncertainty does not seem to depend on the level of R&D investment5,23. It is also worth noting that most of the elicitations sought to isolate the impact of R&D, recognizing that there are factors beyond R&D, such as input prices, learning-by-doing and economies of scale that have driven previous cost reductions. This has, for example, been demonstrated for solar PV20.

According to the studies in Table 1 and the related discussion in Box 2, on average, with significant variation across technologies, experts expect that higher levels of R&D would lead to cost reductions of over 50% between 2010 and 2030. Looking at the tails of the distributions, however, it is possible to detect non-zero probabilities of more significant breakthroughs leading to sustained cost reductions on the order of 10% per year until 20303. Examining only the medians or means would result in ignoring low-probability, high-impact events in decision-making. Among the studies, results vary by technology in terms of forecast rates of cost improvements; however, the uncertainty within and between studies dominates.

When comparing the rate of change in the future cost of technologies implied by the expert elicitation studies of projections based on historical data, it was found19 that the technologies differ in the degree to which the experts see the future as being similar to the past. For example, the median expert forecasts for nuclear power and biofuels are similar to historical data, while expert forecasts for bioelectricity are relatively pessimistic. This is consistent with the possibility that expert judgments represent a semi-independent source of information: experts are influenced by historical trends as well as by their personal experiences and beliefs about future technological developments.

There are many challenges and gaps associated with energy technology expert elicitations (see Box 2). Nevertheless, elicitations provide a systematic way, using the best available knowledge, to provide information on future technological change unavailable from other types of analyses.

Modelling and managing technological uncertainty

It is not enough to have an understanding of the uncertainty surrounding the future cost and performance of energy technologies. The design of efficient and cost-effective energy policies requires understanding the broader social impact of changes in these technologies, including the impact of interactions across technologies and between R&D and other policies. A large body of literature has explored the importance of technological costs on future mitigation pathways under uncertainty without using inputs from expert elicitations (see for example refs 24,​25,​26). In this section, we review studies that explicitly combine elicitation-based uncertainty on energy technology cost and performance with IAMs to support the design of energy technology policy.

Table 2 classifies modelling efforts into Learn-then-Act and Act–Learn–Act, categories introduced in Fig. 1. Sensitivity analysis, which takes the energy R&D investment decision as given, addresses technological uncertainty by experimenting over a range of technological scenarios. The result of sensitivity analysis is a range over the standard outputs of IAMs (for example, investment levels and climate mitigation costs). Most sensitivity analysis efforts do not rely on expert elicitations, but Table 1 includes some sensitivity analysis studies that do.

Table 2: Studies relying on technological uncertainty from expert elicitations.

Uncertainty analyses also take the energy R&D investment decision as given, but move a step further, using probability distributions of future technology costs and performances as inputs, and providing outputs from IAMs as probability distributions. The most well-known method of uncertainty analysis is Monte Carlo analysis.

The last two columns of Table 1 move into the realm of DMUU, which provides insight into near-term strategies given the current state of information. In the 1-stage DMUU approach, as illustrated in Fig. 1, key decisions (in this case R&D investment decisions) are made before uncertainty is resolved. Multi-stage DMUU approaches, on the other hand, allow for recourse — key decisions are revisited after uncertainty is revealed, including future climate policies or results from the previous energy R&D investments. Thus, multi-stage DMUU models can identify near-term decisions or policies with option value12. Among multi-stage DMUU frameworks, it is most common to have two stages, but some models include a larger number of downstream decisions. The two most common methods for implementing multi-stage models are dynamic programming27 and stochastic programming. Dynamic programming is increasingly being implemented with high-dimensionality solution techniques, such as approximate dynamic programming (ADP)28. In general, computational challenges are difficult in all multi-stage frameworks due to the curse of dimensionality: while stochastic programming needs to manage the number of future uncertain states, ADP models need to manage the number of state variables. Some have argued that the addition of decision stages beyond two may not provide considerable insight25,29. In all DMUU studies here, IAM-based deployment decisions take place after uncertainty is resolved. DMUU differs from sensitivity and uncertainty analysis, as the uncertainty explicitly feeds back into the near-term decision.

Sensitivity to energy technology costs. While expert elicitations have been used to calibrate IAMs, this does not fully utilize the probabilistic data assembled in expert elicitations. In Table 1, we include sensitivity analysis studies that use the range of values elicited from experts to analyse the impact of technology scenarios on model outputs. Typically, sensitivity analyses vary technology input parameters one at a time to assess the range of societal outcomes associated with extreme technology assumptions. This helps to identify critical uncertainties and to bound results. Pugh et al.30 performed an informal expert elicitation over various technical outcomes across a wide range of technologies, and used the results to assess emissions reductions and return on investment. Ricci et al.31 and Iyer et al.32 each perform a simple sensitivity analysis to bound possible future costs and efficiency of a single technology (carbon capture and storage (CCS) and nuclear, respectively) and its economic implications. Barron and McJeon33 investigate how much technologies would need to improve to significantly reduce the cost of climate-change mitigation. Olaleye and Baker34 perform a large scale, multi-parameter scenario analysis to identify technologies that are complements or substitutes at a societal level.

In recent years, there has been an effort to go beyond conventional sensitivity analysis by allowing various input parameters to change together, thus capturing interaction effects. Such approaches can use elicited probability distributions for random draws35 and are referred to as global sensitivity analysis. Bosetti et al.36 perform a multi-model global sensitivity analysis of energy technology costs, drawing on three elicitation studies over five technologies. Their results, robust to the choice of the IAM, indicate that when there is no climate policy, among the technologies considered, nuclear cost assumptions are the biggest technology cost determinants of projected societal energy costs and baseline emissions, complementing findings about the role of nuclear with climate policy37. Sensitivity analysis is more computationally tractable than the methods we discuss later, and is appropriate for identifying areas of interest that could be the subject of additional research. It has descriptive value and its results can be easily presented to policymakers or other users, usually in the form of scenarios representing particular narratives. Sensitivity analysis represents a first step towards understanding the resilience of a particular policy. It has the weakness, however, of not taking into account many of the non-linearities mentioned above. Thus, researchers and analysts should consider going beyond sensitivity analysis, even if this presents challenges to communication, because “decision-makers are often far better served by examining the diversity of opinion”38 and thinking about the robustness of policy approaches to the uncertainty.

Towards understanding uncertainty. Expert elicitation data can also be used in uncertainty analysis, most commonly Monte-Carlo analyses. Monte-Carlo analyses use random draws from probability distributions as inputs to models, such as IAMs, resulting in probability distributions over the outputs of interest. Gillenwater39 uses an elicitation on wind to estimate the effect of policies on the probability of private sector deployment investment. A similar, but more sophisticated method, Latin Hypercube Sampling (LHS)40, can reduce the number of needed simulations while preserving the probabilistic interpretation of the model outputs. Chan and Anadon41 account for the dependency between R&D-induced energy technology improvements across a set of 25 technologies (some of which are interrelated) using LHS. The results include probability distributions of future CO2 emissions, oil imports, consumer and producer surplus and other metrics under different R&D scenarios, with and without specific demand-side policies.

Another method of uncertainty analysis is to propagate probability distributions through a model. Olaleye42 uses large-scale scenario analysis to estimate probability distributions over welfare for different R&D portfolios, by associating each technological outcome with a conditional probability derived from elicitations. Lemoine and McJeon43 combine energy technology elicitation results with an IAM to investigate the probability distributions of climate damages and mitigation costs. Several elicitation studies use deterministic runs of an IAM combined with elicited probabilities of particular technology costs to derive conditional probability distributions over the impacts of R&D on the marginal abatement cost curve, with a focus on the different ways in which different technologies impact the curve44,​45,​46.

Nemet and Baker47 combine the results of a solar expert elicitation with a techno–economic model that allows them to derive probability distributions over the cost of solar conditional on both technology R&D and technology subsidies. Similarly, a set of interrelated papers propagate elicited distributions of the outcomes of R&D in CCS technology through different techno-economic models to produce probability distributions of the future technology costs and overall CO2-avoidance for various R&D investment and deployment policies48,​49,​50.

Uncertainty analysis forces researchers and policymakers to think about probabilities, with the added benefits (and added complexities) that this entails51. However, similarly to sensitivity analysis, uncertainty analysis by itself does not directly inform near-term decision-making, such as the optimal amount and allocation of R&D investments.

Modelling insights from DMUU frameworks. The studies presented in Table 3 and discussed in Box 3 show that there are trade-offs involved in selecting DMUU strategies, involving the number of technologies that are explicitly modelled as uncertain, the number of decision stages, the representation of R&D, and use of reduced-form decision models. Baker and Solak52 consider only three technologies, but solve a technologically-detailed stochastic programming version of the IAM DICE, where investments into R&D impact the marginal abatement cost curve and are the first stage decision. All papers with 4 or more technologies feed outputs from IAMs into reduced-form decision models. Chan and Anadon41 include continuous public R&D levels for 6 technology areas that affect the cost and performance of 25 individual technologies, but only one-stage DMUU. Six papers in Table 3 use two-stage DMUU8,42,53,​54,​55,​56, but only have 3–6 technology categories; all but one56 use discrete R&D investment levels. Santen and Anadon17 use ADP to solve a four-stage DMUU, but consider R&D-dependent uncertainty in only one technology (which competes with three others for deployment). Most of the multi-stage DMUU frameworks include R&D as a first-stage decision and deployment in later stages; Santen and Anadon17 include decisions on R&D and deployment in all 4 stages. Some studies optimized given one or several budget constraints41,42,53,57, others solved for the optimal budget level8,17,52, and others do both54,​55,​56,58.

Table 3: Details of the papers used to produce Figs 24.

It is important for researchers and users of information to be aware of the crucial design decisions being made when conducting this analysis and to understand what trade-offs are being made and why. Key design decisions are the number of technologies that are modelled with R&D dependent uncertainty, the number of decision stages, whether or not R&D funding is continuous, and the decision framework.

Another question for researchers and policymakers is whether one piece of the analytical framework depicted in Fig. 1 is driving the results. Is it enough to look at the results of expert elicitations to determine which R&D investments are best to achieve a particular policy goal? Or, on the other hand, do the built-in assumptions of IAMs swamp the assumptions stemming from expert elicitations? We find evidence that both components influence results, along with the decision framework. A comparative analysis8 reported in Fig. 2 shows that the expert elicitation data matters: the IAM and decision frameworks were identical yet the R&D portfolio results vary by elicitations. Fig. 3 shows that the inclusion of more technologies (for example, some studies include vehicles and some do not) also has an impact on results about optimal energy R&D portfolios. Figures 2 and 4 both show that the decision framework matters. Comparing the results of Baker et al.8 who use the UMass elicitations with the McJeon study53 in Fig. 2 shows that while both studies employ the same IAM and elicitation data, they have different results. Santen and Anadon17 also find that the decision framework matters: the results presented in Fig. 4 show that deterministic, Monte-Carlo, and 5-stage DMUU lead to different levels of R&D versus technology deployment investments. Finally, Barron57 shows that the choice of the IAM matters, with GCAM and MESSAGE providing different optimal R&D portfolios using the same elicitations and decision frameworks.

Figure 2: Optimal R&D portfolios under different climate stabilization levels.
Figure 2

This figure shows the fraction of the R&D portfolio devoted to different technology areas for different climate stabilization levels, as determined by different approaches. In the legend, bioenergy refers to an R&D programme in which biofuels and bioelectricity R&D were not differentiated, even if technologies were. Note that many technologies such as wind were not included in these modelling approaches using elicitation data. Group 1 (Ref. 41) uses elicitations from Harvard22,59,106, the MARKAL IAM, 1-stage DMUU, and maximizes consumer and producer surplus under a US$5bn total R&D constraint under no carbon policy (business as usual: ‘BAU’) and a strict carbon policy (‘stringent’) of 83% reduction of energy CO2 emissions by 2050 compared to 2005 levels. Groups 2, 3 and 4 use harmonized elicitations59 conducted by Harvard, UMass and FEEM, respectively, the GCAM IAM, and minimize the cost of climate stabilization at 450 and 550 ppm plus damages and R&D cost8. Group 5 uses the elicitation from UMass59, the GCAM IAM, 2-stage DMUU, and minimizes the cost of achieving climate constraints of 450 and 550 ppm (Ref. 53). Group 6 uses FEEM elicitations59, WITCH IAM, 2-stage DMUU, and minimizes costs of achieving a 2.8 °C global average temperature by 210056. Note that the definitions of ‘lenient’ and ‘stringent’ are not identical in the different studies.

Figure 3: Optimal R&D portfolios under different total energy R&D budget constraints.
Figure 3

This figure shows the fraction of the R&D portfolio devoted to different technology areas for different budget constraints, as determined by different approaches. The R&D budget constraint level is normalized by the lowest R&D budget constraint for each group: for each Group an R&D level of 1 represents the lowest budget considered in that study, with the subsequent budget levels shown as multiples of that value. Group 141 uses elicitations from Harvard22,59,105,106. Group 256 uses elicitation from FEEM59, Groups 354, 452, and 542 use elicitations from UMass59 in different frameworks. Group 658 uses elicitations from Harvard22,59,106, UMass59 and Stanford96. Group 753 uses elicitations from UMass59. Note that many technologies such as wind were not included in these modelling approaches using elicitation data. Climate constraints are the most stringent available in each of the studies, see Table 3 for details. Not all studies included all technologies. Table 3 includes additional details.

Figure 4: Public investment in energy R&D as a percentage of the total investment in energy technologies.
Figure 4

Total investments are estimated as the sum of public energy R&D investment in energy technology capacity deployment investment (including private investment and deployment subsidies) between 2010 and 2040. Santen and Anadon17 include the R&D investment as a percentage of total electricity capacity investment using three decision frameworks (no uncertainty, uncertainty analysis, and DMUU with five decision stages), including continuous variables for R&D investments and for R&D-dependent solar technology costs. Nemet and Baker47 optimize deployment for a given level of R&D. Marangoni et al.56 optimize under two-stage DMUU. The results shown here are scaled down to reflect only the contribution of the United States. The uncertainty ranges for Santen and Anadon17 include the range of future period decisions, which are conditional upon realizations of the past; the marker represents the median of these values.

Policy insights

One important dimension that emerges when thinking about designing optimal energy R&D portfolios is to what extent they depend on the context, including other policies or policy goals. In particular, climate policy stringency and the availability of resources for R&D investments and for deployment of energy technologies are key factors prevalent in policy debates in the energy sector.

We now review the insights for policy that emerge from the literature introduced above. We organize these insights into three areas: the impact of climate change targets on optimal R&D portfolios; the impact of R&D budget constraints on the allocation of R&D investments; and a comparison between funding allocated to public R&D and to technology deployment.

Optimal R&D portfolios under different climate targets. Figure 2 summarizes a range of studies investigating the extent to which different climate constraints affect optimal R&D portfolios. Results are organized into groups with common elicitations, IAMs, and decision methods. Of the six groups analysed, which come from 4 major studies41,53,56,59, four groups (Groups 1–4) compare a baseline ‘no policy’ scenario with either a lenient and stringent climate stabilization scenario or with just a stringent climate stabilization scenario (the Fig. 2 caption provides information about the specific studies), Group 5 compares a lenient and a stringent climate stabilization scenario and Group 6 presents the optimal R&D allocation under a lenient climate stabilization policy. Although the climate scenarios are not entirely comparable, we group them into the broad categories of stringent and lenient climate target scenarios; details and assumptions are reported in Table 3.

There are five technology-specific results that emerge from the analysis presented in Fig. 2. First, solar gets the smallest share of R&D and its role slightly decreases as the target gets more stringent. This is mainly due to the different magnitude of solar R&D investments compared to the other type of technologies. Second, the stricter the climate constraint, the greater the share of CCS in energy R&D portfolios, with the exception of one study8 using the UMass elicitations. CCS is critical for negative emissions (bioenergy plus CCS), which in turn is an increasingly important strategy for stricter climate targets. Third, stricter climate constraints lead to an equal or smaller share of nuclear in energy R&D portfolios, with the exception of one study8 using elicitations from Ref. 22. This appears to be due to large nuclear investments under low-stringency targets, with less room to grow as targets get more stringent. Fourth, all studies that include vehicles find that a significant share of the overall budget is devoted to advanced vehicles R&D, ranging from 30–80% of the total R&D investments; this fraction increases with stricter climate constraints. Fifth, R&D funding for utility-scale energy storage, a technology area included in only one study, increases with stricter climate constraints.

It is important to note that this is a summary of emerging insights, and many gaps and challenges exist, such as model biases and missing or poorly modelled technologies, making it difficult to extract recommendations about the precise level of R&D investments into specific technologies. However, one general conclusion does appear: technologies which provide flexibility see higher R&D investments as climate stringency increases. CCS allows for ex post emission decreases from fossil plants; storage allows higher penetration of intermittent technologies; and vehicles allow abatement to reach into the transportation sector. The fact that we do not see this pattern for biofuels reflects elicitation results, particularly in the US studies, which may in turn reflect high existing levels of R&D.

Optimal R&D portfolios for different R&D budget constraints. Another pressing question in the minds of policymakers, particularly in times of financial crisis, is how R&D investment portfolios should change for different public energy R&D budget constraints. Figure 3 summarizes the results of seven papers22,41,42,52,54,56,58 that consider multiple public energy R&D budget constraints.

Results show that with tighter R&D budgets, portfolios become less diverse, with some technologies not funded at all. Among the four studies that include vehicles22,41,42,53 there is a decrease in the share of vehicles R&D with greater R&D budgets. This is likely because vehicles R&D (in particular batteries) get a large share to begin with in constrained budgets; and because the marginal improvement seen in the elicitations above and beyond those levels is smaller than that of other technologies, especially CCS. Among the three studies that include bioenergy41,42,56 the fraction of R&D devoted to total bioenergy (the combination of biofuels and bioelectricity) stays relatively constant, driven to some extent by assumptions about the availability and cost of biomass.

Solar plays an interesting role in these portfolios. First, we note that, as the R&D budget constraint is relaxed, and consequently the number of technologies considered grows, the share of R&D for solar decreases precipitously (more so than for nuclear, CCS, or vehicles). Moreover, the share of solar is non-monotonic in a number of studies, more prominently than any of the other technologies. This may indicate that solar plays the role of a kind of ‘filler’ technology in these portfolios. The problem of finding an optimal portfolio subject to a constraint is what is known as a ‘knapsack problem’. It is well known that knapsack problems can lead to non-monotonic results as the constraint (in this case the total energy R&D budget) relaxes. Solar R&D plays this role since, especially in comparison to CCS and nuclear, it can be productive even at lab scales. On the other hand, in the absence of cost-effective storage technologies, solar PV has a limited impact on the overall economy, due to structural assumptions about intermittency in many of the IAMs.

The main disagreement in the results in Fig. 3 is for the fraction of R&D devoted to nuclear versus CCS. In two of the papers42,54 nuclear increases at the expense of CCS as the R&D budget grows; while for the other 4 papers41,52,53,58 that include both technologies this trend is reversed. Nuclear and CCS tend to be strong substitutes; thus, depending on the specific IAM assumptions, different technologies take a larger share. Both of these technologies are associated with significant socio-technical issues that are difficult to model, such as public acceptance. Thus, factors embedded in IAMs beyond future cost (for example, limits on the rate of deployment) are likely to play a role on whether one or the other technology receives the largest share of R&D portfolios.

We can also ask to what degree R&D expenditures are justified: what amount is the right amount? This question has been asked in the theoretical literature using hypothetical probability distributions or the comparative statics of risk (see Ref. 60 for a review). Some of the papers reviewed here have taken on this question using elicitation-based distributions. McJeon53 finds that the savings in abatement costs are 2–3 orders of magnitude higher than the R&D investment costs for all portfolios considered; Anadon, Chan, and Lee22 find that current US public energy R&D budgets could be increased by at least an order of magnitude and still be cost-effective in terms of consumer and producer surplus; there are similar results in other papers, especially when opportunity costs are low, potential benefits are high, and a limited set of projects is available. Most papers, however, find that the optimal investment in R&D includes only a subset of the total available projects. Thus, the opportunity cost of R&D investment does play a role. A few papers have found that R&D is more valuable in second-best worlds8,52,61; that under-investment may be more costly than over-investment52; and that the optimal investment decreases with increasing discount rates and opportunity costs, and appears to be non-monotonic in risk, first increasing then decreasing as risk increases52,54.

R&D versus deployment expenditures. A key debate in innovation and climate policy is the question of how to balance public support for R&D as opposed to technology deployment. A recent study62 highlighted this as one of the hardest questions in crafting comprehensive energy policy. In Fig. 4 we highlight the ratio of (mostly public sector) R&D investments compared to the long term (mostly private sector) investments in deployment resulting in some of the elicitation-based studies. Fig. 4 shows the percentage of R&D investment to the total investment (including R&D and all deployment costs, such as capacity investments and government subsides) between 2010 and 2040; this proportion ranges between 0.1–9.2%.

While the innovation literature has highlighted the need for both technology-push and market-pull technology policies63, and the energy economics literature has discussed the need for two instruments to address two market failures of environmental and knowledge externalities64, we are not aware of any attempt to obtain an estimate of what the optimal range of public R&D is when compared to total deployment costs, given our current knowledge. While the range, including uncertainty, spans over one order of magnitude, no study found that the fraction of R&D should be more than 9.2% of the total investment, with the median result around 3.5%. For reference, this ratio for solar PV in the United States in 2015 was roughly 1.1%. This estimate was calculated65 using a federal R&D investment in solar PV in 2015 of US$231 million, and a rough deployment investment of US$20.6bn, estimated using a value of total installed solar capacity of 7.3 GW, split between 4.3 GW of utility-scale solar and 3 GW of non-utility-scale solar66, and assuming average solar costs in 2015 of 2 US$ W–1 for utility-scale solar and 4 US$ W–1 for non-utility-scale solar67. The question of R&D versus deployment is a ripe area for future research, since only limited studies were available for this analysis.

In addition to shedding light on these three areas, the existing literature also includes a few studies that investigate different questions. Ziolkowska68 combines a biomass and bioenergy elicitation with a multi-objective decision framework, considering economic, environmental and social policy objectives in evaluating different biomass feedstocks to inform energy policy. Bistline61 investigates the optimal electricity capacity investment portfolio under uncertainty about technology prices derived from elicitations. Using a two-stage stochastic programme, the paper finds that natural gas prices and the stringency of climate policy are the main risks associated with capacity expansion planning.

Gaps and challenges in combining the three approaches

We have discussed the insights that can be gained about public energy technology R&D policy through an integrated analytic approach that combines expert elicitation, IAMs, and DMUU frameworks. We now address some gaps in knowledge and methodology and potential challenges to this combined approach.

First, the combined approach is necessarily limited by the availability of expert elicitations and the limitations of IAMs. The set of technologies that can be included in a combined framework is limited to those for which elicitations have been performed and which IAMs represent in some detail. In addition, the representation of the energy system in IAMs can strongly influence results. For example, the result that solar PV plays a small role in many R&D portfolios may be related to how IAMs model the challenges of grid integration. Recent work69 has revisited the representation of wind and solar across multiple models, leading to higher shares of these technologies in more recent versions of the models. It will be important to test expert elicitation data on this new generation of IAMs. Beyond these gaps, the limitations of IAMs (Box 1) and expert elicitations (Box 2) affect the combined analysis, and thus must be considered when interpreting results.

A second issue relates more generally to all optimization frameworks used in policy assessments: how, and by whom, is the objective function defined? The papers reviewed here are primarily aimed at a high level, relevant to national governments or international agencies. However, much decision-making takes place at lower levels, such as sub-national agencies. Moreover, it has long been acknowledged that the problem of identifying an objective function, even in the case of a single decision-maker, is quite challenging70. This becomes even more challenging when the problem is in the public realm and consists of multiple policy goals71, stakeholders72, and long timeframes. The role of optimization and decision frameworks in this case is not to provide a single optimal solution, but rather to shed light on the impact of decisions, uncertainty, and preferences in a quantitative framework; ultimately, decision-makers are faced with value judgments over particular optimization criteria. There are considerable gaps in the objectives that have been explored by this approach in the literature so far. Most of the papers employing DMUU either minimize the cost of achieving a climate goal (such as a particular stabilization scenario) or minimize the combined cost of abatement and damages. A few papers explore other objectives, such as maximizing consumer and producer surplus, reducing carbon price, or reducing CO2 emissions in the US22. Ziolkowska68 uses a multi-criteria approach, including criteria such as economic efficiency, reducing GHGs and water use, and supporting local communities. The wide range of possible policy objectives (which might vary based on the geographical and socio-economic context of the decision-maker) has yet to be explored using these frameworks. Some objectives of interest may be general economic development, reduction of inequality or poverty, energy access, and many others. To truly address this gap, researchers must collaborate with policymakers to support decisions in specific contexts. Nevertheless, there is the danger of ‘lamp-posting’ — of only addressing questions that current models are set up to answer.

A third area that could help address some of the limitations of the combined approach is to perform structured multi-model comparisons; including sets of IAMs with different assumptions about technology diffusion, built-in objectives, and decision frameworks, as well as multiple elicitation studies to parameterize technological uncertainty performed using different methods. Structured multi-model comparisons have shown great promise in increasing the robustness of insights in the context of IAMs and the Energy Modeling Forum (EMF) studies6,69,73, and could help increase the robustness of existing insights from the integrated approach presented here, or provide new policy insights.

A fourth area in need of additional attention by the research community is that of understanding the private sector response to government policy, both in terms of R&D investments and the challenges to the commercialization of technologies74, and on the interaction between innovation in different technologies through spillovers. While some headway has been made using patent studies75, little is known about private R&D spending, its effectiveness, and its relation to public R&D and other policies76. Thus, this important piece of the puzzle often gets left out of technology policy analyses. Similarly, the challenges related to spillovers — between locations and between energy R&D and other areas — are not included in the reviewed papers due to the limited availability of data and analysis. A particular concern is a lack of representation of R&D in emerging economies, and related international spillovers. Most broadly, it is important to note that these important mechanisms, including private R&D response and first phase deployment challenges, have not been considered in expert elicitations nor are they well-represented in IAMs. These mechanisms have an important role, and thus are ripe for more research.

Current decision-making on public energy R&D investments is generally opaque: it is unclear to external analysts, stakeholders, or taxpayers where assumptions about future costs and R&D effectiveness come from, and what goal or goals are being prioritized22. The type of analytical approach presented here can serve as a complement to inform and scrutinize public decisions, since the source of the technology assumptions (the names of the experts) is clear, and the models and optimization criteria are specified. However, this approach is only useful to the degree that policymakers buy in to the results and insights. This is a major challenge, not just for this combined framework, but for all approaches that explicitly include uncertainty in the analysis. And thus, a fifth and final area for future work is in uncertainty communication, including two-way communication with policymakers through a process of research codesign, so that frameworks, such as those we have reviewed here, are informed by both the insights of policymakers and responsiveness to their needs.


Governments are increasingly turning their attention to questions about energy technology R&D portfolios and energy innovation policy. The US. Department of Energy recently created the Office of Technology Transitions, responsible for developing and overseeing delivery of strategic vision77 and goals around technology commercialization, including considerations about R&D portfolios. Various global efforts are emerging aimed at improving energy technologies, including Mission Innovation, the Breakthrough Energy Coalition, and the United Nations Framework Convention on Climate Change Green Climate Fund.

Decisions about how to best support innovation in energy technologies are hard for a number of reasons, but among the most important is the deep uncertainty about the outcomes of R&D and other policies, the interactions of technologies, and the science of climate change. The approach we review in this paper — integrating expert elicitations, IAMs, and decision frameworks — is complex. Yet the transition pathway ahead of us demands radical changes in policies and in the economy, as well as strategies to enable such radical changes, in all their complexities. Thus, governments have an imperative to acquire, integrate, and use the best available information in order to support this important decision-making process. Many governments are developing capabilities in IAMs, allowing them to use this set of tools to inform decisions (for example, the US Department of Energy78, and the UK Department of Business, Energy and Industrial Strategy79,80). It would be worthwhile for governments to develop capabilities similar to those presented in this paper to understand the full ramifications of expert elicitations alongside other data and DMUU frameworks81.

Of course, expert elicitations, IAMs, and decision frameworks, while crucial, are themselves only part of the full range of available information and analytic tools that can be used to inform policy decisions. An open question for the future is how to go beyond the already complex and integrative approach presented here, as additional data becomes available. This additional information could come from sources as diverse as prediction markets, increased computing power, and hybrid data–human-based forecasting methods. The challenge rests in incorporating this body of information while all the while maintaining a firm control on the various components of the analysis and ensuring a rigorous validation of the whole process, through methods such as repeated elicitations of experts, controlled comparison of elicitation methods, open-source code for integrated assessment models and frameworks, randomized control trials, and open data sets. Nonetheless, we hope this Review has shown that expert knowledge reflecting uncertainty can be profitably integrated with sophisticated modelling methods to inform energy technology R&D policy.

Author contributions

All authors contributed equally to this work, conceiving the research, gathering and analysing data, creating illustrations, and writing the paper. Authors are listed in alphabetical order.

Additional information

How to cite this article: Anadón, L. D., Baker, E. & Bosetti, V. Integrating uncertainty into public energy research and development decisions. Nat. Energy 2, 17071 (2017).

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


  1. 1.

    Energy At a Crossroads: Global Perspectives and Uncertainties (MIT Press, 2005).

  2. 2.

    & Policy advice: Use experts wisely. Nature 526, 317–318 (2015).

  3. 3.

    , , & Expert views — and disagreements — about the potential of energy technology R&D. Climatic Change 136, 677–691 (2016). This paper brings together three sets of expert elicitations covering five important energy technologies and presents insights that include aggregated and disaggregated results and the impact of R&D on expected costs.

  4. 4.

    , & Special issue on defining robust energy R&D portfolios. Energy Policy 80, 215–218 (2015).

  5. 5.

    , & Quantifying the effects of expert selection and elicitation design on experts’ confidence in their judgments about future energy technologies. Risk Anal. 37, 315–330 (2016).

  6. 6.

    , , , & Introduction to the EMF28 study on scenarios for transforming the european energy system. Clim. Change Econ. 4, 1302001 (2013).

  7. 7.

    Use (and abuse) of expert elicitation in support of decision making for public policy. Proc. Natl Acad. Sci. USA 111, 7176–7184 (2014). This paper provides an overview of the key practices and issues in the use of expert elicitations to inform policy, with examples that include environment and energy applications.

  8. 8.

    , & Decision frameworks and the investment in R&D. Energy Policy 80, 275–285 (2015). This paper isolates the impact of decision frameworks on R&D decisions using distributions over future technology costs from a combined set of expert elicitations.

  9. 9.

    Hedging Strategies for Global Carbon Dioxide Abatement: A Summary of Poll Results — EMF 14 Subgroup Analysis for Decisions under Uncertainty (Energy Modelling Forum, 1996).

  10. 10.

    & Buying Greenhouse Insurance: the Economic Costs of Carbon Dioxide Emission Limits (MIT press, 1992).

  11. 11.

    Micoreconomic Analsysis (Norton, 1992).

  12. 12.

    & Investment Under Uncertainty (Princeton Univ. Press, 1994).

  13. 13.

    et al. Optimal Choice of Policy Instrument and Stringency Under Uncertainty: the Case of Climate Change (Citeseer, 1997).

  14. 14.

    , , & Anticipating climate threshold damages. Environ. Model. Assess. 17, 163–175 (2012).

  15. 15.

    & Uncertainty and climate change. Environ. Resour. Econ. 22, 3–39 (2002).

  16. 16.

    & An expert elicitation of climate, energy and economic uncertainties. Energy Policy 61, 811–821 (2013).

  17. 17.

    & Balancing solar PV deployment and RD&D: a comprehensive framework for managing innovation uncertainty in electricity technology investment planning. Renew. Sustain. Energy Rev. 60, 560–569 (2016). This paper explores the role of the uncertainty framework (including approximate dynamic programming) on the balance between investments in solar R&D versus solar deployment given uncertainty around the future costs of solar PV.

  18. 18.

    , & Climate mitigation under an uncertain technology future: a TIAM-WORLD analysis. Energy Econ. 34, S366–S377 (2012).

  19. 19.

    , , , & The Future Prospects of Energy Technologies: Insights from Expert Elicitations Working Paper 47 2016 (FEEM, 2016).

  20. 20.

    Beyond the learning curve: factors influencing cost reductions in photovoltaics. Energy Policy 34, 3218–3232 (2006).

  21. 21.

    Recent Facts about Photovoltaics in Germany (Fraunhofer ISE, 2016).

  22. 22.

    , & in Transforming US Energy Innovation (eds , & ) Ch. 2, 36–75 (Cambridge Univ. Press, 2014).

  23. 23.

    , , & The effects of expert selection, elicitation design, and R&D assumptions on experts’ estimates of the future costs of photovoltaics. Energy Policy 80, 233–243 (2015).

  24. 24.

    & Modeling uncertainty of induced technological change. Energy Policy 28, 907–921 (2000).

  25. 25.

    , & An approximate dynamic programming framework for modeling global climate policy under decision-dependent uncertainty. Comput. Manag. Sci. 9, 339–362 (2012).

  26. 26.

    et al. in Climate Change 2014: Mitigation of Climate Change (eds et al.) Ch. 6, 413–510 (Cambridge Univ. Press, 2014). This section of the IPCC report covers the wide range of literature considering the impact of technology costs on mitigation without focusing on the role of public energy R&D.

  27. 27.

    Dynamic programming and Lagrange multipliers. Proc. Natl Acad. Sci. USA 42, 767–769 (1956).

  28. 28.

    Approximate Dynamic Programming: Solving the Curses of Dimensionality (John Wiley & Sons, 2007).

  29. 29.

    Increasing risk and increasing informativeness: equivalence theorems. Operations Res. 54, 26–36 (2006).

  30. 30.

    et al. Energy R&D portfolio analysis based on climate change mitigation. Energy Econ. 33, 634–643 (2011).

  31. 31.

    , , & From Expert Elicitations to Integrated Assessment: Future Prospects of Carbon Capture Technologies (FEEM, 2014).

  32. 32.

    , , & Implications of small modular reactors for climate change mitigation. Energy Econ. 45, 144–154 (2014).

  33. 33.

    & The differential impact of low-carbon technologies on climate change mitigation cost under a range of socioeconomic and climate policy scenarios. Energy Policy 80, 264–274 (2015).

  34. 34.

    & Large scale scenario analysis of future low carbon energy options. Energy Economics 49, 203–216 (2015).

  35. 35.

    , , & Uncertainty in climate change modeling: can global sensitivity analysis be of help? Risk Anal. 34, 271–293 (2014).

  36. 36.

    et al. Sensitivity to energy technology costs: a multi-model comparison analysis. Energy Policy 80, 244–263 (2015). This paper conducts a structured sensitivity analysis informed by expert elicitations over energy technology costs across three important energy–economic models, shedding light on the impact of model structure on mitigation costs.

  37. 37.

    & How much can nuclear power reduce climate mitigation cost? — Critical parameters and sensitivity. Energy Strategy Rev. 6, 12–19 (2015).

  38. 38.

    Our knowledge of the world is often not simple: policymakers should not duck that fact, but should deal with it. Risk Anal. 35, 19–20 (2015).

  39. 39.

    Probabilistic decision model of wind power investment and influence of green power market. Energy Policy 63, 1111–1125 (2013).

  40. 40.

    & A distribution-free approach to inducing rank correlation among input variables. Commun. Stat. Simulation Comput. 11, 311–334 (1982).

  41. 41.

    & Improving decision making for public R&D investment in energy: utilizing expert elicitation in parametric models. Preprint at (2016). This paper combines expert elicitations and LHS to optimize public R&D investments in six energy technology areas affecting costs in a total of 25 technologies, identifying areas with the largest returns.

  42. 42.

    Role of Low Carbon Energy Technologies in Near Term Energy Policy. PhD thesis, Univ. Mass. Amherst (2016).

  43. 43.

    & Trapped between two tails: trading off scientific uncertainties via climate targets. Environ. Res. Lett. 8, 34019 (2013).

  44. 44.

    , & Carbon capture and storage: combining economic analysis with expert elicitations to inform climate policy. Climatic Change 96, 379–408 (2009).

  45. 45.

    , & Advanced solar R&D: combining economic analysis with expert elicitations to inform climate policy. Energy Econ. 31, S37–S49 (2009).

  46. 46.

    , & Battery technology for electric and hybrid vehicles: expert views about prospects for advancement. Technol. Forecast. Soc. Change 77, 1139–1146 (2010).

  47. 47.

    & Demand subsidies versus R&D: comparing the uncertain impacts of policy on a pre-commercial low-carbon energy technology. Energy J. 30, 49–80 (2009).

  48. 48.

    , & Expert elicitations of energy penalties for carbon capture technologies. Int. J. Greenhouse Gas Control 12, 136–145 (2013).

  49. 49.

    , & Modeling the future costs of carbon capture using experts’ elicited probabilities under policy scenarios. Energy 56, 218–228 (2013).

  50. 50.

    , , & Characterizing the effects of policy instruments on the future costs of carbon capture for coal power plants. Climatic Change 133, 155–168 (2015).

  51. 51.

    Thinking, Fast and Slow (Macmillan, 2011).

  52. 52.

    & Management of energy technology for sustainability: how to fund energy technology research and development. Products Operat. Manag. 23, 348–365 (2014).

  53. 53.

    Energy Technology Development And Climate Change Mitigation. PhD Thesis, Univ. Maryland (2012).

  54. 54.

    , & How grid integration costs impact the optimal R&D portfolio into electricity supply technologies in the face of climate change. Sustain. Energy Technol. Assess. 7, 22–29 (2014).

  55. 55.

    & Climate change and optimal energy technology R&D policy. Eur. J. Oper. Res. 213, 442–454 (2011).

  56. 56.

    , & Optimal Clean Energy R&D Investments Under Uncertainty. FEEM Working Paper no. 2017.016 (2017).

  57. 57.

    Analysis of the Impact of Technological Change on the Cost of Achieving Climate Change Mitigation Targets. PhD Thesis, Univ. Mass. Amherst (2015).

  58. 58.

    Energy technology R&D portfolio management: modeling uncertain returns and market diffusion. Appl. Energy 183, 1181–1196 (2016).

  59. 59.

    , , , & Future costs of key low-carbon energy technologies: harmonization and aggregation of energy technology expert elicitation data. Energy Policy 80, 219–232 (2015).

  60. 60.

    , & Technical change and the marginal cost of abatement. Energy Econ. 30, 2799–2816 (2008).

  61. 61.

    Electric sector capacity planning under uncertainty: climate policy and natural gas in the US. Energy Econ. 51, 236–251 (2015).

  62. 62.

    The Power of Change: Innovation for Development and Deployment of Increasingly Clean Electric Power Technologies (US National Academies Press, 2016).

  63. 63.

    & The influence of market demand upon innovation: a critical review of some recent empirical studies. Res. Policy 8, 102–153 (1979).

  64. 64.

    , & A tale of two market failures: technology and environmental policy. Ecol. Econ. 54, 164–174 (2005). This paper discusses some of the key reasons why public investment in energy R&D is very important.

  65. 65.

    & DOE Budget Authority for Energy Research, Development & Demonstration Database (Belfer Center for Science and International Affairs, 2016); -2

  66. 66.

    US solar market sets new record, installing 7.3 GW of solar PV in 2015. Solar Energy Industries Association (22 February 2016).

  67. 67.

    Median installed price of solar in the United States fell by 5–12% in 2015. Berkeley Lab News Center (24 August 2016).

  68. 68.

    Evaluating sustainability of biofuels feedstocks: a multi-objective framework for supporting decision making. Biomass Bioenergy 59, 425–440 (2013).

  69. 69.

    et al. System integration of wind and solar power in Integrated Assessment Models: A cross-model evaluation of new approaches. Energy Econ. (2016).

  70. 70.

    Some unsolved problems in problem solving. Operat. Res. Q. 13, 1–11 (1962).

  71. 71.

    Missions-oriented RD&D institutions in energy between 2000 and 2010: a comparative analysis of China, the United Kingdom, and the United States. Res. Policy 41, 1742–1756 (2012).

  72. 72.

    & Dilemmas in a general theory of planning. Policy Sci. 4, 155–169 (1973).

  73. 73.

    et al. The role of technology for achieving climate policy objectives: overview of the EMF 27 study on global technology and climate policy strategies. Climatic Change 123, 353–367 (2014). This paper presents a large effort to estimate the value of investing in R&D to improve energy technologies without a direct focus on uncertainty or on optimizing R&D investments across a range of areas.

  74. 74.

    Accelerating the development and diffusion of new energy technologies: beyond the ‘valley of death’. Energy Econ. 33, 674–682 (2011).

  75. 75.

    Inter-technology knowledge spillovers for energy technologies. Energy Econ. 34, 1259–1270 (2012).

  76. 76.

    , , & Bottom-up estimation of industrial and public R&D investment by technology in support of policy-making: the case of selected low-carbon energy technologies. Res. Policy 41, 116–131 (2012).

  77. 77.

    Mission (Department of Energy, 2015);

  78. 78.

    Office of Technology Transitions — Overview (US Department of Energy, accessed 10 April 2017);

  79. 79.

    Carbon Valuation in UK Policy Appraisal: a Revised Approach (Department of Energy and Climate Change, 2009).

  80. 80.

    Quality Assurance Tools and Guidance in BEIS (Department for Business, Energy and Industrial Strategy, 2016).

  81. 81.

    , & Special issue on defining robust energy R&D portfolios. Energy Policy 80, 215–218 (2015).

  82. 82.

    , , & The power of biomass: experts disclose the potential for success of bioenergy technologies. Energy Policy 65, 94–114 (2014).

  83. 83.

    , , & Advanced biofuels: future perspectives from an expert elicitation survey. Energy Policy 56, 293–311 (2013).

  84. 84.

    & Cellulosic biofuels: expert views on prospects for advancement. Energy 36, 595–605 (2011).

  85. 85.

    , , & Evaluation of potential cost reductions from improved amine-based CO2 capture systems. Energy Policy 34, 3765–3772 (2006).

  86. 86.

    , & Expert assessments of retrofitting coal-fired power plants with carbon dioxide capture technologies. Energy Policy 39, 5609–5620 (2011).

  87. 87.

    , , & Expert elicitation of cost, performance, and RD&D budgets for coal power with CCS. Energy Procedia 4, 2685–2692 (2011).

  88. 88.

    Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) (NRC, 2007). This report by the US National Research Council makes the case that public energy R&D decisions need to be informed by methods that account for the inherent uncertainty in the results of such investments.

  89. 89.

    , & Expert assessments of the cost of light water small modular reactors. Proc. Natl Acad. Sci. USA 110, 9686–9691 (2013).

  90. 90.

    , , , & Expert judgments about RD&D and the future of nuclear energy. Environ. Sci. Technol. 46, 11497–11504 (2012).

  91. 91.

    , & Advanced nuclear power: combining economic analysis with expert elicitations to inform climate policy. Preprint at (2008).

  92. 92.

    , & Expert assessments of future photovoltaic technologies. Environ. Sci. Technol. 42, 9031–9038 (2008).

  93. 93.

    , , & The future prospect of PV and CSP solar technologies: an expert elicitation survey. Energy Policy 49, 308–317 (2012).

  94. 94.

    How Low Will Photovoltaic Prices Go? An Expert Discussion (Near Zero, 2012).

  95. 95.

    , , & Going electric: expert survey on the future of battery technologies for electric vehicles. Energy Policy 61, 403–413 (2013).

  96. 96.

    Energy technology expert elicitations: an application to natural gas turbine efficiencies. Technol. Forecast. Soc. Change 86, 177–187 (2014).

  97. 97.

    Probabilistic decision model of wind power investment and influence of green power market. Energy Policy 63, 1111–1125 (2013).

  98. 98.

    et al. Expert elicitation survey on future wind energy costs. Nat. Energy 1, 16135 (2016).

  99. 99.

    , , & Characterizing the effects of policy instruments on the future costs of carbon capture for coal power plants. Clim. Change 133, 155–168 (2015).

  100. 100.

    & Demand subsidies versus R&D: comparing the uncertain impacts of policy on a pre-commercial low-carbon energy technology. Energy J. 30, 49–80 (2009).

  101. 101.

    & Electric sector investments under technological and policy-related uncertainties: a stochastic programming approach. Clim. Change 121, 143–160 (2013). This work uses a stochastic dynamic programming approach without relying on expert elicitations, and thus gives a sense of the literature that does not include such elicitations for parameterizing uncertainty.

  102. 102.

    et al. Technology interactions among low-carbon energy technologies: what can we learn from a large number of scenarios? Energy Econ. 33, 619–631 (2011).

  103. 103.

    , & Finding common ground when experts disagree: belief dominance over portfolios of alternatives Working Paper 46.2016 (FEEM, 2016).

  104. 104.

    , & Water–CO2 trade-offs in electricity generation planning. Nat. Clim. Change 3, 1029–1032 (2013).

  105. 105.

    , & (eds) Transforming US Energy Innovation (Cambridge Univ. Press, 2014).

  106. 106.

    , , , & Expert judgments about RD&D and the future of nuclear energy. Environ. Sci. Technol. 46, 11497–11504 (2012).

  107. 107.

    & Markal, a linear-programming model for energy systems analysis: technical description of the bnl version. Int. J. Energy Res. 5, 353–375 (1981).

  108. 108.

    et al. in Climate Change 1995: Economic and Social Dimensions of Climate Change (eds , & ) Ch. 10, 367–396 (IPCC, Cambridge Univ. Press, 1996).

  109. 109.

    Climate change policy: what do the models tell us? J. Econ. Lit. 51, 860–872 (2013).

  110. 110.

    A Question of Balance: Weighing the Options on Global Warming Policies. (Yale Univ. Press, 2014).

  111. 111.

    & Global energy and CO2 to the year 2050. Energy J. 4, 21–47 (1983).

  112. 112.

    , , , & WITCH A world induced technical change hybrid model. Energy J. 27, 13–37 (2006).

  113. 113.

    & Judgment under uncertainty: heuristics and biases. Science 185, 1124–1131 (1974).

  114. 114.

    et al. Tracking the Sun VIII: the Installed Price of Residential and Non-Residential Photovoltaic Systems in the United States (Lawrence Berkeley National Laboratory, 2015).

  115. 115.

    The aggregation of expert judgment: do good things come to those who weight? Risk Anal. 35, 12–15 (2015).

Download references


We are grateful to Nidhi Santen for useful insights and feedback on the description on ADP, to Franklyn Kanyako for assistance with the figures, and to Patricia McLaughlin for her work proofreading it. L.D.A. is grateful to the Climate Change Initiative of the Doris Duke Charitable Foundation and the Science, Technology and Public Policy programme at the Harvard Kennedy School for supporting early work on this paper and to the EU H2020 research and innovation programme under the Grant Agreement No 730403 (INNOPATHS). V.B. is grateful to funding from the European Research Council under the European Community’s Programme ‘Ideas’ — Call identifier: ERC-2013-StG / ERC grant agreement number 336703 — project RISICO ‘RISk and uncertainty in developing and Implementing Climate change pOlicies’. We also acknowledge financial support from the United Nations Environment Program through the Green Growth Knowledge Platform (GGKP) Research Committee on Technology and Innovation for the preparation of GGKP Working paper 01-2016, a report that helped us identify the need for this Review. E.B. acknowledges the NSF-sponsored IGERT: Offshore Wind Energy Engineering, Environmental Science, and Policy (NSF DGE 1068864).

Author information

Author notes

    • Laura Díaz Anadón
    • , Erin Baker
    •  & Valentina Bosetti

    These authors contributed equally to this work.


  1. Department of Politics and International Studies, University of Cambridge, 7 West Road, Cambridge CB3 9DT, UK.

    • Laura Díaz Anadón
  2. John F. Kennedy School of Government, Harvard University, 79 JFK Street, Cambridge, Massachusetts 02138, USA.

    • Laura Díaz Anadón
  3. Department of Mechanical and Industrial Engineering, 120 Governors Drive, University of Massachusetts Amherst, Amherst, Massachusetts 01002, USA.

    • Erin Baker
  4. Euro-Mediterranean Center on Climate Change and Fondazione Eni Enrico Mattei, Corso Magenta 63, 20123, Milano, Italy.

    • Valentina Bosetti
  5. Department of Economics, Bocconi University, via Sarfatti 25, 20136, Milano, Italy.

    • Valentina Bosetti


  1. Search for Laura Díaz Anadón in:

  2. Search for Erin Baker in:

  3. Search for Valentina Bosetti in:

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Laura Díaz Anadón.