Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Bringing rigour to energy innovation policy evaluation

Nature Energy volume 5, pages 284–290 (2020)Cite this article

Subjects

Abstract

Clean energy innovation is pivotal for low-cost energy sector decarbonization. Substantial public research and development funding is spent on energy innovation. Generating more evidence on which support mechanisms most effectively drive clean energy innovations, and why, could improve their design moving forward. In this Perspective, we discuss five challenges that researchers often face when attempting to rigorously evaluate energy innovation policies and public subsidy programmes. We recommend solutions, such as developing new innovation outcome metrics that consider unique features of the energy sector and building databases that cover long time periods. We also suggest that researchers and funding agencies work together to implement randomized control trials or conduct quasi-experimental evaluation of existing programmes and policies wherever possible.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Evidence on spending, innovation and economic outcomes.

Similar content being viewed by others

Data availability

Please see the Supplementary Information file for the data used to create Fig. 1.

References

  1. Cunliff, C. Omission Innovation 2.0: Diagnosing the Global Clean Energy Innovation System (Information Technology & Innovation Foundation, 2019); https://itif.org/publications/2019/09/23/omission-innovation-20-diagnosing-global-clean-energy-innovation-system

  2. Innovation: Grants, Loans and Subsidies (What Works Centre for Local Economic Growth, 2015); http://www.whatworksgrowth.org/public/files/Policy_Reviews/15-10-20-Innovation-Grants-Loans-Subsidies-Report.pdf.

  3. Gallagher, K. S., Anadón, L. D., Kempener, R. & Wilson, C. Trends in global energy technology innovation. Wiley Interdiscip. Rev. Clim. Change 2, 373–396 (2011).

    Google Scholar 

  4. Grubler, A., Wilson, C. & Nemet, G. F. Apples, oranges, and consistent comparisons of the temporal dynamics of energy transitions. Energy Res. Soc. Sci. 22, 18–25 (2016).

    Google Scholar 

  5. Grubler, A. & Wilson, C. (eds) Energy Technology Innovation: Learning from Historical Successes and Failures (Cambridge Univ. Press, 2014).

  6. Tracking Clean Energy Progress 2017 (International Energy Agency (IEA), 2017); https://www.iea.org/reports/tracking-clean-energy-progress-2017

  7. Anadón, L. D., Gallagher, K. S. & Holdren, J. P. Rescue US energy innovation. Nat. Energy 2, 760–763 (2017).

    Google Scholar 

  8. An Assessment of ARPA-E (The National Academies Press, 2017); https://doi.org/10.17226/24778

  9. ARPA-E impacts: a sampling of project outcomes, volume II. (US Department of Energy, 2017); https://arpa-e.energy.gov/sites/default/files/documents/files/Volume%202_ARPA-E_ImpactSheetCompilation_FINAL.pdf.

  10. Jaffe, A. B., Newell, R. G. & Stavins, R. N. A tale of two market failures: technology and environmental policy. Ecol. Econ. 54, 164–174 (2005).

    Google Scholar 

  11. Dechezleprêtre, A., Martin, R. & Mohnen, M. Knowledge Spillovers From Clean and Dirty Technologies Working Paper 135 (Grantham Research Institute on Climate Change and the Environment, 2017).

  12. Gaddy, B., Sivaram, V. & O’Sullivan, F. Venture Capital and Cleantech: The Wrong Model For Clean Energy Innovation Working Paper MITEI-WP-2016–06 (MIT Energy Initiative, 2016).

  13. Sivaram, V. Unlocking clean energy. Issues in Science and Technology https://issues.org/unlocking-clean-energy/ (2017).

  14. Nelson, R. R. The simple economics of basic scientific research. J. Polit. Econ. 67, 297–306 (1959).

    Google Scholar 

  15. Arrow, K. J. The economic implications of learning by doing. Rev. Econ. Stud. 29, 155–173 (1962).

    Google Scholar 

  16. Goulder, L. H. & Schneider, S. H. Induced technological change and the attractiveness of CO2 abatement policies. Resour. Energy Econ. 21, 211–52 (1999).

    Google Scholar 

  17. Heckman, J. H., Ichimura, H., Smith, J. & Todd, P. Characterizing selection bias using experimental data. Econometrica 66, 1017–98 (1998).

    MathSciNet  MATH  Google Scholar 

  18. Jaffe, A. Building programme evaluation into the design of public research-support programmes. Oxford Rev. Econ. Policy 18, 22–34 (2002).

    Google Scholar 

  19. Duflo, E., Glennerster, R. & Kremer, M. Using Randomization in Development Economics Research: A Toolkit (National Bureau of Economic Research, 2016).

  20. Athey, S. & Imbens, G. W. in Handbook of Economic Field Experiments (eds Banerjee, A. V. & Duflo, E.) Ch. 3 (ScienceDirect, 2017).

  21. Lee, D. S. & Lemieux, T. Regression discontinuity designs in economics. J. Econ. Lit. 48, 281–355 (2010).

    Google Scholar 

  22. Howell, S. Financing innovation: evidence from R&D grants. Am. Econ. Rev. 107, 1136–1164 (2017).

    Google Scholar 

  23. Bronzini, R. & Iachini, E. Are incentives for R&D effective? Evidence from a regression discontinuity approach. Am. Econ. J. Econ. Policy 6, 100–134 (2014).

    Google Scholar 

  24. Agrawal A., Rosell C. & Simcoe, T. S. Tax credits and small firm R&D spending. Am. Econ. J. Econ. Policy (in the press).

  25. Bronzini, R. & Piselli, P. The impact of R&D subsidies on firm innovation. Res. Policy 45, 442–457 (2016).

    Google Scholar 

  26. Dechezleprêtre A., Einiö E., Martin R., Nguyen K.-T. & Van Reenen J. Do Tax Incentives for Research Increase Firm Innovation? An RD Design For R&D CEP Discussion Paper No 1413 (Centre for Economic Performance, 2016).

  27. Ganguli, I. Saving soviet science: the impact of grants when government R&D funding disappears. Am. Econ. J. Appl. Econ. 9, 165–201 (2017).

    Google Scholar 

  28. Wallsten, S. The effects of government-industry R&D programs on private R&D: the case of the Small Business Innovation Research Program. Rand J. Econ. 21, 82–100 (2000).

    Google Scholar 

  29. Bloom, N., Griffith, R. & van Reenen, J. Do R&D tax credits work? Evidence from a panel of countries 1979–1997. J. Public. Econ. 85, 1–31 (2002).

    Google Scholar 

  30. Rao, N. Do tax credits stimulate R&D spending? The effect of the R&D tax credit in its first decade. J. Public. Econ. 140, 1–12 (2016).

    Google Scholar 

  31. Lane, J. Assessing the impact of science funding. Science 324, 1273–1275 (2009).

    Google Scholar 

  32. Bonvillian, W. B. & Weiss, C. Technological Innovation in Legacy Sectors (Oxford University Press, 2015).

  33. Popp, D. Economic analysis of scientific publications and implications for energy research and development. Nat. Energy 1, 16020 (2016).

    Google Scholar 

  34. Popp, D. The effect of new technology on energy consumption. Resour. Energy Econ. 23, 215–239 (2001).

    Google Scholar 

  35. Popp, D. Induced innovation and energy prices. Am. Econ. Rev. 92, 160–180 (2002).

    Google Scholar 

  36. Popp, D. They don’t invent them like they used to: an examination of energy patent citations over time. Econ. Innov. New Technol. 15, 753–776 (2006).

    Google Scholar 

  37. Johnstone, N., Haščič, I. & Popp, D. Renewable energy policies and technological innovation: evidence based on patent counts. Environ. Resour. Econ. (Dordr) 45, 133–155 (2010).

    Google Scholar 

  38. Verdolini, E. & Gaelotti, M. At home and abroad: an empirical analysis of innovation and diffusion in energy technologies. J. Environ. Econ. Manage. 61, 119–134 (2011).

    Google Scholar 

  39. Einiö, E. R&D subsidies and company performance: evidence from geographic variation in government funding based on the ERDF population-density rule. Rev. Econ. Stat. 96, 710–728 (2014).

    Google Scholar 

  40. Czarnitzki, D., Hanel, P. & Rosa, J. M. Evaluating the impact of R&D tax credits on innovation: a microeconometric study on Canadian firms. Res. Policy 40, 217–229 (2011).

    Google Scholar 

  41. Colombo, M. G., Grilli, L. & Murtinu, S. R&D subsidies and the performance of high-tech start-ups. Economics Letters 112, 97–99 (2011).

    Google Scholar 

  42. Acemoglu, D., Aghion, P., Bursztyn, L. & Hemous, D. The environment and directed technical change. American Economic Review 102, 131–166 (2012).

    Google Scholar 

  43. Acemoglu, D., Akcigit, U., Hanley, D. & Kerr, W. Transition to clean technology. J. Polit. Econ. 124, 52–104 (2016).

    Google Scholar 

  44. Kattel, R. & Mazzucato, M. Mission-oriented innovation policy and dynamic capabilities in the public sector. Ind. Corp. Change 27, 787–801 (2018).

    Google Scholar 

  45. Aghion, P., Dechezlepretre, A., Hemous, D., Martin, R. & van Reenen, J. Carbon taxes, path dependency, and directed technical change: evidence from the auto industry. J. Polit. Econ. 124, 1–51 (2016).

    Google Scholar 

  46. Goulder, L. H. & Parry, I. Instrument choice in environmental policy. Rev. Environ. Econ. Policy 2, 152–74 (2008).

    Google Scholar 

  47. Fischer, C. & Newell, R. G. Environmental and technology policies for climate mitigation. J. Environ. Econ. Manage. 55, 142–62 (2008).

    MATH  Google Scholar 

  48. Fischer, C., Preonas, L. & Newell, R. Environmental and technology policy options in the electricity sector: are we deploying too many? J. Assoc. Environ. Resour. Econ. 4, 959–984 (2017).

    Google Scholar 

  49. Adam, D. Science funders gamble on grant lotteries. Nature 575, 574–575 (2019).

    Google Scholar 

  50. IGL Trials Database (Innovation Growth Lab, 2019); https://www.innovationgrowthlab.org/igl-database

  51. Afcha, S. Analyzing the interaction between R&D subsidies and firm’s innovation strategy. Journal of Technol. Manag. Innov. 7, 57–70 (2012).

    Google Scholar 

  52. Antonioli, D., Marzucchi, A. & Montresor, S. Regional innovation policy and innovative behaviour: Looking for Additional Effects. Eur. Plan. Stud. 22, 64–83 (2014).

    Google Scholar 

  53. Branstetter, L. & Sakakibara, M. Japanese research consortia: a microeconometric analysis of industrial policy. J. Ind. Econ. 46, 207–233 (2003).

    Google Scholar 

  54. Coupé, T. Science is golden: academic R&D and university patents. J. Technol. Transf. 28, 31–46 (2003).

    Google Scholar 

  55. Czarnitzki, D., Ebersberger, B. & Fier, A. The relationship between R&D collaboration, subsidies and R&D performance: empirical evidence from Finland and Germany. J. Appl. Econ. 22, 1347–1366 (2007).

    MathSciNet  Google Scholar 

  56. Kaiser, U. & Kuh, J. Long-Run Effects of Public-Private Research Joint Ventures: the Case of The Danish Innovation Consortia Support Scheme IZA Discussion Paper 5986 (IZA Institute for Labor Economics, 2011).

  57. Nishimura, J. & Okamuro, H. Subsidy and networking: the effects of direct and indirect support programs of the cluster policy. Res. Policy 40, 714–727 (2011).

    Google Scholar 

  58. Teirlinck, P. & Spithoven, A. Fostering industry-science cooperation through public funding: differences between universities and public research centres. J. Technol. Transf. 37, 676–695 (2013).

    Google Scholar 

  59. Lechevalier, S., Ikeda, Y. & Nishimura, J. The effect of participation in government consortia on the R&D productivity of firms: a case study of robot technology in Japan. Discussion Paper Series A 500 (2008).

  60. Aguiar, L. & Gagnepain, P. European Cooperative R&D and Firm Performance: Evidence Based on Funding Differences in Key Actions CEPR Discussion Paper DP9426 (Paris School of Economics, 2013).

  61. Aerts, K. & Schmidt, T. Two For the Price of One? On Additionality Effects Of R&D Subsidies: A Comparison Between Flanders And Germany ZEW Discussion Paper no. 06-063 (ZEW - Centre for European Economic Research Discussion, 2006).

  62. Bayona-Sáez, C. & García-Marco, T. Assessing the effectiveness of the Eureka program. Res. Policy 39, 1375–1386 (2010).

    Google Scholar 

  63. Benavente, J. M., Crespi, G. & Maffioli, A. Public Support to Firm-Level Innovation: An Evaluation of the FONTEC Program OVE Working Paper 05 07 (Office of Evaluation and Oversight, 2007).

  64. Benavente, J. M., Crespi, G., Garone, L. F. & Maffioli, A. The impact of national research funds: a regression discontinuity approach to the chilean FONDECYT. Res. Policy 41, 1467–1475 (2012).

    Google Scholar 

  65. Callejon, M. & Garcia-Quevedo, J. Public subsidies to business R&D: do they stimulate private expenditures? Environ. Plann. C Gov. Policy 23, 279–293 (2005).

    Google Scholar 

  66. Cannone, G. & Ughetto, E. Funding innovation at regional level: an analysis of a public policy intervention in the Piedmont region. Reg. Stud. 48, 270–283 (2012).

    Google Scholar 

  67. Economic Impact of International Research and Innovation Cooperation - Analysis of 25 years of Danish participation in EUREKA (Danish Agency for Science Technology and Innovation, 2011).

  68. Duch, N., Montolio, D. & Mediavilla, M. Evaluating the impact of public subsidies on a firm’s performance: A two-stage quasi-experimental approach. Investigaciones Regionales 16, 143–165 (2009).

    Google Scholar 

  69. Dumont, M. The Impact of Subsidies and Fiscal Incentives on Corporate R&D Expenditures in Belgium (2001–2009) Federal Planning Bureau Working Paper 1–13 (De Boeck Université, 2013).

  70. Foreman-Peck, J. Effectiveness and Efficiency of SME Innovation Policy Cardiff Economics Working Papers E2012/4 (Cardiff Business School, 2012).

  71. Fornahl, D., Broekel, T. & Boschma, R. What drives patent performance of German biotech firms? the impact of R&D subsidies, knowledge networks and their location. Reg. Sci. 90, 395–419 (2011).

    Google Scholar 

  72. Gonzalez, X. & Pazo, C. Do public subsidies stimulate private R&D spending? Res. Policy 37, 371–389 (2008).

    Google Scholar 

  73. Görg, H. & Strobl, E. The Effect of R&D Subsidies on Private R&D Globalisation, Productivity and Technology Research Paper 2005/38 (University of Nottingham, 2005).

  74. Grilli, L. & Murtinu, S. Do public subsidies affect the performance of new technology-based firms? The importance of evaluation schemes and agency goals. Prometheus: Crit. Stud. Innovation 30, 97–111 (2012).

    Google Scholar 

  75. Hewitt-Dundas, N. & Roper, S. Output Addtionality of Public Support for Innovation: Evidence for Irish Manufacturing Plants Working Paper No. 103 (Warwick Business School’s Small and Medium Sized Enterprise Centre, 2009).

  76. Hujer, R. & Dubravko, R. Evaluating the Impacts of Subsidies on Innovation Activities in Germany ZEW Discussion Paper 05–43 (ZEW, 2005).

  77. Kolympiris, C., Kalaitzandonakes, N. & Miller, D. Public funds and local biotechnology firm creation. Res. Policy 43, 121–137 (2014). (2014).

    Google Scholar 

  78. Lach, S. Do R&D subsidies stimulate or displace private R&D? Evidence from Israel. J. Ind. Econ. 50, 369–390 (2002).

    Google Scholar 

  79. Merito, M., Giannangeli, S. & Bonaccorsi, A. Do incentives to industrial R&D enhance research productivity and firm growth? Evidence from the Italian Case. L’industria 2, 221–242 (2007).

    Google Scholar 

  80. Moretti, E. & Wilson, D. J. State incentives for innovation, star scientists and jobs: evidence from biotech. J. Urban Econ. 79, 20–38 (2014).

    Google Scholar 

  81. Morris, M. & Herrmann, O. J. Beyond surveys: the research frontier moves to the use of administrative data to evaluate R&D Grants. Res. Eval. 22, 298–306 (2013).

    Google Scholar 

  82. Sissoko, A. R&D Subsidies and Firm-Level Productivity: Evidence From France Institut de Recherches Economiques et Sociales Discussion Paper 2011–2 (IRES, 2013).

  83. Wu, Y. NSF’s experimental program to stimulate competitive research (epscor): subsidizing academic research or state budgets? J. Policy Anal. Manage. 28, 479–495 (2009).

    Google Scholar 

  84. Fantino, D. & Cannone, G. Evaluating the Efficacy of European Regional Funds for R&D Working Paper no. 902 (Bank of Italy, 2013).

  85. Romero-Jordan, D., Delgrado-Rodriguez, M., Alvaerz-Ayuso, I. & de Lucas-Santos, S. Assessment of the Public Tools Used to Promote R&D Investment in Spanish SMEs. Small Bus. Econ. 43, 959–976 (2014).

    Google Scholar 

  86. Broekel, T., Brachert, M., Duschl, M. & Brenner, T. Joint R&D subsidies, related variety, and regional innovation. Int. Regional Sci. Rev. https://doi.org/10.1177/0160017615589007 (2015).

  87. Henningsen, M., Hægeland, T. & Møen, J. Estimating the additionality of R&D subsidies using proposal evaluation data to control for research intentions. J. Technol. Transfer 40, 227–251 (2015).

    Google Scholar 

  88. Azoulay, P., Graff Zivin, J., Li, D. & Sampat, B. Public R&D investments and private-sector patenting: evidence from NIH funding rules. Rev. Econ. Stud. 86, 117–152 (2019).

    MATH  Google Scholar 

  89. Le, T. & Jaffe, A. B. The impact of R&D subsidy on innovation: evidence from New Zealand firms. Econ. Innov. New Technol. 26, 429–452 (2017).

    Google Scholar 

Download references

Acknowledgements

We are grateful to D. Popp and J. Rhys for comments on an early version of this Perspective. The authors gratefully acknowledge the Oxford Martin Programme on Integrating Renewable Energy at the Oxford Martin School for financial support. N.F. also acknowledges funding through the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement no. 743582.

Author information

Authors and Affiliations

  1. MIT Sloan School of Management, Cambridge, MA, USA

    Jacquelyn Pless

  2. Smith School of Enterprise and the Environment, University of Oxford, Oxford, UK

    Jacquelyn Pless & Cameron Hepburn

  3. Institute for New Economic Thinking at the Oxford Martin School, University of Oxford, Oxford, UK

    Cameron Hepburn

  4. Grantham Research Institute, London School of Economics and Political Science, London, UK

    Cameron Hepburn

  5. Queen’s Management School, Queen’s University Belfast, Belfast, UK

    Niall Farrell

  6. Potsdam Institute for Climate Impact Research (PIK), Potsdam, Germany

    Niall Farrell

Authors
  1. Jacquelyn Pless
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cameron Hepburn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Niall Farrell
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All three authors contributed to the writing of this paper.

Corresponding author

Correspondence to Jacquelyn Pless.

Ethics declarations

Competing interests

C.H. is the director of Aurora Energy Research Limited, an energy analytics firm, Vivid Economics Limited, an economics consultancy firm, has several clients in the energy sector and has had academic funding from Shell.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Source data

Source Data Fig. 1

Data points for all the papers in Fig. 1.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pless, J., Hepburn, C. & Farrell, N. Bringing rigour to energy innovation policy evaluation. Nat Energy 5, 284–290 (2020). https://doi.org/10.1038/s41560-020-0557-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41560-020-0557-1

This article is cited by

Search

Advanced search

Quick links

Nature Briefing Anthropocene

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: Anthropocene