Bringing rigour to energy innovation policy evaluation

Abstract

Clean energy innovation is pivotal for low-cost energy sector decarbonization. Substantial public research and development funding is spent on energy innovation. Generating more evidence on which support mechanisms most effectively drive clean energy innovations, and why, could improve their design moving forward. In this Perspective, we discuss five challenges that researchers often face when attempting to rigorously evaluate energy innovation policies and public subsidy programmes. We recommend solutions, such as developing new innovation outcome metrics that consider unique features of the energy sector and building databases that cover long time periods. We also suggest that researchers and funding agencies work together to implement randomized control trials or conduct quasi-experimental evaluation of existing programmes and policies wherever possible.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Evidence on spending, innovation and economic outcomes.

Data availability

Please see the Supplementary Information file for the data used to create Fig. 1.

References

  1. 1.

    Cunliff, C. Omission Innovation 2.0: Diagnosing the Global Clean Energy Innovation System (Information Technology & Innovation Foundation, 2019); https://itif.org/publications/2019/09/23/omission-innovation-20-diagnosing-global-clean-energy-innovation-system

  2. 2.

    Innovation: Grants, Loans and Subsidies (What Works Centre for Local Economic Growth, 2015); http://www.whatworksgrowth.org/public/files/Policy_Reviews/15-10-20-Innovation-Grants-Loans-Subsidies-Report.pdf.

  3. 3.

    Gallagher, K. S., Anadón, L. D., Kempener, R. & Wilson, C. Trends in global energy technology innovation. Wiley Interdiscip. Rev. Clim. Change 2, 373–396 (2011).

  4. 4.

    Grubler, A., Wilson, C. & Nemet, G. F. Apples, oranges, and consistent comparisons of the temporal dynamics of energy transitions. Energy Res. Soc. Sci. 22, 18–25 (2016).

  5. 5.

    Grubler, A. & Wilson, C. (eds) Energy Technology Innovation: Learning from Historical Successes and Failures (Cambridge Univ. Press, 2014).

  6. 6.

    Tracking Clean Energy Progress 2017 (International Energy Agency (IEA), 2017); https://www.iea.org/reports/tracking-clean-energy-progress-2017

  7. 7.

    Anadón, L. D., Gallagher, K. S. & Holdren, J. P. Rescue US energy innovation. Nat. Energy 2, 760–763 (2017).

  8. 8.

    An Assessment of ARPA-E (The National Academies Press, 2017); https://doi.org/10.17226/24778

  9. 9.

    ARPA-E impacts: a sampling of project outcomes, volume II. (US Department of Energy, 2017); https://arpa-e.energy.gov/sites/default/files/documents/files/Volume%202_ARPA-E_ImpactSheetCompilation_FINAL.pdf.

  10. 10.

    Jaffe, A. B., Newell, R. G. & Stavins, R. N. A tale of two market failures: technology and environmental policy. Ecol. Econ. 54, 164–174 (2005).

  11. 11.

    Dechezleprêtre, A., Martin, R. & Mohnen, M. Knowledge Spillovers From Clean and Dirty Technologies Working Paper 135 (Grantham Research Institute on Climate Change and the Environment, 2017).

  12. 12.

    Gaddy, B., Sivaram, V. & O’Sullivan, F. Venture Capital and Cleantech: The Wrong Model For Clean Energy Innovation Working Paper MITEI-WP-2016–06 (MIT Energy Initiative, 2016).

  13. 13.

    Sivaram, V. Unlocking clean energy. Issues in Science and Technology https://issues.org/unlocking-clean-energy/ (2017).

  14. 14.

    Nelson, R. R. The simple economics of basic scientific research. J. Polit. Econ. 67, 297–306 (1959).

  15. 15.

    Arrow, K. J. The economic implications of learning by doing. Rev. Econ. Stud. 29, 155–173 (1962).

  16. 16.

    Goulder, L. H. & Schneider, S. H. Induced technological change and the attractiveness of CO2 abatement policies. Resour. Energy Econ. 21, 211–52 (1999).

  17. 17.

    Heckman, J. H., Ichimura, H., Smith, J. & Todd, P. Characterizing selection bias using experimental data. Econometrica 66, 1017–98 (1998).

  18. 18.

    Jaffe, A. Building programme evaluation into the design of public research-support programmes. Oxford Rev. Econ. Policy 18, 22–34 (2002).

  19. 19.

    Duflo, E., Glennerster, R. & Kremer, M. Using Randomization in Development Economics Research: A Toolkit (National Bureau of Economic Research, 2016).

  20. 20.

    Athey, S. & Imbens, G. W. in Handbook of Economic Field Experiments (eds Banerjee, A. V. & Duflo, E.) Ch. 3 (ScienceDirect, 2017).

  21. 21.

    Lee, D. S. & Lemieux, T. Regression discontinuity designs in economics. J. Econ. Lit. 48, 281–355 (2010).

  22. 22.

    Howell, S. Financing innovation: evidence from R&D grants. Am. Econ. Rev. 107, 1136–1164 (2017).

  23. 23.

    Bronzini, R. & Iachini, E. Are incentives for R&D effective? Evidence from a regression discontinuity approach. Am. Econ. J. Econ. Policy 6, 100–134 (2014).

  24. 24.

    Agrawal A., Rosell C. & Simcoe, T. S. Tax credits and small firm R&D spending. Am. Econ. J. Econ. Policy (in the press).

  25. 25.

    Bronzini, R. & Piselli, P. The impact of R&D subsidies on firm innovation. Res. Policy 45, 442–457 (2016).

  26. 26.

    Dechezleprêtre A., Einiö E., Martin R., Nguyen K.-T. & Van Reenen J. Do Tax Incentives for Research Increase Firm Innovation? An RD Design For R&D CEP Discussion Paper No 1413 (Centre for Economic Performance, 2016).

  27. 27.

    Ganguli, I. Saving soviet science: the impact of grants when government R&D funding disappears. Am. Econ. J. Appl. Econ. 9, 165–201 (2017).

  28. 28.

    Wallsten, S. The effects of government-industry R&D programs on private R&D: the case of the Small Business Innovation Research Program. Rand J. Econ. 21, 82–100 (2000).

  29. 29.

    Bloom, N., Griffith, R. & van Reenen, J. Do R&D tax credits work? Evidence from a panel of countries 1979–1997. J. Public. Econ. 85, 1–31 (2002).

  30. 30.

    Rao, N. Do tax credits stimulate R&D spending? The effect of the R&D tax credit in its first decade. J. Public. Econ. 140, 1–12 (2016).

  31. 31.

    Lane, J. Assessing the impact of science funding. Science 324, 1273–1275 (2009).

  32. 32.

    Bonvillian, W. B. & Weiss, C. Technological Innovation in Legacy Sectors (Oxford University Press, 2015).

  33. 33.

    Popp, D. Economic analysis of scientific publications and implications for energy research and development. Nat. Energy 1, 16020 (2016).

  34. 34.

    Popp, D. The effect of new technology on energy consumption. Resour. Energy Econ. 23, 215–239 (2001).

  35. 35.

    Popp, D. Induced innovation and energy prices. Am. Econ. Rev. 92, 160–180 (2002).

  36. 36.

    Popp, D. They don’t invent them like they used to: an examination of energy patent citations over time. Econ. Innov. New Technol. 15, 753–776 (2006).

  37. 37.

    Johnstone, N., Haščič, I. & Popp, D. Renewable energy policies and technological innovation: evidence based on patent counts. Environ. Resour. Econ. (Dordr) 45, 133–155 (2010).

  38. 38.

    Verdolini, E. & Gaelotti, M. At home and abroad: an empirical analysis of innovation and diffusion in energy technologies. J. Environ. Econ. Manage. 61, 119–134 (2011).

  39. 39.

    Einiö, E. R&D subsidies and company performance: evidence from geographic variation in government funding based on the ERDF population-density rule. Rev. Econ. Stat. 96, 710–728 (2014).

  40. 40.

    Czarnitzki, D., Hanel, P. & Rosa, J. M. Evaluating the impact of R&D tax credits on innovation: a microeconometric study on Canadian firms. Res. Policy 40, 217–229 (2011).

  41. 41.

    Colombo, M. G., Grilli, L. & Murtinu, S. R&D subsidies and the performance of high-tech start-ups. Economics Letters 112, 97–99 (2011).

  42. 42.

    Acemoglu, D., Aghion, P., Bursztyn, L. & Hemous, D. The environment and directed technical change. American Economic Review 102, 131–166 (2012).

  43. 43.

    Acemoglu, D., Akcigit, U., Hanley, D. & Kerr, W. Transition to clean technology. J. Polit. Econ. 124, 52–104 (2016).

  44. 44.

    Kattel, R. & Mazzucato, M. Mission-oriented innovation policy and dynamic capabilities in the public sector. Ind. Corp. Change 27, 787–801 (2018).

  45. 45.

    Aghion, P., Dechezlepretre, A., Hemous, D., Martin, R. & van Reenen, J. Carbon taxes, path dependency, and directed technical change: evidence from the auto industry. J. Polit. Econ. 124, 1–51 (2016).

  46. 46.

    Goulder, L. H. & Parry, I. Instrument choice in environmental policy. Rev. Environ. Econ. Policy 2, 152–74 (2008).

  47. 47.

    Fischer, C. & Newell, R. G. Environmental and technology policies for climate mitigation. J. Environ. Econ. Manage. 55, 142–62 (2008).

  48. 48.

    Fischer, C., Preonas, L. & Newell, R. Environmental and technology policy options in the electricity sector: are we deploying too many? J. Assoc. Environ. Resour. Econ. 4, 959–984 (2017).

  49. 49.

    Adam, D. Science funders gamble on grant lotteries. Nature 575, 574–575 (2019).

  50. 50.

    IGL Trials Database (Innovation Growth Lab, 2019); https://www.innovationgrowthlab.org/igl-database

  51. 51.

    Afcha, S. Analyzing the interaction between R&D subsidies and firm’s innovation strategy. Journal of Technol. Manag. Innov. 7, 57–70 (2012).

  52. 52.

    Antonioli, D., Marzucchi, A. & Montresor, S. Regional innovation policy and innovative behaviour: Looking for Additional Effects. Eur. Plan. Stud. 22, 64–83 (2014).

  53. 53.

    Branstetter, L. & Sakakibara, M. Japanese research consortia: a microeconometric analysis of industrial policy. J. Ind. Econ. 46, 207–233 (2003).

  54. 54.

    Coupé, T. Science is golden: academic R&D and university patents. J. Technol. Transf. 28, 31–46 (2003).

  55. 55.

    Czarnitzki, D., Ebersberger, B. & Fier, A. The relationship between R&D collaboration, subsidies and R&D performance: empirical evidence from Finland and Germany. J. Appl. Econ. 22, 1347–1366 (2007).

  56. 56.

    Kaiser, U. & Kuh, J. Long-Run Effects of Public-Private Research Joint Ventures: the Case of The Danish Innovation Consortia Support Scheme IZA Discussion Paper 5986 (IZA Institute for Labor Economics, 2011).

  57. 57.

    Nishimura, J. & Okamuro, H. Subsidy and networking: the effects of direct and indirect support programs of the cluster policy. Res. Policy 40, 714–727 (2011).

  58. 58.

    Teirlinck, P. & Spithoven, A. Fostering industry-science cooperation through public funding: differences between universities and public research centres. J. Technol. Transf. 37, 676–695 (2013).

  59. 59.

    Lechevalier, S., Ikeda, Y. & Nishimura, J. The effect of participation in government consortia on the R&D productivity of firms: a case study of robot technology in Japan. Discussion Paper Series A 500 (2008).

  60. 60.

    Aguiar, L. & Gagnepain, P. European Cooperative R&D and Firm Performance: Evidence Based on Funding Differences in Key Actions CEPR Discussion Paper DP9426 (Paris School of Economics, 2013).

  61. 61.

    Aerts, K. & Schmidt, T. Two For the Price of One? On Additionality Effects Of R&D Subsidies: A Comparison Between Flanders And Germany ZEW Discussion Paper no. 06-063 (ZEW - Centre for European Economic Research Discussion, 2006).

  62. 62.

    Bayona-Sáez, C. & García-Marco, T. Assessing the effectiveness of the Eureka program. Res. Policy 39, 1375–1386 (2010).

  63. 63.

    Benavente, J. M., Crespi, G. & Maffioli, A. Public Support to Firm-Level Innovation: An Evaluation of the FONTEC Program OVE Working Paper 05 07 (Office of Evaluation and Oversight, 2007).

  64. 64.

    Benavente, J. M., Crespi, G., Garone, L. F. & Maffioli, A. The impact of national research funds: a regression discontinuity approach to the chilean FONDECYT. Res. Policy 41, 1467–1475 (2012).

  65. 65.

    Callejon, M. & Garcia-Quevedo, J. Public subsidies to business R&D: do they stimulate private expenditures? Environ. Plann. C Gov. Policy 23, 279–293 (2005).

  66. 66.

    Cannone, G. & Ughetto, E. Funding innovation at regional level: an analysis of a public policy intervention in the Piedmont region. Reg. Stud. 48, 270–283 (2012).

  67. 67.

    Economic Impact of International Research and Innovation Cooperation - Analysis of 25 years of Danish participation in EUREKA (Danish Agency for Science Technology and Innovation, 2011).

  68. 68.

    Duch, N., Montolio, D. & Mediavilla, M. Evaluating the impact of public subsidies on a firm’s performance: A two-stage quasi-experimental approach. Investigaciones Regionales 16, 143–165 (2009).

  69. 69.

    Dumont, M. The Impact of Subsidies and Fiscal Incentives on Corporate R&D Expenditures in Belgium (2001–2009) Federal Planning Bureau Working Paper 1–13 (De Boeck Université, 2013).

  70. 70.

    Foreman-Peck, J. Effectiveness and Efficiency of SME Innovation Policy Cardiff Economics Working Papers E2012/4 (Cardiff Business School, 2012).

  71. 71.

    Fornahl, D., Broekel, T. & Boschma, R. What drives patent performance of German biotech firms? the impact of R&D subsidies, knowledge networks and their location. Reg. Sci. 90, 395–419 (2011).

  72. 72.

    Gonzalez, X. & Pazo, C. Do public subsidies stimulate private R&D spending? Res. Policy 37, 371–389 (2008).

  73. 73.

    Görg, H. & Strobl, E. The Effect of R&D Subsidies on Private R&D Globalisation, Productivity and Technology Research Paper 2005/38 (University of Nottingham, 2005).

  74. 74.

    Grilli, L. & Murtinu, S. Do public subsidies affect the performance of new technology-based firms? The importance of evaluation schemes and agency goals. Prometheus: Crit. Stud. Innovation 30, 97–111 (2012).

  75. 75.

    Hewitt-Dundas, N. & Roper, S. Output Addtionality of Public Support for Innovation: Evidence for Irish Manufacturing Plants Working Paper No. 103 (Warwick Business School’s Small and Medium Sized Enterprise Centre, 2009).

  76. 76.

    Hujer, R. & Dubravko, R. Evaluating the Impacts of Subsidies on Innovation Activities in Germany ZEW Discussion Paper 05–43 (ZEW, 2005).

  77. 77.

    Kolympiris, C., Kalaitzandonakes, N. & Miller, D. Public funds and local biotechnology firm creation. Res. Policy 43, 121–137 (2014). (2014).

  78. 78.

    Lach, S. Do R&D subsidies stimulate or displace private R&D? Evidence from Israel. J. Ind. Econ. 50, 369–390 (2002).

  79. 79.

    Merito, M., Giannangeli, S. & Bonaccorsi, A. Do incentives to industrial R&D enhance research productivity and firm growth? Evidence from the Italian Case. L’industria 2, 221–242 (2007).

  80. 80.

    Moretti, E. & Wilson, D. J. State incentives for innovation, star scientists and jobs: evidence from biotech. J. Urban Econ. 79, 20–38 (2014).

  81. 81.

    Morris, M. & Herrmann, O. J. Beyond surveys: the research frontier moves to the use of administrative data to evaluate R&D Grants. Res. Eval. 22, 298–306 (2013).

  82. 82.

    Sissoko, A. R&D Subsidies and Firm-Level Productivity: Evidence From France Institut de Recherches Economiques et Sociales Discussion Paper 2011–2 (IRES, 2013).

  83. 83.

    Wu, Y. NSF’s experimental program to stimulate competitive research (epscor): subsidizing academic research or state budgets? J. Policy Anal. Manage. 28, 479–495 (2009).

  84. 84.

    Fantino, D. & Cannone, G. Evaluating the Efficacy of European Regional Funds for R&D Working Paper no. 902 (Bank of Italy, 2013).

  85. 85.

    Romero-Jordan, D., Delgrado-Rodriguez, M., Alvaerz-Ayuso, I. & de Lucas-Santos, S. Assessment of the Public Tools Used to Promote R&D Investment in Spanish SMEs. Small Bus. Econ. 43, 959–976 (2014).

  86. 86.

    Broekel, T., Brachert, M., Duschl, M. & Brenner, T. Joint R&D subsidies, related variety, and regional innovation. Int. Regional Sci. Rev. https://doi.org/10.1177/0160017615589007 (2015).

  87. 87.

    Henningsen, M., Hægeland, T. & Møen, J. Estimating the additionality of R&D subsidies using proposal evaluation data to control for research intentions. J. Technol. Transfer 40, 227–251 (2015).

  88. 88.

    Azoulay, P., Graff Zivin, J., Li, D. & Sampat, B. Public R&D investments and private-sector patenting: evidence from NIH funding rules. Rev. Econ. Stud. 86, 117–152 (2019).

  89. 89.

    Le, T. & Jaffe, A. B. The impact of R&D subsidy on innovation: evidence from New Zealand firms. Econ. Innov. New Technol. 26, 429–452 (2017).

Download references

Acknowledgements

We are grateful to D. Popp and J. Rhys for comments on an early version of this Perspective. The authors gratefully acknowledge the Oxford Martin Programme on Integrating Renewable Energy at the Oxford Martin School for financial support. N.F. also acknowledges funding through the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement no. 743582.

Author information

All three authors contributed to the writing of this paper.

Correspondence to Jacquelyn Pless.

Ethics declarations

Competing interests

C.H. is the director of Aurora Energy Research Limited, an energy analytics firm, Vivid Economics Limited, an economics consultancy firm, has several clients in the energy sector and has had academic funding from Shell.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Source data

Source Data Fig. 1

Data points for all the papers in Fig. 1.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pless, J., Hepburn, C. & Farrell, N. Bringing rigour to energy innovation policy evaluation. Nat Energy (2020). https://doi.org/10.1038/s41560-020-0557-1

Download citation