Burning embers: towards more transparent and robust climate-change risk assessments

Abstract

The Intergovernmental Panel on Climate Change (IPCC) reports provide policy-relevant insights about climate impacts, vulnerabilities and adaptation through a process of peer-reviewed literature assessments underpinned by expert judgement. An iconic output from these assessments is the burning embers diagram, first used in the Third Assessment Report to visualize reasons for concern, which aggregate climate-change-related impacts and risks to various systems and sectors. These burning embers use colour transitions to show changes in the assessed level of risk to humans and ecosystems as a function of global mean temperature. In this Review, we outline the history and evolution of the burning embers and associated reasons for concern framework, focusing on the methodological approaches and advances. While the assessment framework and figure design have been broadly retained over time, refinements in methodology have occurred, including the consideration of different risks, use of confidence statements, more formalized protocols and standardized metrics. Comparison across reports reveals that the risk level at a given temperature has generally increased with each assessment cycle, reflecting accumulating scientific evidence. For future assessments, an explicit, transparent and systematic process of expert elicitation is needed to enhance comparability, quality and credibility of burning embers.

Key points

  • The Intergovernmental Panel on Climate Change (IPCC) has used the reasons for concern framework and burning embers diagrams since 2001 to assess and communicate risks from increasing global mean temperature on human and natural systems.

  • The framework and burning embers diagrams are developed using expert judgement, based on available information about climate impacts.

  • While assessment methods and figure design have been broadly retained across IPCC reports, risk levels at given temperatures have generally increased over time, owing to more comprehensive science.

  • Structured expert-elicitation methods can reduce bias and increase reproducibility by specifying the process for selecting experts, providing external information, eliciting individual and consensus judgements, and facilitating group interaction.

  • Recent IPCC Special Reports introduced formal protocols and standardized metrics to elicit risk thresholds. Despite challenges, these changes contributed to transparency and reliability.

  • Further development and use of standardized and transparent methods to elicit risk thresholds and build burning embers diagrams will continue to increase the robustness and credibility of IPCC assessments.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Third Assessment Report representation of burning embers and the reasons for concern framework.
Fig. 2: The evolution of the burning embers diagram.
Fig. 3: Comparison of risk thresholds across Intergovernmental Panel on Climate Change assessments.
Fig. 4: Flow chart of expert elicitation used in the Special Report on Climate Change and Land.
Fig. 5: Expert elicitation for the food stability ember in the Special Report on Climate Change and Land.
Fig. 6: Flow chart of sea-level-rise risk assessment in the Special Report on the Ocean and Cryosphere in a Changing Climate.

References

  1. 1.

    Lenton, T. M. et al. Climate tipping points — too risky to bet against. Nature 575, 592–595 (2019).

    Google Scholar 

  2. 2.

    United Nations. United Nations Framework Convention on Climate Change (1992).

  3. 3.

    Garner, G., Reed, P. & Keller, K. Climate risk management requires explicit representation of societal trade-offs. Clim. Change 134, 713–723 (2016).

    Google Scholar 

  4. 4.

    Nordhaus, W. D. Managing the Global Commons: The Economics of Climate Change (MIT Press, 1994).

  5. 5.

    Leemans, R. & Vellinga, P. The scientific motivation of the internationally agreed ‘well below 2 °C’ climate protection target: a historical perspective. Curr. Opin. Environ. Sustain. 26–27, 134–142 (2017).

    Google Scholar 

  6. 6.

    Fischlin, A., Ji, Z., Vladu, F. & Bisiaux, A. Report on the Structured Expert Dialogue on the 2013–2015 Review of the United Nations Framework Convention on Climate Change (UNFCCC). Final Report FCCC/SB/2015/INF.1 (UNFCCC Secretariat, 2015).

  7. 7.

    Smith, J. et al. in Climate Change 2001: Impacts, Adaptation, and Vulnerability. Contribution of Working Group II to the Third Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) Ch. 19 (eds. McCarthy, J. J., Canziani, O. F., Leary, N. A., Dokken, D. J. & White, K. S.) 913–967 (Cambridge Univ. Press, 2001).

  8. 8.

    McCarthy J. J., Canziani O., Leary N. A., Dokken D. J. & White K. S. (eds) Climate Change 2001: Impacts, Adaptation, and Vulnerability. Contribution of Working Group II to the Third Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) (Cambridge Univ. Press, 2001).

  9. 9.

    O’Neill, B. C. et al. IPCC reasons for concern regarding climate change risks. Nat. Clim. Change 7, 28–37 (2017).

    Google Scholar 

  10. 10.

    Matthews, J. B. R. in Global Warming of 1.5°C. An IPCC Special Report on the Impacts of Global Warming of 1.5°C Above Pre-industrial Levels and Related Global Greenhouse Gas Emission Pathways, in the Context of Strengthening the Global Response to the Threat of Climate Change, Sustainable Development, and Efforts to Eradicate Poverty (eds Masson-Delmotte, V. et al.) 539–562 (IPCC, 2018).

  11. 11.

    Paris Agreement. United Nations Treaty Collection. United Nations https://treaties.un.org/doc/Treaties/2016/02/20160215%2006-03%20PM/Ch_XXVII-7-d.pdf (2016).

  12. 12.

    Intergovernmental Panel on Climate Change (IPCC). Chapter Outline of the Working Group II Contribution to the IPCC Sixth Assessment Report (AR6). 46th Session of the IPCC (IPCC, 2017).

  13. 13.

    Mahony, M. Climate change and the geographies of objectivity: the case of the IPCC’s burning embers diagram. Trans. Inst. Br. Geogr. 40, 153–167 (2015).

    Google Scholar 

  14. 14.

    Oppenheimer, M., Little, C. M. & Cooke, R. M. Expert judgement and uncertainty quantification for climate change. Nat. Clim. Change 6, 445–451 (2016).

    Google Scholar 

  15. 15.

    Budescu, D. V., Por, H. H. & Broomell, S. B. Effective communication of uncertainty in the IPCC reports. Clim. Change 113, 181–200 (2012).

    Google Scholar 

  16. 16.

    Swart, R., Bernstein, L., Ha-Duong, M. & Petersen, A. Agreeing to disagree: uncertainty management in assessing climate change, impacts and responses by the IPCC. Clim. Change 92, 1–29 (2009).

    Google Scholar 

  17. 17.

    Socolow, R. H. High-consequence outcomes and internal disagreements: tell us more, please. Clim. Change 108, 775–790 (2011).

    Google Scholar 

  18. 18.

    Adler, C. E. & Hirsch Hadorn, G. The IPCC and treatment of uncertainties: topics and sources of dissensus. Wiley Interdiscip. Rev. Clim. Change 5, 663–676 (2014).

    Google Scholar 

  19. 19.

    Sutton, R. T. ESD Ideas: A simple proposal to improve the contribution of IPCC WGI to the assessment and communication of climate change risks. Earth Syst. Dyn. 9, 1155–1158 (2018).

    Google Scholar 

  20. 20.

    Yohe, G. & Oppenheimer, M. Evaluation, characterization, and communication of uncertainty by the Intergovernmental Panel on Climate Change — an introductory essay. Clim. Change 108, 629–639 (2011).

    Google Scholar 

  21. 21.

    Moss, R. H. & Schneider, S. H. in Guidance Papers on the Cross Cutting Issues of the Third Assessment Report of the IPCC (eds Pachauri, R., Taniguchi, T. & Tanaka, K.) 33–51 (World Meteorological Organization, 2000).

  22. 22.

    Manning, M. et al. (eds) Describing Scientific Uncertainties in Climate Change to Support Analysis of Risk and of Options. Report of IPCC Workshop 11–13 (IPCC, 2004).

  23. 23.

    Mastrandrea, M. D. et al. Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties (IPCC, 2010).

  24. 24.

    Intergovernmental Panel on Climate Change (IPCC). Climate Change and Land: An IPCC Special Report on Climate Change, Desertification, Land Degradation, Sustainable Land Management, Food Security, and Greenhouse Gas Fluxes in Terrestrial Ecosystems (eds Shukla, P. R. et al) (IPCC, 2019).

  25. 25.

    Intergovernmental Panel on Climate Change (IPCC). in IPCC Special Report on the Ocean and Cryosphere in a Changing Climate (eds Pörtner, H.-O. et al.) (IPCC, 2019).

  26. 26.

    Jones, J. & Hunter, D. Qualitative research: consensus methods for medical and health services research. BMJ 311, 376–380 (1995).

    Google Scholar 

  27. 27.

    Peel, A. et al. Use of expert judgement across NICE guidance-making programmes: A review of current processes and suitability of existing tools to support the use of expert elicitation. Appl. Health Econ. Health Policy 16, 819–836 (2018).

    Google Scholar 

  28. 28.

    Waggoner, J., Carline, J. D. & Durning, S. J. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad. Med. 91, 663–668 (2016).

    Google Scholar 

  29. 29.

    Butler, A. J., Thomas, M. K. & Pintar, K. D. M. Systematic review of expert elicitation methods as a tool for source attribution of enteric illness. Foodborne Pathog. Dis. 12, 367–382 (2015).

    Google Scholar 

  30. 30.

    Morgan, M. G. Use (and abuse) of expert elicitation in support of decision making for public policy. Proc. Natl Acad. Sci. USA 111, 7176–7184 (2014).

    Google Scholar 

  31. 31.

    Aspinall, W. P. & Cooke, R. M. in Quantifying scientific uncertainty from expert judgement elicitation. In Risk and Uncertainty Assessment for Natural Hazards (eds Rougier, J., Sparks, S. & Hill, L. J.) 64-99 (Cambridge Univ. Press, 2011).

  32. 32.

    Graefe, A. & Armstrong, J. S. Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. Int. J. Forecast. 27, 183–195 (2011).

    Google Scholar 

  33. 33.

    Black, N. et al. Consensus development methods: a review of best practice in creating clinical guidelines. J. Health Serv. Res. Policy 4, 236–248 (1999).

    Google Scholar 

  34. 34.

    Tol, R. S. J. Equitable cost-benefit analysis of climate change policies. Ecol. Econ. 36, 71–85 (2001).

    Google Scholar 

  35. 35.

    Tol, R. S. J. Estimates of the damage costs of climate change: part 1: benchmark estimates. Environ. Resour. Econ. 21, 47–73 (2002).

    Google Scholar 

  36. 36.

    Tol, R. S. J. Estimates of the damage costs of climate change: Part II. Dynamic estimates. Environ. Resour. Econ. 21, 135–160 (2002).

    Google Scholar 

  37. 37.

    Intergovernmental Panel on Climate Change (IPCC). Climate Change 2001: Synthesis report (Cambridge Univ. Press, 2001).

  38. 38.

    Caddy, J. F. Limit reference points, traffic lights, and holistic approaches to fisheries management with minimal stock assessment input. Fish. Res. 56, 133–137 (2002).

    Google Scholar 

  39. 39.

    Wilson, D. C. & Pascoe, S. Chapter 13 Delivering complex scientific advice to multiple stakeholders. Dev. Aquacult. Fish. Sci. 36, 329–353 (2006).

    Google Scholar 

  40. 40.

    Bystrom, A. B., Naranjo-Madrigal, H. & Wehrtmann, I. S. Recomendaciones de manejo basadas en indicadores para la pesca artesanal con líneas de fondo en Costa Rica, Centroamérica. Rev. Biol. Trop. 65, 475–493 (2017).

    Google Scholar 

  41. 41.

    Simms, R. A. et al. Development of maternity dashboards across a UK health region; current practice, continuing problems. Eur. J. Obstet. Gynecol. Reprod. Biol. 170, 119–124 (2013).

    Google Scholar 

  42. 42.

    Donnelly, L. & Harrison, M. Geomorphological and geoforensic interpretation of maps, aerial imagery, conditions of diggability and the colour-coded RAG prioritization system in searches for criminal burials. Geol. Soc. Spec. Publ. 384, 174–194 (2013).

    Google Scholar 

  43. 43.

    Ruffell, A. & McAllister, S. RAG system for the management forensic and archaeological searches of burial grounds. Int. J. Archaeol. 3, 1–8 (2015).

    Google Scholar 

  44. 44.

    Mahony, M. & Hulme, M. The colour of risk: An exploration of the IPCC’s “burning embers” diagram. Spont. Gen. 6, 75–89 (2012).

    Google Scholar 

  45. 45.

    Vellinga, P. & Swart, R. The greenhouse marathon: a proposal for a global strategy. Clim. Change 18, vii–xii (1991).

    Google Scholar 

  46. 46.

    Schneider, B. Burning worlds of cartography: a critical approach to climate cosmograms of the Anthropocene. Geo Geogr. Env. 3, e00027 (2016).

    Google Scholar 

  47. 47.

    White, K. S. et al. Technical summary in Climate Change 2001: Impacts, Adaptation, and Vulnerability. Contribution of Working Group II to the Third Assessment Report of the Intergovernmental Panel on Climate Change (eds McCarthy, J. J., Canziani, O. F., Leary, N. A. & Dokken, D. J.) 19–73 (Cambridge Univ. Press, 2001).

  48. 48.

    Mastrandrea, M. D. & Schneider, S. H. Probabilistic integrated assessment of “dangerous” climate change. Science 304, 571–575 (2004).

    Google Scholar 

  49. 49.

    Gattuso, J. P. et al. Contrasting futures for ocean and society from different anthropogenic CO2 emissions scenarios. Science 349, aac4722 (2015).

    Google Scholar 

  50. 50.

    Magnan, A. K. et al. Implications of the Paris agreement for the ocean. Nat. Clim. Change 6, 732–735 (2016).

    Google Scholar 

  51. 51.

    Fischlin, A. Berücksichtigen wir in der Klimapolitik genügend Sicherheitsmargen? Do we have sufficient safety margins in climate policy? GAIA Ecol. Perspect. Sci. Soc. 18, 193–199 (2009).

    Google Scholar 

  52. 52.

    Schneider, S. H. et al. in Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) Ch. 19 (eds Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P. J. & Hanson, C. E.) 779–810 (Cambridge Univ. Press, 2007).

  53. 53.

    Smith, J. B. et al. Assessing dangerous climate change through an update of the Intergovernmental Panel on Climate Change (IPCC) “reasons for concern”. Proc. Natl Acad. Sci. USA 106, 4133–4137 (2009).

    Google Scholar 

  54. 54.

    Intergovernmental Panel on Climate Change (IPCC). in Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) (eds Solomon, S. et al.) 1–18 (Cambridge Univ. Press, 2007).

  55. 55.

    Intergovernmental Panel on Climate Change (IPCC) in Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) (eds Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P. J. & Hanson, C. E.) 7–22 (Cambridge Univ. Press, 2007).

  56. 56.

    Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P. J. & Hanson, C. E. in Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) (eds Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P. J. & Hanson, C. E.) 843–868 (Cambridge Univ. Press, 2007).

  57. 57.

    Fischlin, A. et al. in Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) Ch. 4 (eds Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P. J. & Hanson, C. E.) 211–272 (Cambridge Univ. Press, 2007).

  58. 58.

    Hennessy, K. et al. in Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) Ch. 11 (eds Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P. J. & Hanson, C. E.) 507–540 (Cambridge Univ. Press, 2007).

  59. 59.

    Intergovernmental Panel on Climate Change (IPCC). in Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) (eds Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P. J. & Hanson, C. E.) 23–78 (Cambridge Univ. Press, 2007).

  60. 60.

    Oppenheimer, M. et al. in Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) Ch. 19 (eds Field, C. B. et al.) 1039–1099 (Cambridge University Press, 2014).

  61. 61.

    Intergovernmental Panel on Climate Change (IPCC). Climate Change 2014: Synthesis Report. Contribution of Working Groups I, II and III to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (eds Core Writing Team, Pachauri, R. K. and Meyer, L. A.) (IPCC, 2014).

  62. 62.

    Field, C. B. et al. Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation: Special Report of the Intergovernmental Panel on Climate Change (Cambridge Univ. Press, 2012).

  63. 63.

    Intergovernmental Panel on Climate Change (IPCC). in Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation. A Special Report of Working Groups I and II of the Intergovernmental Panel on Climate Change (eds Field, C. B. et al) 1–19 (Cambridge Univ. Press, 2012).

  64. 64.

    Hoegh-Guldberg, O. et al. in Global Warming of 1.5°C. An IPCC Special Report on the Impacts of Global Warming of 1.5 °C Above Pre-industrial Levels and Related Global Greenhouse Gas Emission Pathways, in the Context of Strengthening the Global Response to the Threat of Climate Change, Sustainable Development, and Efforts to Eradicate Poverty Ch. 3 (eds Masson-Delmotte, V. et al.) 175–311 (IPCC, 2018).

  65. 65.

    Palazzo, A. et al. Linking regional stakeholder scenarios and shared socioeconomic pathways: quantified West African food and climate futures in a global context. Glob. Environ. Change 45, 227–242 (2017).

    Google Scholar 

  66. 66.

    Byers, E. et al. Global exposure and vulnerability to multi-sector development and climate change hotspots. Environ. Res. Lett. 13, 055012 (2018).

    Google Scholar 

  67. 67.

    Hanasaki, N. et al. A global water scarcity assessment under Shared Socio-economic Pathways - Part 2: Water availability and scarcity. Hydrol. Earth Syst. Sci. 17, 2393–2413 (2013).

    Google Scholar 

  68. 68.

    Hanasaki, N. et al. A global water scarcity assessment under Shared Socio-economic Pathways - Part 1: Water use. Hydrol. Earth Syst. Sci. 17, 2375–2391 (2013).

    Google Scholar 

  69. 69.

    Oppenheimer, M. et al. in IPCC Special Report on the Ocean and Cryosphere in a Changing Climate Ch. 4 (eds Pörtner, H.-O. et al.) 321–445 (IPCC, 2019).

  70. 70.

    World Meteorological Organization. United In Science: High-Level Synthesis Report of Latest Climate Science Information Convened by the Science Advisory Group of the UN Climate Action Summit 2019 (World Meteorological Organization, 2019).

  71. 71.

    Robinson, A., Calov, R. & Ganopolski, A. Multistability and critical thresholds of the Greenland ice sheet. Nat. Clim. Change 2, 429–432 (2012).

    Google Scholar 

  72. 72.

    Levermann, A. et al. The multimillennial sea-level commitment of global warming. Proc. Natl Acad. Sci. USA 110, 13745–13750 (2013).

    Google Scholar 

  73. 73.

    Levermann, A. et al. Projecting Antarctic ice discharge using response functions from SeaRISE ice-sheet models. Earth Syst. Dyn. 5, 271–293 (2014).

    Google Scholar 

  74. 74.

    Golledge, N. R. et al. The multi-millennial Antarctic commitment to future sea-level rise. Nature 526, 421–425 (2015).

    Google Scholar 

  75. 75.

    Fürst, J. J., Goelzer, H. & Huybrechts, P. Ice-dynamic projections of the Greenland ice sheet in response to atmospheric and oceanic warming. Cryosphere 9, 1039–1062 (2015).

    Google Scholar 

  76. 76.

    Pattyn, F. et al. The Greenland and Antarctic ice sheets under 1.5 °C global warming. Nat. Clim. Change 8, 1053–1061 (2018).

    Google Scholar 

  77. 77.

    Pörtner, H.-O. et al. in Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) Ch. 9 (eds Field, C. B. et al.) 411–484 (Cambridge Univ. Press, 2014).

  78. 78.

    National Institute for Health and Care Excellence (NICE). Developing NICE Guidelines: The Manual. Process and Methods Guides (NICE, 2014).

  79. 79.

    National Institute for Health and Care Excellence (NICE). Guide to the Methods of Technology Appraisal 2013 (NICE, 2013).

  80. 80.

    Stafford, A. C., Bindoff, I. K., Tenni, P. C., Peterson, G. M. & Doran, C. M. A methodological framework for estimating the clinical and economic value of community pharmacists’ clinical interventions using expert opinion. J. Clin. Pharm. Ther. 37, 378–385 (2012).

    Google Scholar 

  81. 81.

    Khodyakov, D. et al. Conducting online expert panels: a feasibility and experimental replicability study. BMC Med. Res. Methodol. 11, 174 (2011).

    Google Scholar 

  82. 82.

    Sinha, I. P., Smyth, R. L. & Williamson, P. R. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 8, e1000393 (2011).

    Google Scholar 

  83. 83.

    Moher, D., Schulz, K. F., Simera, I. & Altman, D. G. Guidance for developers of health research reporting guidelines. PLoS Med. 7, e1000217 (2010).

    Google Scholar 

  84. 84.

    Ferri, C. P. et al. Global prevalence of dementia: a Delphi consensus study. Lancet 366, 2112–2117 (2005).

    Google Scholar 

  85. 85.

    Cresswell, K. M. et al. Global research priorities to better understand the burden of iatrogenic harm in primary care: an international Delphi exercise. PLoS Med. 10, e1001554 (2013).

    Google Scholar 

  86. 86.

    Hasson, F., Keeney, S. & McKenna, H. Research guidelines for the Delphi survey technique. J. Adv. Nurs. 32, 1008–1015 (2000).

    Google Scholar 

  87. 87.

    Shrader-Frechette, K. Experts in Uncertainty: Opinion and Subjective Probability in Science. Ethics Vol. 103 (ed. Cooke, R. M.) 599–601 (Oxford Univ. Press, 1993).

  88. 88.

    Ferguson, J. H. The NIH consensus development program: the evolution of guidelines. Int. J. Technol. Assess. Health Care 12, 460–474 (1996).

    Google Scholar 

  89. 89.

    Dalkey, N. & Helmer, O. An experimental application of the DELPHI method to the use of experts. Manag. Sci. 9, 458–467 (1963).

    Google Scholar 

  90. 90.

    Delbecq, A. L. & Van de Ven, A. H. A group process model for problem identification and program planning. J. Appl. Behav. Sci. 7, 466–492 (1971).

    Google Scholar 

  91. 91.

    Dias, L. C., Morton, A. & Quigley, J. in Elicitation: the Science and Art of Structuring Judgement (eds Dias, L. C., Morton, A. & Quigley, J.) 1–14 (Springer, 2017).

  92. 92.

    James, D. & Warren-Forward, H. Research methods for formal consensus development. Nurse Res. 22, 35–40 (2015).

    Google Scholar 

  93. 93.

    Clemen, R. T. & Winkler, R. L. Combining probability distributions from experts in risk analysis. Risk Anal. 19, 187–203 (1999).

    Google Scholar 

  94. 94.

    European Food Safety Authority. Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA J. 12, 3734 (2014).

    Google Scholar 

  95. 95.

    Walker, K. D., Evans, J. S. & MacIntosh, D. Use of expert judgment in exposure assessment. Part I. Characterization of personal exposure to benzene. J. Expo. Sci. Environ. Epidemiol. 11, 308–322 (2001).

    Google Scholar 

  96. 96.

    McCormack, C. G. et al. Key impacts of climate engineering on biodiversity and ecosystems, with priorities for future research. J. Integr. Environ. Sci. 13, 103–128 (2016).

    Google Scholar 

  97. 97.

    Griscom, B. W. et al. Natural climate solutions. Proc. Natl Acad. Sci. USA 114, 11645–11650 (2017).

    Google Scholar 

  98. 98.

    Mach, K. J. et al. Climate as a risk factor for armed conflict. Nature 571, 193–197 (2019).

    Google Scholar 

  99. 99.

    Bamber, J. L. & Aspinall, W. P. An expert judgement assessment of future sea level rise from the ice sheets. Nat. Clim. Change 3, 424–427 (2013).

    Google Scholar 

  100. 100.

    Mukherjee, N. et al. The Delphi technique in ecology and biological conservation: Applications and guidelines. Methods Ecol. Evol. 6, 1097–1109 (2015).

    Google Scholar 

  101. 101.

    Bhave, A. G., Conway, D., Dessai, S. & Stainforth, D. A. Water resource planning under future climate and socioeconomic uncertainty in the Cauvery River Basin in Karnataka, India. Water Resour. Res. 54, 708–728 (2018).

    Google Scholar 

  102. 102.

    Zommers, Z. & Alverson, K. in Resilience: The Science of Adaptation to Climate Change (eds Zommers, Z. & Alverson, K.). 329–336 (Elsevier, 2018).

  103. 103.

    Berrang-Ford, L., Pearce, T. & Ford, J. D. Systematic review approaches for climate change adaptation research. Reg. Environ. Change 15, 755–769 (2015).

    Google Scholar 

  104. 104.

    Zickfeld, K., Morgan, M. G., Frame, D. J. & Keith, D. W. Expert judgments about transient climate response to alternative future trajectories of radiative forcing. Proc. Natl Acad. Sci. USA 107, 12451–12456 (2010).

    Google Scholar 

  105. 105.

    Kriegler, E., Hall, J. W., Held, H., Dawson, R. & Schellnhuber, H. J. Imprecise probability assessment of tipping points in the climate system. Proc. Natl Acad. Sci. USA 106, 5041–5046 (2009).

    Google Scholar 

  106. 106.

    Bantel, K. A. Comprehensiveness of strategic planning: the importance of heterogeneity of a top team. Psychol. Rep. 73, 35–49 (1993).

    Google Scholar 

  107. 107.

    Bolger, F. & Wright, G. Improving the Delphi process: lessons from social psychological research. Technol. Forecast. Soc. Change 78, 1500–1513 (2011).

    Google Scholar 

  108. 108.

    Hurlbert, M. et al. in Climate Change and Land: An IPCC Special Report on Climate Change, Desertification, Land Degradation, Sustainable Land Management, Food Security, and Greenhouse Gas Fluxes in Terrestrial Ecosystems Ch. 7 (eds Shukla, P. R. et al.) 673–800 (IPCC, 2019).

  109. 109.

    Jairath, N. & Weinstein, J. The Delphi methodology (part two): a useful administrative approach. Can. J. Nurs. Adm. 7, 7–20 (1994).

    Google Scholar 

  110. 110.

    Bolger, F., Stranieri, A., Wright, G. & Yearwood, J. Does the Delphi process lead to increased accuracy in group-based judgmental forecasts or does it simply induce consensus amongst judgmental forecasters? Technol. Forecast. Soc. Change 78, 1671–1680 (2011).

    Google Scholar 

  111. 111.

    Singh, G. G. et al. Group elicitations yield more consistent, yet more uncertain experts in understanding risks to ecosystem services in New Zealand bays. PLoS One 12, e0190326 (2017).

    Google Scholar 

  112. 112.

    Speirs-Bridge, A. et al. Reducing overconfidence in the interval judgments of experts. Risk Anal. 30, 512–523 (2010).

    Google Scholar 

  113. 113.

    Sperber, D., Mortimer, D., Lorgelly, P. & Berlowitz, D. An expert on every street corner? Methods for eliciting distributions in geographically dispersed opinion pools. Value Health 16, 434–437 (2013).

    Google Scholar 

  114. 114.

    Adger, W. N., Brown, I. & Surminski, S. Advances in risk assessment for climate change adaptation policy. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 376, 20180106 (2018).

    Google Scholar 

  115. 115.

    Yohe, G. “Reasons for concern” (about climate change) in the United States. Clim. Change 99, 295–302 (2010).

    Google Scholar 

  116. 116.

    Warszawski, L. et al. The inter-sectoral impact model intercomparison project (ISI–MIP): Project framework. Proc. Natl Acad. Sci. USA 111, 3228–3232 (2014).

    Google Scholar 

  117. 117.

    Frieler, K. et al. Assessing the impacts of 1.5°C global warming – simulation protocol of the Inter-Sectoral Impact Model Intercomparison Project (ISIMIP2b). Geosci. Model Dev. 10, 4321–4345 (2017).

    Google Scholar 

  118. 118.

    Sharpe, S. Telling the boiling frog what he needs to know: why climate change risks should be plotted as probability over time. Geosci. Commun. 2, 95–100 (2019).

    Google Scholar 

  119. 119.

    Colson, A. R. & Cooke, R. M. Expert elicitation: using the classical model to validate experts’ judgments. Rev. Environ. Econ. Policy 12, 113–132 (2018).

    Google Scholar 

  120. 120.

    Cooke, R., Mendel, M. & Thijs, W. Calibration and information in expert resolution; a classical approach. Automatica 24, 87–93 (1988).

    Google Scholar 

  121. 121.

    Field, C. B. et al. (eds) Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC) (Cambridge Univ. Press, 2014).

Download references

Acknowledgements

The authors thank J. Smith for useful reviews and informed comments on the paper. A.K.M. thanks the French National Research Agency (‘Investments for the Future’ programme, ANR-10-LABX-14-01). All views expressed herein are the responsibility of the authors and do not necessarily reflect their respective institutions, the Intergovernmental Panel on Climate Change or the United Nations.

Author information

Affiliations

Authors

Contributions

Z.Z. and W.T. designed the study. Z.Z., E.L.D., W.T., K.C., M. Howden, K.W., J.P.E. and S.G. developed the improved expert-elicitation procedure for the SRCCL. A.K.M., H.-O.P. and Z.S. contributed to the SROCC burning embers development. P.M. harmonized the burning embers for three RFCs across the IPCC reports and developed figures 2 and 3. M. Howden, P.M. and W.T. led the analysis and design of figure 5. Z.Z.I., A.F. and C.R. contributed substantial additions to the history section. B.C.O. and P.M. helped research and draft the section on changes in risks. S.G., M. Hulbert and Z.Z.I. wrote the section on expert elicitation. Z.Z. led the drafting of the paper. All authors contributed to the editing and revision of the manuscript and helped develop conclusions.

Corresponding author

Correspondence to Zinta Zommers.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Earth & Environment thanks Kristie Ebi, Michael Mastrandrea and Michael Oppenheimer for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

The Ember Factory in the Python Package Index: https://pypi.org/project/EmberFactory/

The Ember Factory online application: https://climrisk.org/emberfactory

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zommers, Z., Marbaix, P., Fischlin, A. et al. Burning embers: towards more transparent and robust climate-change risk assessments. Nat Rev Earth Environ 1, 516–529 (2020). https://doi.org/10.1038/s43017-020-0088-0

Download citation

Search

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing