Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Expert judgement and uncertainty quantification for climate change

Abstract

Expert judgement is an unavoidable element of the process-based numerical models used for climate change projections, and the statistical approaches used to characterize uncertainty across model ensembles. Here, we highlight the need for formalized approaches to unifying numerical modelling with expert judgement in order to facilitate characterization of uncertainty in a reproducible, consistent and transparent fashion. As an example, we use probabilistic inversion, a well-established technique used in many other applications outside of climate change, to fuse two recent analyses of twenty-first century Antarctic ice loss. Probabilistic inversion is but one of many possible approaches to formalizing the role of expert judgement, and the Antarctic ice sheet is only one possible climate-related application. We recommend indicators or signposts that characterize successful science-based uncertainty quantification.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: The effect of probabilistic inversion on SLR probability distributions.
Figure 2: Scatter plots of 10,000 randomly selected samples from before (grey) and after (black) the probabilistic inversion.
Figure 3

Similar content being viewed by others

References

  1. The Economic and Financial Risks of a Changing Climate: Insights from Leading Experts Workshop Report (AAAS, 2014).

  2. Draper, D. Assessment and propagation of model uncertainty. J. R. Statis. Soc. B 57, 45–97 (1995).

    Google Scholar 

  3. Oppenheimer, M. et al. in Climate Change: Impacts, Adaptation, and Vulnerability. (eds Field, C. B. et al.) 1039–1099 (IPCC, Cambridge Univ. Press, 2014).

    Google Scholar 

  4. Frigg, R., Smith, L. A. & Stainforth, D. A. The myopia of imperfect climate models: the case of UKCP09. Philos. Sci. 80, 886–897 (2013).

    Article  Google Scholar 

  5. Knutti, R. & Hegerl, G. C. The equilibrium sensitivity of the Earth's temperature to radiation changes. Nature Geosci. 1, 735–743 (2008).

    Article  CAS  Google Scholar 

  6. Morgan, M. G. & Keith, D. W. Subjective judgements by climate experts. Environ. Sci. Technol. 29, 468–476 (1995).

    Google Scholar 

  7. Zickfeld, K., Morgan, M. G., Frame, D. J. & Keith, D. W. Expert judgements about transient climate response to alternative future trajectories of radiative forcing. Proc. Natl Acad. Sci. USA 107, 12451–12456 (2010).

    Article  Google Scholar 

  8. Church, J. A. et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) (IPCC, Cambridge Univ. Press, Cambridge, 2013).

    Google Scholar 

  9. Morgan, G. M. et al. Best Practice Approaches for Characterizing, Communicating, and incorporating Scientific Uncertainty in Climate Decisions Synthesis and Assessment Product 5.2. (US Climate Change Science Program, 2009).

    Google Scholar 

  10. Mastrandrea, M. D. et al. Guidance note for lead authors of the IPCC Fifth Assessment Report on consistent treatment of uncertainties (IPCC, 2010).

    Google Scholar 

  11. Knutti, R. et al. in Meeting Report of the Intergovernmental Panel on Climate Change: Expert Meeting on Assessing and Combining Multi Model Climate (eds Stocker, T. F., Qin, D., Plattner, G.-K., Tignor, M. & Midgley, P. M.) 1–13 (IPCC, 2010).

    Google Scholar 

  12. Moss, R. H. & Schneider, S. H. in Guidance papers on the cross cutting issues of the Third Assessment Report of the IPCC (eds R. Pachauri, T. Taniguchi and K. Tanaka) 33–51 (IPCC, 2000).

    Google Scholar 

  13. Collins . et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) Ch. 13 (Cambridge Univ. press, 2013).

    Google Scholar 

  14. Goossens, L. H. J. & Kelly, G. N. Radiation protection dosimetry expert judgement and accident consequence. J. Uncertainty Anal. 90, 295–301 (2000).

    Google Scholar 

  15. Mason, D. & Knutti, R. Predictor screening, calibration, and observational constraints in climate model ensembles: an illustration using climate sensitivity. J. Clim. 26, 887–898 (2013).

    Article  Google Scholar 

  16. Murphy, J. M. et al. A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Phil. Trans. Royal Soc. A 365, 1993–2028 (2007).

    Article  CAS  Google Scholar 

  17. Knutti, R., Furrer, R., Tebaldi, C., Cermak, J. & Meehl, G. A. Challenges in combining projections from multiple climate models. J. Clim. 23, 2739–2758 (2010).

    Article  Google Scholar 

  18. Tebaldi, C. & Knutti, R. The use of the multi-model ensemble in probabilistic climate projections. Phil. Trans. R. Soc. A 365, 2053–2075 (2007).

    Article  Google Scholar 

  19. Rougier, J. C., Goldstein, M. & House, L. Second-order exchangeability analysis for multi-model ensembles. J. Am. Statist. Assoc. 108, 852–863 (2013).

    Article  CAS  Google Scholar 

  20. Goldstein, M. & Rougier, J. Reified Bayesian modelling and inference for physical systems. J. Statist. Plan. Infer. 139, 1221–1239 (2009).

    Article  Google Scholar 

  21. Murphy, J. M. et al. UK Climate Projections Science Report: Climate Change Projections (Met Office, 2009).

  22. Sexton, D. M. H., Murphy, J. M., Collins, M. & Webb, M. J. Multivariate probabilistic projections using imperfect climate models part I: outline of methodology Clim. Dynam. 38, 2513–2542 (2012).

    Article  Google Scholar 

  23. Sexton, D. M. H. & Murphy, J. M. Multivariate probabilistic projections using imperfect climate models part II: robustness of methodological choices and consequences for climate sensitivity Clim. Dynam. 38, 2543–2558 (2012).

    Article  Google Scholar 

  24. Parker, W. S. Ensemble modeling, uncertainty and robust predictions. WIREs Clim. Change 4, 213–223 (2013).

    Article  Google Scholar 

  25. O'Reilly, J., Oreskes, N. & Oppenheimer, M. The rapid disintegration of predictions: climate science, bureaucratic institutions, and the West Antarctic ice sheet. Social Stud. Sci. 42, 709–731 (2012).

    Article  Google Scholar 

  26. van der Sluijs, J. et al. Anchoring devices in science for policy: the case of consensus around climate sensitivity. Social Stud. Sci. 28, 291–323 (1998).

    Article  Google Scholar 

  27. Houser, T. et al. American Climate Prospectus: Economic Risks in the United States (Rhodium Group, 2014).

    Google Scholar 

  28. Hinkel, J. et al. Coastal flood damage and adaptation costs under 21st century sea-level rise. Proc. Natl Acad. Sci. USA 111, 3292–3297 (2014).

    Article  CAS  Google Scholar 

  29. Pattyn, F. et al. Grounding-line migration in plan-view marine ice-sheet models: results of the ice2sea MISMIP3d intercomparison. J. Glaciol. 59, 410–422 (2013).

    Article  Google Scholar 

  30. Durand, G. & Pattyn, F. Reducing uncertainties in projections of Antarctic ice mass loss. Cryos. Discuss. 9, 2625–2654 (2015).

    Article  Google Scholar 

  31. Pollard, D., DeConto, R. M. & Alley, R. B. Potential Antarctic ice sheet retreat driven by hydrofracturing and ice cliff failure. Earth Planet. Sci. Lett. 412, 112–121 (2015).

    Article  CAS  Google Scholar 

  32. Bindschadler, R. A. et al. Ice-sheet model sensitivities to environmental forcing and their use in projecting future sea level (the SeaRISE project). J. Glaciol. 59, 195–224 (2013).

    Article  Google Scholar 

  33. Nowicki, S. et al. Insights into spatial sensitivities of ice mass response to environmental change from the searise ice sheet modeling project ii: Greenland. J. Geophys. Res. Earth Surf. 118, 1025–1044 (2013).

    Article  Google Scholar 

  34. Vaughan, D. & Spouge, J. Risk estimation of collapse of the West Antarctic ice sheet. Climatic Change 52, 65–91 (2002).

    Article  Google Scholar 

  35. Rahmstorf, S., Perrette, M. & Vermeer, M. Testing the robustness of semi-empirical sea level projections. Clim. Dynam. 39, 861–875 (2012).

    Article  Google Scholar 

  36. Oppenheimer, M., O'Neill, B. & Webster, M. Negative learning. Climatic Change 89, 155–172 (2008).

    Article  Google Scholar 

  37. Oppenheimer, M. et al. The limits of consensus. Science 317, 1505–1506 (2007).

    Article  CAS  Google Scholar 

  38. Bamber, J. L. & Aspinall, W. P. An expert judgement assessment of future sea. level rise from the ice sheets. Nature Clim. Change 3, 424–427 (2013).

    Article  Google Scholar 

  39. Little, C. M., Oppenheimer, M. & Urban, N. M. Upper bounds on twenty-first-century Antarctic ice loss assessed using a probabilistic framework, Nature Clim. Change 3, 654–659 (2013).

    Article  Google Scholar 

  40. Kopp, R. E. et al. Probabilistic 21st and 22nd century sea-level projections at a global network of tide gauge sites. Earth's Future 2, 383–406 (2014).

    Article  Google Scholar 

  41. Jevrejeva, S., Grinsted, A. & Moore, J. C. Upper limit for sea level projections by 2100. Environ. Res. Lett. 9, 104008 (2014).

    Article  Google Scholar 

  42. Kraan, B. C. P. & Bedford. T. J. Probabilistic inversion of expert judgements in the quantification of model uncertainty. Manag. Sci. 51, 995–1006 (2005).

    Article  Google Scholar 

  43. Du, C., Kurowicka, D. & Cooke, R. M. Techniques for generic probabilistic inversion, Comp. Stat. Data Anal. 50, 1164–1187 (2006).

    Article  Google Scholar 

  44. Joughin I., Smith, B. & Medley, B. Marine ice sheet collapse potentially under way for Thwaites Glacier Basin, West Antarctica. Science 344, 735–738 (2014).

    Article  CAS  Google Scholar 

  45. Rignot, E., Mouginot, J., Morlighem, M., Seroussi, H. & Scheuchl, B. Widespread, rapid grounding line retreat of Pine Island, Thwaites, Smith, and Kohler glaciers, West Antarctica, from 1992 to 2011. Geophys. Res. Lett. 41, 3502–3509 (2014).

    Article  Google Scholar 

  46. de Vries, H. & van de Wal, R. S. W. How to interpret expert judgement assessments of 21st century sea-level rise. Climatic Change 130, 87–100 (2015).

    Article  CAS  Google Scholar 

  47. Horton, B. P., Rahmstorf, S., Engelhart, S. E. & Kemp, A. C. Expert assessment of sea-level rise by AD 2100 and AD 2300. Quatern. Sci. Rev. 84, 1–6 (2014).

    Article  Google Scholar 

  48. Global sea level rise scenarios for the United States National Climate Assessment (Climate Program Office, 2012).

  49. Little, C. M., Urban, N. M. & Oppenheimer, M. Probabilistic framework for assessing the ice sheet contribution to sea level change. Proc. Natl Acad. Sci. USA 110, 3264–3269 (2013).

    Article  CAS  Google Scholar 

  50. Jones, J. A. et al. Probabilistic Accident Consequence Uncertainty Assessment Using COSYMA Uncertainty from the Atmospheric Dispersion and Deposition Module EUR 18822EN (European Commission, 2001).

    Google Scholar 

  51. Popper, K. R. The Logic of Scientific Discovery (Hutchinson, 1959).

    Google Scholar 

  52. Lakatos, I. The Methodology of Scientific Research Programmes Philos. Papers Vol. 1 (Cambridge Univ. Press, 1978).

    Book  Google Scholar 

  53. Oreskes, N., Shrader-Frechete, K. & Belitz, K. Verification, validation and confirmation of numerical models in the Earth sciences. Science 26, 641–646 (1994).

    Article  Google Scholar 

  54. Cooke, R. M. Uncertainty in dispersion and deposition in accident consequence modelling assessed with performance-based expert judgement. Rel. Eng. Syst. Saf. 45, 35–46 (1994).

    Article  Google Scholar 

  55. Aspinall, W. P. & Cooke, R. M. in Risk and Uncertainty Assessment in Natural Hazards (eds Hill, L., Rougier, J. C. & Sparks R. S. J.) 64–99 (Cambridge University Press, 2013).

    Book  Google Scholar 

  56. Rasmussen, N. C. et al. Reactor Safety Study: An Assessment of Accident Risks in US Commercial Nuclear Power Plants WASH-1400 (NUREG75/014) (US Nuclear Regulatory Commission, 1975).

    Google Scholar 

  57. Lewis, H. et al. Risk Assessment Review Group Report to the US Nuclear Regulatory Commission NUREG/CR-04000 (Chemical Rubber Company, 1979).

    Google Scholar 

  58. Science and Judgement in Risk Assessment (The National Academies, 1994).

  59. Guidelines for Carcinogen risk assessment EPA/630/P-03/001F (US Environmental Protection Agency, 2005).

  60. Expert Elicitation Task Force White Paper (US Environmental Protection Agency, 2011).

  61. Cooke, R. M. & Goossens, L. H. J. TU Delft Expert judgment data base, special issue on expert judgement. Rel. Eng. Syst. Saf. 93, 657–674 (2008).

    Article  Google Scholar 

  62. Aspinall, W. P. A route to more tractable expert advice. Nature 463, 294–95 (2010).

    Article  CAS  Google Scholar 

  63. Eggstaff, J. W., Mazzuchi, T. A. & Sarkani, S. The effect of the number of seed variables on the performance of Cooke's classical model. Rel. Eng. Syst. Saf. 121, 72–82, (2014).

    Article  Google Scholar 

  64. Aspinall, W. P. in Statistics in Volcanology. (eds Mader, H. M., Coles, S. G., Connor, C. B. & Connor, L. J.) 15–30 (Geological Society, 2006).

    Book  Google Scholar 

  65. Cooke, R. M. et al. A probabilistic characterization of the relationship between fine particulate matter and mortality: elicitation of European experts. Environ. Sci. Technol. 41, 6598–6605 (2007).

    Article  CAS  Google Scholar 

  66. Tuomisto, J. T., Wilson, A., Cooke, R. M., Tainio, M. & Evans J. S. Mortality in Kuwait due to PM from oil fires after the Gulf War: combining expert elicitation assessments. Epidemiol. 16, S74–S75 (2005).

    Article  Google Scholar 

  67. Evans J. S., Wilson A., Tuomisto J. T., Tainio M. & Cooke R. M. What risk assessment can tell us about the mortality impacts of the Kuwaiti oil fires. Epidemiol. 16, S137–S138 (2005).

    Article  Google Scholar 

  68. Cooke, R. M. et al. Out-of-sample validation for structured expert judgement of Asian carp establishment in Lake Erie. Integr. Environ. Assess. Manag. 10, 522–528 (2014).

    Article  Google Scholar 

  69. Hoffmann, S. et al. Research synthesis methods in an age of globalized risks: lessons from the global burden of foodborne disease expert elicitation. Risk Analysis 36, 191–202 (2015).

    Google Scholar 

  70. Koch, B. J. et al. Suburban watershed nitrogen retention: estimating the effectiveness of storm water management structures. Elementa: Sci. Anthropocene 3, 1–18 (2015).

    Google Scholar 

  71. Wittmann, M. E., Cooke, R. M., Rothlisberger, J. D. & Lodge, D. M. Using structured expert judgement to assess invasive species prevention: Asian carp and the Mississippi — Great Lakes hydrologic connection. Environ. Sci. Technol. 48, 2150–2156 (2014).

    Article  CAS  Google Scholar 

  72. Wittmann, M. E. et al. Structured expert judgement to forecast species invasions: bighead and silver carp in Lake Erie. Cons. Biol. 29, 187–197 (2014).

    Article  Google Scholar 

  73. Tyshenko, M. G. et al. 2010 expert elicitation for the judgement of prion disease risk uncertainties using the classical model and EXCALIBUR J. Toxicol. Environ. Health A. 74, 261–285 (2011).

    Article  CAS  Google Scholar 

  74. Gerstenberger, M. C., McVerry, G. H., Rhoades, D. A. & Stirling, M. W. Seismic hazard modeling for the recovery of Christchurch, New Zealand. Earthquake Spectra 30, 17–29 (2014).

    Article  Google Scholar 

  75. Christophersen, A., Nicol, A. & Gerstenberger, M. C. The Feasibility of Using Seed Questions for Weighting Expert Opinion in CCS Risk Assessment CO2CRC Report RPT11–2868 (Cooperative Research Centre for Greenhouse Gas Technologies, 2011).

    Google Scholar 

  76. Gerstenberger, M. C. et al. in 11th International Conference on Greenhouse Gas Control Technologies (eds Dixon, T. & Yamaji, K.) 2775–2782 (Elsevier, 2013).

    Google Scholar 

Download references

Acknowledgements

We thank J. Hall (Oxford University, UK), K. Keller (Pennsylvania State University, USA), R. Kopp (Rutgers University, USA), J. Rougier (University of Bristol, UK), D. Sexton (Met Office Hadley Centre, UK) and C. Tebaldi (National Center for Atmospheric Research, USA) for either helpful comments on an earlier draft of the manuscript, or useful discussions of the issues raised, or both.

Author information

Authors and Affiliations

Authors

Contributions

M.O., C.M.L. and R.M.C. designed the research, conducted analysis of data and results, and contributed to writing, editing and revision. C.M.L. and R.M.C. performed statistical modelling.

Corresponding author

Correspondence to Michael Oppenheimer.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Information

Supplementary Information (PDF 607 kb)

Supplementary Information

Supplementary Information (XLS 88 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Oppenheimer, M., Little, C. & Cooke, R. Expert judgement and uncertainty quantification for climate change. Nature Clim Change 6, 445–451 (2016). https://doi.org/10.1038/nclimate2959

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nclimate2959

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing