Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

New vigour involving statisticians to overcome ensemble fatigue


Climate simulation data comprise a range of different phenomena with complex and interacting processes. Yet our understanding of the climate is incomplete despite the huge volumes of data, of which only a small fraction has been explored, and many questions remain, particularly those on the character and origin of uncertainties associated with model simulations and how further modelling efforts can improve understanding. Here, we question whether climate model information could be used more effectively and how so-called 'ensembles of opportunity' should be interpreted. Statisticians can contribute substantially to designing 'smarter' ensemble experiments, improving the distillation of information from ensembles, and helping interpret the relative merits of additional simulations. Future progress may be enhanced by increasing collaborations with statisticians.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Figure 1: A schematic of the state probability space for a multi-model ensemble and the real world represented by a multi-dimensional probability density function.
Figure 2: Uncertainty of a statistical downscaling of 2050 sea-level rise projections relative to 1999 in Bergen, Norway.
Figure 3


  1. 1

    Weigel, A. P., Liniger, M. A. & Appenzeller, C. Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Q. J. R. Meteorol. Soc. 134, 241–260 (2008).

    Article  Google Scholar 

  2. 2

    Bruyère, C. et al. Impact of Climate Change on Gulf of Mexico Hurricanes Technical Note NCAR/TN-535+STR. (NCAR, 2017).

    Google Scholar 

  3. 3

    Smith, M. J. et al. Changing how Earth system modeling is done to provide more useful information for decision making, science, and society. Bull. Am. Meteorol. Soc. 95, 1453–1464 (2014). This paper calls for new ways of developing Earth system models to enable more relevant and informative projections.

    Article  Google Scholar 

  4. 4

    Taylor, K. E., Stouffer, R. J. & Meehl, G. A. An overview of CMIP5 and the experiment design. Bull. Am. Meteorol. Soc. 93, 485–498 (2012).

    Article  Google Scholar 

  5. 5

    Benestad, R. In Oxford Research Encyclopedia of Climate Science (Oxford Univ. Press, 2016).

    Google Scholar 

  6. 6

    Overpeck, J. T., Meehl, G. A., Bony, S. & Easterling, D. R. Climate data challenges in the 21st century. Science 331, 700–702 (2011).

    CAS  Article  Google Scholar 

  7. 7

    Christensen, J. H. & Christensen, O. B. A summary of the PRUDENCE model projections of changes in European climate by the end of this century. Climatic Change 81, 7–30 (2007).

    Article  Google Scholar 

  8. 8

    van der Linden, P. & Mitchell, F. B. (eds) Ensembles: Climate Change and its Impacts: Summary of Research and Results from the ENSEMBLES Project (Met Office Hadley Centre, 2009);

    Google Scholar 

  9. 9

    Giorgi, F. & Gutowski, W. J. Regional dynamical downscaling and the CORDEX initiative. Annu. Rev. Environ. Resour. 40, 467–490 (2015).

    Article  Google Scholar 

  10. 10

    Eyring, V. et al. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. 9, 1937–1958 (2016).

    Article  Google Scholar 

  11. 11

    IPCC Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) (Cambridge Univ. Press, 2013).

  12. 12

    IPCC Climate Change 2007: The Physical Science Basis (eds Solomon. S. et al.) (Cambridge Univ. Press, 2007).

  13. 13

    Tebaldi, C. & Knutti, R. The use of the multi-model ensemble in probabilistic climate projections. Phil. Trans. R. Soc. A 365, 2053–2075 (2007).

    Article  Google Scholar 

  14. 14

    Smith, R. L., Tebaldi, C., Nychka, D. & Mearns, L. O. Bayesian modeling of uncertainty in ensembles of climate models. J. Am. Stat. Assoc. 104, 97–116 (2009).

    CAS  Article  Google Scholar 

  15. 15

    Sanderson, B. M., Knutti, R. & Caldwell, P. Addressing interdependency in a multimodel ensemble by interpolation of model properties. J. Clim. 28, 5150–5170 (2015).

    Article  Google Scholar 

  16. 16

    Stainforth, D., Allen, M., Tredger, E. & Smith, L. Confidence, uncertainty and decision-support relevance in climate predictions. Phil. Trans. R. Soc. A 365, 2145–2161 (2007).

    CAS  Article  Google Scholar 

  17. 17

    Knutti, R. & Sedláček, J. Robustness and uncertainties in the new CMIP5 climate model projections. Nat. Clim. Change 3, 369–373 (2013). Argues that the new generation of more complex models running scenarios for the IPCC's AR5 is widely, and perhaps naively, expected to provide more detailed and more certain projections.

    Article  Google Scholar 

  18. 18

    Gneiting, T. & Raftery, A. E. Weather forecasting with ensemble methods. Science 310, 248–249 (2005).

    CAS  Article  Google Scholar 

  19. 19

    Rougier, J. Ensemble averaging and mean squared error. J. Clim. 29, 8865–8870 (2016).

    Article  Google Scholar 

  20. 20

    Rougier, J. Probabilistic inference for future climate using an ensemble of climate model evaluations. Climatic Change 81, 247–264 (2007).

    Article  Google Scholar 

  21. 21

    Schefzik, R., Thorarinsdottir, T. L. & Gneiting, T. Uncertainty quantification in complex simulation models using ensemble copula coupling. Stat. Sci. 28, 616–640 (2013).

    Article  Google Scholar 

  22. 22

    Sansom, P. G., Ferro, C. A. T., Stephenson, D. B., Goddard, L. & Mason, S. J. Best practices for postprocessing ensemble climate forecasts. Part I: Selecting appropriate recalibration methods. J. Clim. 29, 7247–7264 (2016). Describes a new recalibration method that involves adjustments for both unconditional and conditional biases in the mean, variance, and trend.

    Article  Google Scholar 

  23. 23

    Jolliffe, I. T. & Stephenson, D. B. Forecast Verification: a Practitioner's Guide in Atmospheric Science (Wiley, 2003).

    Google Scholar 

  24. 24

    Notz, D. How well must climate models agree with observations? Phil. Trans. R. Soc. A 373, 20140164 (2015).

    Article  Google Scholar 

  25. 25

    Collins, M. Ensembles and probabilities: a new era in the prediction of climate change. Phil. Trans. R. Soc. A 365, 1957–1970 (2007).

    Article  Google Scholar 

  26. 26

    Deser, C., Knutti, R., Solomon, S. & Phillips, A. S. Communication of the role of natural variability in future North American climate. Nat. Clim. Change 2, 775–779 (2012).

    Article  Google Scholar 

  27. 27

    Katz, R. W. et al. Uncertainty analysis in climate change assessments. Nat. Clim. Change 3, 769–771 (2013).

    Article  Google Scholar 

  28. 28

    Jun, M., Knutti, R. & Nychka, D. W. Spatial analysis to quantify numerical model bias and dependence: how many climate models are there? J. Am. Stat. Assoc. 103, 934–947 (2008).

    CAS  Article  Google Scholar 

  29. 29

    Edwards, P. N. A Vast Machine: Computer Models, Climate Data, and the Politics of Global Warming (MIT Press, 2010).

    Google Scholar 

  30. 30

    Chandler, R. E. Exploiting strength, discounting weakness: combining information from multiple climate simulators. Phil. Trans. R. Soc. A 371, 20120388 (2013).

    Article  Google Scholar 

  31. 31

    Stephenson, D. B., Collins, M., Rougier, J. C. & Chandler, R. E. Statistical problems in the probabilistic prediction of climate change. Environmetrics 23, 364–372 (2012).

    Article  Google Scholar 

  32. 32

    Rougier, J., Goldstein, M. & House, L. Second-order exchangeability analysis for multimodel ensembles. J. Am. Stat. Assoc. 108, 852–863 (2013).

    CAS  Article  Google Scholar 

  33. 33

    Thorarinsdottir, T. L., Guttorp, P., Drews, M., Kaspersen, P. S. & de Bruin, K. Sea level adaptation decisions under uncertainty. Water Resour. Res. (in the press).

  34. 34

    Tye, M. R., Holland, G. J. & Done, J. M. Rethinking failure: time for closer engineer–scientist collaborations on design. Proc. Inst. Civ. Eng. Forensic Eng. 168, 49–57 (2015).

    Google Scholar 

  35. 35

    Gutowski Jr., W. J. et al. WCRP COordinated Regional Downscaling EXperiment (CORDEX): a diagnostic MIP for CMIP6. Geosci Model Dev. 9, 4087–4095 (2016).

    Article  Google Scholar 

  36. 36

    Weaver, C. P. et al. Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks. WIREs Clim. Change 4, 39–60 (2013). This paper discusses how climate modelling is used to support so-called decision-making and improve the contribution of climate information.

    Article  Google Scholar 

  37. 37

    Ouzeau, G., Soubeyroux, J.-M., Schneider, M., Vautard, R. & Planton, S. Heat waves analysis over France in present and future climate: application of a new method on the EURO-CORDEX ensemble. Clim. Serv. 4, 1–12 (2016).

    Article  Google Scholar 

  38. 38

    Wilby, R. L. & Dessai, S. Robust adaptation to climate change. Weather 65, 180–185 (2010).

    Article  Google Scholar 

  39. 39

    Druyan, L. M. Studies of 21st-century precipitation trends over West Africa. Int. J. Climatol. 31, 1415–1424 (2011).

    Article  Google Scholar 

  40. 40

    Biasutti, M. Forced Sahel rainfall trends in the CMIP5 archive. J. Geophys. Res. Atmos. 118, 1613–1623 (2013).

    Article  Google Scholar 

  41. 41

    Hourdin, F. et al. The art and science of climate model tuning. Bull. Am. Meteorol. Soc. 98, 589–602 (2016).

    Article  Google Scholar 

  42. 42

    Katragkou, E. et al. Regional climate hindcast simulations within EURO-CORDEX: evaluation of a WRF multi-physics ensemble. Geosci. Model Dev. 8, 603–618 (2015).

    Article  Google Scholar 

  43. 43

    Box, G. E. P., Hunter, J. S. & Hunter, W. G. Statistics for Experimenters: Design, Innovation, and Discovery (Wiley, 2005).

    Google Scholar 

  44. 44

    Mearns, L. O. et al. Climate change projections of the North American Regional Climate Change Assessment Program (NARCCAP). Climatic Change 120, 965–975 (2013).

    Article  Google Scholar 

  45. 45

    Benestad, R. E., Senan, R. & Orsolini, Y. The use of regression for assessing a seasonal forecast model experiment. Earth Syst. Dynam. 7, 851–861 (2016).

    Article  Google Scholar 

  46. 46

    Warner, T. T. Quality assurance in atmospheric modeling. Bull. Am. Meteorol. Soc. 92, 1601–1610 (2011).

    Article  Google Scholar 

  47. 47

    Walton, P. J., Yarker, M. B., Mesquita, M. D. S. & Otto, F. E. L. Helping to make sense of regional climate modeling: professional development for scientists and decision-makers anytime, anywhere. Bull. Am. Meteorol. Soc. 97, 1173–1185 (2016).

    Article  Google Scholar 

  48. 48

    Skamarock, W. et al. A Description of the Advanced Research WRF Version 3 (UCAR/NCAR, 2008);

    Google Scholar 

  49. 49

    Salvador, N. et al. Evaluation of weather research and forecasting model parameterizations under sea-breeze conditions in a North Sea coastal environment. J. Meteorol. Res. 30, 998–1018 (2016).

    Article  Google Scholar 

  50. 50

    Harrison, S. P. et al. Evaluation of CMIP5 palaeo-simulations to improve climate projections. Nat. Clim. Change 5, 735–743 (2015).

    Article  Google Scholar 

  51. 51

    Gleckler, P. et al. A more powerful reality test for climate models. Eos 97, (2016).

  52. 52

    Runge, M. C., Stroeve, J. C., Barrett, A. P. & McDonald-Madden, E. Detecting failure of climate predictions. Nat. Clim. Change 6, 861–864 (2016).

    Article  Google Scholar 

  53. 53

    Perkins, S. E., Pitman, A. J., Holbrook, N. J. & McAneney, J. Evaluation of the AR4 climate models' simulated daily maximum temperature, minimum temperature, and precipitation over Australia using probability density functions. J. Clim. 20, 4356–4376 (2007).

    Article  Google Scholar 

  54. 54

    Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).

    Article  Google Scholar 

  55. 55

    Sansom, P. G., Stephenson, D. B., Ferro, C. A. T., Zappa, G. & Shaffrey, L. Simple uncertainty frameworks for selecting weighting schemes and interpreting multimodel ensemble climate change experiments. J. Clim. 26, 4017–4037 (2013).

    Article  Google Scholar 

  56. 56

    Lindgren, F. & Rue, H. Bayesian spatial modelling with R-INLA. J. Stat. Softw. 63, (2015).

  57. 57

    Masson, D. & Knutti, R. Climate model genealogy. Geophys. Res. Lett. 38, L08703 (2011).

    Article  Google Scholar 

  58. 58

    Hawkins, E. & Sutton, R. The potential to narrow uncertainty in regional climate predictions. Bull Am. Meteorol. Soc. 90, 1095–1107 (2009).

    Article  Google Scholar 

  59. 59

    Murphy, J. M. et al. A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Phil. Trans. R. Soc. A 365, 1993–2028 (2007).

    CAS  Article  Google Scholar 

  60. 60

    Fischer, E. M., Lawrence, D. M. & Sanderson, B. M. Quantifying uncertainties in projections of extremes—a perturbed land surface parameter experiment. Clim. Dynam. 37, 1381–1398 (2011).

    Article  Google Scholar 

  61. 61

    Tran, G. T. et al. Building a traceable climate model hierarchy with multi-level emulators. Adv. Stat. Climatol. Meteorol. Oceanogr. 2, 17–37 (2016).

    Article  Google Scholar 

  62. 62

    Baumberger, C., Knutti, R. & Hirsch Hadorn, G. Building confidence in climate model projections: an analysis of inferences from fit. WIREs Clim. Change 8, e454 (2017).

    Article  Google Scholar 

  63. 63

    Nguyen, H., Katzfuss, M., Cressie, N. & Braverman, A. Spatio-temporal data fusion for very large remote sensing datasets. Technometrics 56, 174–185 (2014).

    Article  Google Scholar 

  64. 64

    Katzfuss, M. A Multi-resolution approximation for massive spatial datasets. J. Am. Stat. Assoc. 112, 201–214 (2017).

    CAS  Article  Google Scholar 

  65. 65

    Stott, P. A., Allen, M. R. & Jones, G. S. Estimating signal amplitudes in optimal finger printing. Part II: application to general circulation models. Clim. Dynam. 21, 493–500 (2002).

    Article  Google Scholar 

  66. 66

    Guttorp, P. et al. Assessing the uncertainty in projecting local mean sea level from global temperature. J. Appl. Meteorol. Climatol. 53, 2163–2170 (2014).

    Article  Google Scholar 

  67. 67

    Hasselmann, K. Optimal fingerprints for the detection of time-dependent climate change. J. Clim. 6, 1957–1971 (1993).

    Article  Google Scholar 

  68. 68

    Hegerl, G. C. et al. Detecting greenhouse-gas-induced climate change with an optimal fingerprint method. J. Clim. 9, 2281–2306 (1996).

    Article  Google Scholar 

  69. 69

    Barnett, T. P. Comparison of near-surface air temperature variability in 11 coupled global climate models. J. Clim. 12, 511–518 (1999).

    Article  Google Scholar 

  70. 70

    Oppenheimer, M. et al. In Climate Change 2014: Impacts, Adaptation, and Vulnerability (eds Field, C. B. et al.) 1039–1099 (Cambridge Univ. Press, 2014).

    Google Scholar 

  71. 71

    Ferro, C. A. T., Jupp, T. E., Lambert, F. H., Huntingford, C. & Cox, P. M. Model complexity versus ensemble size: allocating resources for climate prediction. Phil. Trans. R. Soc. A 370, 1087–1099 (2012).

    Article  Google Scholar 

Download references


We thank E. U. Reed for his help with designing Fig. 3, and J. Rougier for providing constructive feedback on a previous version of the manuscript. J.S. is supported through the Norwegian Research Council projects ClimateXL (grant 243953) and TWEX (grant 255037). This work has been supported by the Statistical Analysis of Climate Projections project (eSACP; NordForsk grant number 74456), the COWI Foundation, the C-ICE Project (the Norwegian Research Council, project number 248803) and EU MCSA grant 707262 – LAWINE. The National Center for Atmospheric Research is sponsored by the National Science Foundation (NSF); M.R.T. is supported by the NSF's Decadal and Regional Climate Prediction using Earth System Models (EaSM-3) grants AGS-1419563, AGS-1419558 and AGS-1419504; T.L.T. is supported by the Norwegian Research Council through grant 243814.

Author information




R.B., J.S. and T.L.T. conceived the original idea for the paper, wrote the text and generated Figs 1 and 3 and the figure in Box 1. T.L.T., P.G. and M.D. made Fig. 2. All authors contributed to discussions and writing of the text.

Corresponding author

Correspondence to Rasmus Benestad.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Benestad, R., Sillmann, J., Thorarinsdottir, T. et al. New vigour involving statisticians to overcome ensemble fatigue. Nature Clim Change 7, 697–703 (2017).

Download citation

Further reading


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing