New vigour involving statisticians to overcome ensemble fatigue


Climate simulation data comprise a range of different phenomena with complex and interacting processes. Yet our understanding of the climate is incomplete despite the huge volumes of data, of which only a small fraction has been explored, and many questions remain, particularly those on the character and origin of uncertainties associated with model simulations and how further modelling efforts can improve understanding. Here, we question whether climate model information could be used more effectively and how so-called 'ensembles of opportunity' should be interpreted. Statisticians can contribute substantially to designing 'smarter' ensemble experiments, improving the distillation of information from ensembles, and helping interpret the relative merits of additional simulations. Future progress may be enhanced by increasing collaborations with statisticians.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Figure 1: A schematic of the state probability space for a multi-model ensemble and the real world represented by a multi-dimensional probability density function.
Figure 2: Uncertainty of a statistical downscaling of 2050 sea-level rise projections relative to 1999 in Bergen, Norway.
Figure 3


  1. 1

    Weigel, A. P., Liniger, M. A. & Appenzeller, C. Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Q. J. R. Meteorol. Soc. 134, 241–260 (2008).

  2. 2

    Bruyère, C. et al. Impact of Climate Change on Gulf of Mexico Hurricanes Technical Note NCAR/TN-535+STR. (NCAR, 2017).

  3. 3

    Smith, M. J. et al. Changing how Earth system modeling is done to provide more useful information for decision making, science, and society. Bull. Am. Meteorol. Soc. 95, 1453–1464 (2014). This paper calls for new ways of developing Earth system models to enable more relevant and informative projections.

  4. 4

    Taylor, K. E., Stouffer, R. J. & Meehl, G. A. An overview of CMIP5 and the experiment design. Bull. Am. Meteorol. Soc. 93, 485–498 (2012).

  5. 5

    Benestad, R. In Oxford Research Encyclopedia of Climate Science (Oxford Univ. Press, 2016).

  6. 6

    Overpeck, J. T., Meehl, G. A., Bony, S. & Easterling, D. R. Climate data challenges in the 21st century. Science 331, 700–702 (2011).

  7. 7

    Christensen, J. H. & Christensen, O. B. A summary of the PRUDENCE model projections of changes in European climate by the end of this century. Climatic Change 81, 7–30 (2007).

  8. 8

    van der Linden, P. & Mitchell, F. B. (eds) Ensembles: Climate Change and its Impacts: Summary of Research and Results from the ENSEMBLES Project (Met Office Hadley Centre, 2009);

  9. 9

    Giorgi, F. & Gutowski, W. J. Regional dynamical downscaling and the CORDEX initiative. Annu. Rev. Environ. Resour. 40, 467–490 (2015).

  10. 10

    Eyring, V. et al. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. 9, 1937–1958 (2016).

  11. 11

    IPCC Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) (Cambridge Univ. Press, 2013).

  12. 12

    IPCC Climate Change 2007: The Physical Science Basis (eds Solomon. S. et al.) (Cambridge Univ. Press, 2007).

  13. 13

    Tebaldi, C. & Knutti, R. The use of the multi-model ensemble in probabilistic climate projections. Phil. Trans. R. Soc. A 365, 2053–2075 (2007).

  14. 14

    Smith, R. L., Tebaldi, C., Nychka, D. & Mearns, L. O. Bayesian modeling of uncertainty in ensembles of climate models. J. Am. Stat. Assoc. 104, 97–116 (2009).

  15. 15

    Sanderson, B. M., Knutti, R. & Caldwell, P. Addressing interdependency in a multimodel ensemble by interpolation of model properties. J. Clim. 28, 5150–5170 (2015).

  16. 16

    Stainforth, D., Allen, M., Tredger, E. & Smith, L. Confidence, uncertainty and decision-support relevance in climate predictions. Phil. Trans. R. Soc. A 365, 2145–2161 (2007).

  17. 17

    Knutti, R. & Sedláček, J. Robustness and uncertainties in the new CMIP5 climate model projections. Nat. Clim. Change 3, 369–373 (2013). Argues that the new generation of more complex models running scenarios for the IPCC's AR5 is widely, and perhaps naively, expected to provide more detailed and more certain projections.

  18. 18

    Gneiting, T. & Raftery, A. E. Weather forecasting with ensemble methods. Science 310, 248–249 (2005).

  19. 19

    Rougier, J. Ensemble averaging and mean squared error. J. Clim. 29, 8865–8870 (2016).

  20. 20

    Rougier, J. Probabilistic inference for future climate using an ensemble of climate model evaluations. Climatic Change 81, 247–264 (2007).

  21. 21

    Schefzik, R., Thorarinsdottir, T. L. & Gneiting, T. Uncertainty quantification in complex simulation models using ensemble copula coupling. Stat. Sci. 28, 616–640 (2013).

  22. 22

    Sansom, P. G., Ferro, C. A. T., Stephenson, D. B., Goddard, L. & Mason, S. J. Best practices for postprocessing ensemble climate forecasts. Part I: Selecting appropriate recalibration methods. J. Clim. 29, 7247–7264 (2016). Describes a new recalibration method that involves adjustments for both unconditional and conditional biases in the mean, variance, and trend.

  23. 23

    Jolliffe, I. T. & Stephenson, D. B. Forecast Verification: a Practitioner's Guide in Atmospheric Science (Wiley, 2003).

  24. 24

    Notz, D. How well must climate models agree with observations? Phil. Trans. R. Soc. A 373, 20140164 (2015).

  25. 25

    Collins, M. Ensembles and probabilities: a new era in the prediction of climate change. Phil. Trans. R. Soc. A 365, 1957–1970 (2007).

  26. 26

    Deser, C., Knutti, R., Solomon, S. & Phillips, A. S. Communication of the role of natural variability in future North American climate. Nat. Clim. Change 2, 775–779 (2012).

  27. 27

    Katz, R. W. et al. Uncertainty analysis in climate change assessments. Nat. Clim. Change 3, 769–771 (2013).

  28. 28

    Jun, M., Knutti, R. & Nychka, D. W. Spatial analysis to quantify numerical model bias and dependence: how many climate models are there? J. Am. Stat. Assoc. 103, 934–947 (2008).

  29. 29

    Edwards, P. N. A Vast Machine: Computer Models, Climate Data, and the Politics of Global Warming (MIT Press, 2010).

  30. 30

    Chandler, R. E. Exploiting strength, discounting weakness: combining information from multiple climate simulators. Phil. Trans. R. Soc. A 371, 20120388 (2013).

  31. 31

    Stephenson, D. B., Collins, M., Rougier, J. C. & Chandler, R. E. Statistical problems in the probabilistic prediction of climate change. Environmetrics 23, 364–372 (2012).

  32. 32

    Rougier, J., Goldstein, M. & House, L. Second-order exchangeability analysis for multimodel ensembles. J. Am. Stat. Assoc. 108, 852–863 (2013).

  33. 33

    Thorarinsdottir, T. L., Guttorp, P., Drews, M., Kaspersen, P. S. & de Bruin, K. Sea level adaptation decisions under uncertainty. Water Resour. Res. (in the press).

  34. 34

    Tye, M. R., Holland, G. J. & Done, J. M. Rethinking failure: time for closer engineer–scientist collaborations on design. Proc. Inst. Civ. Eng. Forensic Eng. 168, 49–57 (2015).

  35. 35

    Gutowski Jr., W. J. et al. WCRP COordinated Regional Downscaling EXperiment (CORDEX): a diagnostic MIP for CMIP6. Geosci Model Dev. 9, 4087–4095 (2016).

  36. 36

    Weaver, C. P. et al. Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks. WIREs Clim. Change 4, 39–60 (2013). This paper discusses how climate modelling is used to support so-called decision-making and improve the contribution of climate information.

  37. 37

    Ouzeau, G., Soubeyroux, J.-M., Schneider, M., Vautard, R. & Planton, S. Heat waves analysis over France in present and future climate: application of a new method on the EURO-CORDEX ensemble. Clim. Serv. 4, 1–12 (2016).

  38. 38

    Wilby, R. L. & Dessai, S. Robust adaptation to climate change. Weather 65, 180–185 (2010).

  39. 39

    Druyan, L. M. Studies of 21st-century precipitation trends over West Africa. Int. J. Climatol. 31, 1415–1424 (2011).

  40. 40

    Biasutti, M. Forced Sahel rainfall trends in the CMIP5 archive. J. Geophys. Res. Atmos. 118, 1613–1623 (2013).

  41. 41

    Hourdin, F. et al. The art and science of climate model tuning. Bull. Am. Meteorol. Soc. 98, 589–602 (2016).

  42. 42

    Katragkou, E. et al. Regional climate hindcast simulations within EURO-CORDEX: evaluation of a WRF multi-physics ensemble. Geosci. Model Dev. 8, 603–618 (2015).

  43. 43

    Box, G. E. P., Hunter, J. S. & Hunter, W. G. Statistics for Experimenters: Design, Innovation, and Discovery (Wiley, 2005).

  44. 44

    Mearns, L. O. et al. Climate change projections of the North American Regional Climate Change Assessment Program (NARCCAP). Climatic Change 120, 965–975 (2013).

  45. 45

    Benestad, R. E., Senan, R. & Orsolini, Y. The use of regression for assessing a seasonal forecast model experiment. Earth Syst. Dynam. 7, 851–861 (2016).

  46. 46

    Warner, T. T. Quality assurance in atmospheric modeling. Bull. Am. Meteorol. Soc. 92, 1601–1610 (2011).

  47. 47

    Walton, P. J., Yarker, M. B., Mesquita, M. D. S. & Otto, F. E. L. Helping to make sense of regional climate modeling: professional development for scientists and decision-makers anytime, anywhere. Bull. Am. Meteorol. Soc. 97, 1173–1185 (2016).

  48. 48

    Skamarock, W. et al. A Description of the Advanced Research WRF Version 3 (UCAR/NCAR, 2008);

  49. 49

    Salvador, N. et al. Evaluation of weather research and forecasting model parameterizations under sea-breeze conditions in a North Sea coastal environment. J. Meteorol. Res. 30, 998–1018 (2016).

  50. 50

    Harrison, S. P. et al. Evaluation of CMIP5 palaeo-simulations to improve climate projections. Nat. Clim. Change 5, 735–743 (2015).

  51. 51

    Gleckler, P. et al. A more powerful reality test for climate models. Eos 97, (2016).

  52. 52

    Runge, M. C., Stroeve, J. C., Barrett, A. P. & McDonald-Madden, E. Detecting failure of climate predictions. Nat. Clim. Change 6, 861–864 (2016).

  53. 53

    Perkins, S. E., Pitman, A. J., Holbrook, N. J. & McAneney, J. Evaluation of the AR4 climate models' simulated daily maximum temperature, minimum temperature, and precipitation over Australia using probability density functions. J. Clim. 20, 4356–4376 (2007).

  54. 54

    Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).

  55. 55

    Sansom, P. G., Stephenson, D. B., Ferro, C. A. T., Zappa, G. & Shaffrey, L. Simple uncertainty frameworks for selecting weighting schemes and interpreting multimodel ensemble climate change experiments. J. Clim. 26, 4017–4037 (2013).

  56. 56

    Lindgren, F. & Rue, H. Bayesian spatial modelling with R-INLA. J. Stat. Softw. 63, (2015).

  57. 57

    Masson, D. & Knutti, R. Climate model genealogy. Geophys. Res. Lett. 38, L08703 (2011).

  58. 58

    Hawkins, E. & Sutton, R. The potential to narrow uncertainty in regional climate predictions. Bull Am. Meteorol. Soc. 90, 1095–1107 (2009).

  59. 59

    Murphy, J. M. et al. A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Phil. Trans. R. Soc. A 365, 1993–2028 (2007).

  60. 60

    Fischer, E. M., Lawrence, D. M. & Sanderson, B. M. Quantifying uncertainties in projections of extremes—a perturbed land surface parameter experiment. Clim. Dynam. 37, 1381–1398 (2011).

  61. 61

    Tran, G. T. et al. Building a traceable climate model hierarchy with multi-level emulators. Adv. Stat. Climatol. Meteorol. Oceanogr. 2, 17–37 (2016).

  62. 62

    Baumberger, C., Knutti, R. & Hirsch Hadorn, G. Building confidence in climate model projections: an analysis of inferences from fit. WIREs Clim. Change 8, e454 (2017).

  63. 63

    Nguyen, H., Katzfuss, M., Cressie, N. & Braverman, A. Spatio-temporal data fusion for very large remote sensing datasets. Technometrics 56, 174–185 (2014).

  64. 64

    Katzfuss, M. A Multi-resolution approximation for massive spatial datasets. J. Am. Stat. Assoc. 112, 201–214 (2017).

  65. 65

    Stott, P. A., Allen, M. R. & Jones, G. S. Estimating signal amplitudes in optimal finger printing. Part II: application to general circulation models. Clim. Dynam. 21, 493–500 (2002).

  66. 66

    Guttorp, P. et al. Assessing the uncertainty in projecting local mean sea level from global temperature. J. Appl. Meteorol. Climatol. 53, 2163–2170 (2014).

  67. 67

    Hasselmann, K. Optimal fingerprints for the detection of time-dependent climate change. J. Clim. 6, 1957–1971 (1993).

  68. 68

    Hegerl, G. C. et al. Detecting greenhouse-gas-induced climate change with an optimal fingerprint method. J. Clim. 9, 2281–2306 (1996).

  69. 69

    Barnett, T. P. Comparison of near-surface air temperature variability in 11 coupled global climate models. J. Clim. 12, 511–518 (1999).

  70. 70

    Oppenheimer, M. et al. In Climate Change 2014: Impacts, Adaptation, and Vulnerability (eds Field, C. B. et al.) 1039–1099 (Cambridge Univ. Press, 2014).

  71. 71

    Ferro, C. A. T., Jupp, T. E., Lambert, F. H., Huntingford, C. & Cox, P. M. Model complexity versus ensemble size: allocating resources for climate prediction. Phil. Trans. R. Soc. A 370, 1087–1099 (2012).

Download references


We thank E. U. Reed for his help with designing Fig. 3, and J. Rougier for providing constructive feedback on a previous version of the manuscript. J.S. is supported through the Norwegian Research Council projects ClimateXL (grant 243953) and TWEX (grant 255037). This work has been supported by the Statistical Analysis of Climate Projections project (eSACP; NordForsk grant number 74456), the COWI Foundation, the C-ICE Project (the Norwegian Research Council, project number 248803) and EU MCSA grant 707262 – LAWINE. The National Center for Atmospheric Research is sponsored by the National Science Foundation (NSF); M.R.T. is supported by the NSF's Decadal and Regional Climate Prediction using Earth System Models (EaSM-3) grants AGS-1419563, AGS-1419558 and AGS-1419504; T.L.T. is supported by the Norwegian Research Council through grant 243814.

Author information

R.B., J.S. and T.L.T. conceived the original idea for the paper, wrote the text and generated Figs 1 and 3 and the figure in Box 1. T.L.T., P.G. and M.D. made Fig. 2. All authors contributed to discussions and writing of the text.

Correspondence to Rasmus Benestad.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Benestad, R., Sillmann, J., Thorarinsdottir, T. et al. New vigour involving statisticians to overcome ensemble fatigue. Nature Clim Change 7, 697–703 (2017).

Download citation

Further reading