Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Detecting failure of climate predictions


The practical consequences of climate change challenge society to formulate responses that are more suited to achieving long-term objectives, even if those responses have to be made in the face of uncertainty1,2. Such a decision-analytic focus uses the products of climate science as probabilistic predictions about the effects of management policies3. Here we present methods to detect when climate predictions are failing to capture the system dynamics. For a single model, we measure goodness of fit based on the empirical distribution function, and define failure when the distribution of observed values significantly diverges from the modelled distribution. For a set of models, the same statistic can be used to provide relative weights for the individual models, and we define failure when there is no linear weighting of the ensemble models that produces a satisfactory match to the observations. Early detection of failure of a set of predictions is important for improving model predictions and the decisions based on them. We show that these methods would have detected a range shift in northern pintail 20 years before it was actually discovered, and are increasingly giving more weight to those climate models that forecast a September ice-free Arctic by 2055.

This is a preview of subscription content

Access options

Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Figure 1: Analysis of trends in distribution of northern pintails, 1961–2015.
Figure 2: Analysis of trends in the extent of sea ice in the Arctic, 1953–2015.
Figure 3: Forecast extent of sea ice in the Arctic in 2055 as a function of the weights on the 11 CMIP5 models over the course of the observed record, and assuming the RCP8.5 forcing scenario.


  1. McDonald-Madden, E., Runge, M. C., Martin, T. G. & Possingham, H. Optimal timing for managed relocation of species faced with climate change. Nature Clim. Change 1, 261–265 (2011).

    Article  Google Scholar 

  2. Conroy, M. J., Runge, M. C., Nichols, J. D., Stodola, K. W. & Cooper, R. J. Conservation in the face of climate change: the roles of alternative models, monitoring, and adaptation in confronting and reducing uncertainty. Biol. Conserv. 144, 1204–1213 (2011).

    Article  Google Scholar 

  3. Terando, A., Keller, K. & Easterling, W. E. Probabilistic projections of agro-climate indices in North America. J. Geophys. Res. 117, D08115 (2012).

    Article  Google Scholar 

  4. Lawler, J. J. et al. Resource management in a changing and uncertain climate. Front. Ecol. Environ. 8, 35–43 (2010).

    Article  Google Scholar 

  5. IPCC Climate Change 2013: The Physical Science Basis (Cambridge Univ. Press, 2013).

  6. Taleb, N. N. The Black Swan: The Impact of the Highly Improbable (Random House, 2007).

    Google Scholar 

  7. Pahl-Wostl, C. A conceptual framework for analysing adaptive capacity and multi-level learning processes in resource governance regimes. Glob. Environ. Change 19, 354–365 (2009).

    Article  Google Scholar 

  8. Argyris, C. & Schön, D. A. Organizational Learning: A Theory of Action Perspective (Addison-Wesley, 1978).

    Google Scholar 

  9. Stroeve, J., Holland, M. M., Meier, W., Scambos, T. & Serreze, M. Arctic sea ice decline: faster than forecast. Geophys. Res. Lett. 34, L09501 (2007).

    Article  Google Scholar 

  10. Miller, M. R. & Duncan, D. C. The northern pintail in North America: status and conservation needs of a struggling population. Wildl. Soc. Bull. 27, 788–800 (1999).

    Google Scholar 

  11. Dawid, A. P. Statistical theory: the prequential approach. J. R. Stat. Soc. A 147, 278–292 (1984).

    Article  Google Scholar 

  12. Williams, B. K., Johnson, F. A. & Wilkins, K. Uncertainty and the adaptive management of waterfowl harvests. J. Wildl. Manage. 60, 223–232 (1996).

    Article  Google Scholar 

  13. Gneiting, T., Balabdaoui, F. & Raftery, A. E. Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. B 69, 243–268 (2007).

    Article  Google Scholar 

  14. Seillier-Moiseiwitsch, F. & Dawid, A. P. On testing the validity of sequential probability forecasts. J. Am. Statist. Assoc. 88, 355–359 (1993).

    Google Scholar 

  15. Stephens, M. A. in Goodness-of-fit Techniques Vol. 68 (eds D’Agostino, R. B. & Stephens, M. A.) Ch. 4, 97–193 (Marcel Dukker, 1986).

    Google Scholar 

  16. Austin, J. E. & Miller, M. R. in The Birds of North America (ed. Poole, A.) (Cornell Laboratory of Ornithology, 1995).

    Google Scholar 

  17. Hestbeck, J. B. Response of northern pintail breeding populations to drought, 1961–92. J. Wildl. Manage. 59, 9–15 (1995).

    Article  Google Scholar 

  18. US Fish and Wildlife Service Adaptive Harvest Management: 2014 Hunting Season (United States Department of Interior, 2014).

  19. Stroeve, J. et al. The Arctic’s rapidly shrinking sea ice cover: a research synthesis. Climatic Change 110, 1005–1027 (2012).

    Article  Google Scholar 

  20. Amstrup, S. C. et al. Greenhouse gas mitigation can reduce sea-ice loss and increase polar bear persistence. Nature 468, 955–958 (2010).

    CAS  Article  Google Scholar 

  21. Hunter, C. M. et al. Climate change threatens polar bear populations: a stochastic demographic analysis. Ecology 91, 2883–2897 (2010).

    Article  Google Scholar 

  22. Stroeve, J. C. et al. Trends in Arctic sea ice extent from CMIP5, CMIP3 and observations. Geophys. Res. Lett. 39, L16502 (2012).

    Article  Google Scholar 

  23. Cavalieri, D. J., Parkinson, C. L., Gloersen, P. & Zwally, H. J. Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I-SSMIS Passive Microwave Data (NASA DAAC at the National Snow and Ice Data Center, 1996).

    Google Scholar 

  24. Rayner, N. A. et al. Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century. J. Geophys. Res. 108, 4407 (2003).

    Article  Google Scholar 

  25. Meier, W. N., Stroeve, J. & Fetterer, F. Whither Arctic sea ice? A clear signal of decline regionally, seasonally and extending beyond the satellite record. Ann. Glaciol. 46, 428–434 (2007).

    Article  Google Scholar 

  26. Stroeve, J., Barrett, A., Serreze, M. & Schweiger, A. Using records from submarine, aircraft and satellites to evaluate climate model simulations of Arctic sea ice thickness. Cryosphere 8, 1839–1854 (2014).

    Article  Google Scholar 

  27. Babu, G. J. & Rao, C. R. Goodness-of-fit tests when parameters are estimated. Sankhyā: Indian J. Stat. 66, 63–74 (2004).

    Google Scholar 

  28. Fletcher, R. Practical Methods of Optimization (John Wiley, 1987).

    Google Scholar 

  29. Shanno, D. F. Conditioning of quasi-Newton methods for function minimization. Math. Comput. 24, 647–656 (1970).

    Article  Google Scholar 

Download references


The authors acknowledge the support of the NOAA Climate Program Office, Modeling, Analysis, Predictions, and Projections (MAPP) Program as part of the CMIP5 Task Force. J.C.S. was supported by NOAA grant NA10OAR4320142. E.M.-M. was supported by an ARC DECRA Fellowship and by the ARC Centre for Excellence in Environmental Decisions.

Author information

Authors and Affiliations



M.C.R. and E.M.-M. conceived of the methods; J.C.S. and A.P.B. extracted the sea-ice forecasts from the CMIP5 models; M.C.R. analysed the pintail and sea-ice data and prepared the figures; and M.C.R., E.M.-M. and J.C.S. co-wrote the paper.

Corresponding author

Correspondence to Michael C. Runge.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Information

Supplementary Information (PDF 954 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Runge, M., Stroeve, J., Barrett, A. et al. Detecting failure of climate predictions. Nature Clim Change 6, 861–864 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:

Further reading


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing