Detecting failure of climate predictions

Published online:


The practical consequences of climate change challenge society to formulate responses that are more suited to achieving long-term objectives, even if those responses have to be made in the face of uncertainty1,2. Such a decision-analytic focus uses the products of climate science as probabilistic predictions about the effects of management policies3. Here we present methods to detect when climate predictions are failing to capture the system dynamics. For a single model, we measure goodness of fit based on the empirical distribution function, and define failure when the distribution of observed values significantly diverges from the modelled distribution. For a set of models, the same statistic can be used to provide relative weights for the individual models, and we define failure when there is no linear weighting of the ensemble models that produces a satisfactory match to the observations. Early detection of failure of a set of predictions is important for improving model predictions and the decisions based on them. We show that these methods would have detected a range shift in northern pintail 20 years before it was actually discovered, and are increasingly giving more weight to those climate models that forecast a September ice-free Arctic by 2055.

  • Subscribe to Nature Climate Change for full access:



Additional access options:

Already a subscriber?  Log in  now or  Register  for online access.


  1. 1.

    , , & Optimal timing for managed relocation of species faced with climate change. Nature Clim. Change 1, 261–265 (2011).

  2. 2.

    , , , & Conservation in the face of climate change: the roles of alternative models, monitoring, and adaptation in confronting and reducing uncertainty. Biol. Conserv. 144, 1204–1213 (2011).

  3. 3.

    , & Probabilistic projections of agro-climate indices in North America. J. Geophys. Res. 117, D08115 (2012).

  4. 4.

    et al. Resource management in a changing and uncertain climate. Front. Ecol. Environ. 8, 35–43 (2010).

  5. 5.

    IPCC Climate Change 2013: The Physical Science Basis (Cambridge Univ. Press, 2010).

  6. 6.

    The Black Swan: The Impact of the Highly Improbable (Random House, 2010).

  7. 7.

    A conceptual framework for analysing adaptive capacity and multi-level learning processes in resource governance regimes. Glob. Environ. Change 19, 354–365 (2009).

  8. 8.

    & Organizational Learning: A Theory of Action Perspective (Addison-Wesley, 2009).

  9. 9.

    , , , & Arctic sea ice decline: faster than forecast. Geophys. Res. Lett. 34, L09501 (2007).

  10. 10.

    & The northern pintail in North America: status and conservation needs of a struggling population. Wildl. Soc. Bull. 27, 788–800 (1999).

  11. 11.

    Statistical theory: the prequential approach. J. R. Stat. Soc. A 147, 278–292 (1984).

  12. 12.

    , & Uncertainty and the adaptive management of waterfowl harvests. J. Wildl. Manage. 60, 223–232 (1996).

  13. 13.

    , & Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. B 69, 243–268 (2007).

  14. 14.

    & On testing the validity of sequential probability forecasts. J. Am. Statist. Assoc. 88, 355–359 (1993).

  15. 15.

    in Goodness-of-fit Techniques Vol. 68 (eds D’Agostino, R. B. & Stephens, M. A.) Ch. 4, 97–193 (Marcel Dukker, 1993).

  16. 16.

    & in The Birds of North America (ed. Poole, A.) (Cornell Laboratory of Ornithology, 1993).

  17. 17.

    Response of northern pintail breeding populations to drought, 1961–92. J. Wildl. Manage. 59, 9–15 (1995).

  18. 18.

    US Fish and Wildlife Service Adaptive Harvest Management: 2014 Hunting Season (United States Department of Interior, 1995).

  19. 19.

    et al. The Arctic’s rapidly shrinking sea ice cover: a research synthesis. Climatic Change 110, 1005–1027 (2012).

  20. 20.

    et al. Greenhouse gas mitigation can reduce sea-ice loss and increase polar bear persistence. Nature 468, 955–958 (2010).

  21. 21.

    et al. Climate change threatens polar bear populations: a stochastic demographic analysis. Ecology 91, 2883–2897 (2010).

  22. 22.

    et al. Trends in Arctic sea ice extent from CMIP5, CMIP3 and observations. Geophys. Res. Lett. 39, L16502 (2012).

  23. 23.

    , , & Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I-SSMIS Passive Microwave Data (NASA DAAC at the National Snow and Ice Data Center, 2012).

  24. 24.

    et al. Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century. J. Geophys. Res. 108, 4407 (2003).

  25. 25.

    , & Whither Arctic sea ice? A clear signal of decline regionally, seasonally and extending beyond the satellite record. Ann. Glaciol. 46, 428–434 (2007).

  26. 26.

    , , & Using records from submarine, aircraft and satellites to evaluate climate model simulations of Arctic sea ice thickness. Cryosphere 8, 1839–1854 (2014).

  27. 27.

    & Goodness-of-fit tests when parameters are estimated. Sankhyā: Indian J. Stat. 66, 63–74 (2004).

  28. 28.

    Practical Methods of Optimization (John Wiley, 2004).

  29. 29.

    Conditioning of quasi-Newton methods for function minimization. Math. Comput. 24, 647–656 (1970).

Download references


The authors acknowledge the support of the NOAA Climate Program Office, Modeling, Analysis, Predictions, and Projections (MAPP) Program as part of the CMIP5 Task Force. J.C.S. was supported by NOAA grant NA10OAR4320142. E.M.-M. was supported by an ARC DECRA Fellowship and by the ARC Centre for Excellence in Environmental Decisions.

Author information


  1. USGS Patuxent Wildlife Research Center, Laurel, Maryland 20708, USA

    • Michael C. Runge
  2. National Snow and Ice Data Center, University of Colorado, Boulder, Colorado 80309, USA

    • Julienne C. Stroeve
    •  & Andrew P. Barrett
  3. University College London, London WC1E 6BT, UK

    • Julienne C. Stroeve
  4. School of Geography, Planning, and Environmental Management, University of Queensland, St Lucia, Queensland 4072, Australia

    • Eve McDonald-Madden


  1. Search for Michael C. Runge in:

  2. Search for Julienne C. Stroeve in:

  3. Search for Andrew P. Barrett in:

  4. Search for Eve McDonald-Madden in:


M.C.R. and E.M.-M. conceived of the methods; J.C.S. and A.P.B. extracted the sea-ice forecasts from the CMIP5 models; M.C.R. analysed the pintail and sea-ice data and prepared the figures; and M.C.R., E.M.-M. and J.C.S. co-wrote the paper.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Michael C. Runge.

Supplementary information

PDF files

  1. 1.

    Supplementary Information

    Supplementary Information