Abstract
The practical consequences of climate change challenge society to formulate responses that are more suited to achieving long-term objectives, even if those responses have to be made in the face of uncertainty1,2. Such a decision-analytic focus uses the products of climate science as probabilistic predictions about the effects of management policies3. Here we present methods to detect when climate predictions are failing to capture the system dynamics. For a single model, we measure goodness of fit based on the empirical distribution function, and define failure when the distribution of observed values significantly diverges from the modelled distribution. For a set of models, the same statistic can be used to provide relative weights for the individual models, and we define failure when there is no linear weighting of the ensemble models that produces a satisfactory match to the observations. Early detection of failure of a set of predictions is important for improving model predictions and the decisions based on them. We show that these methods would have detected a range shift in northern pintail 20 years before it was actually discovered, and are increasingly giving more weight to those climate models that forecast a September ice-free Arctic by 2055.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
McDonald-Madden, E., Runge, M. C., Martin, T. G. & Possingham, H. Optimal timing for managed relocation of species faced with climate change. Nature Clim. Change 1, 261–265 (2011).
Conroy, M. J., Runge, M. C., Nichols, J. D., Stodola, K. W. & Cooper, R. J. Conservation in the face of climate change: the roles of alternative models, monitoring, and adaptation in confronting and reducing uncertainty. Biol. Conserv. 144, 1204–1213 (2011).
Terando, A., Keller, K. & Easterling, W. E. Probabilistic projections of agro-climate indices in North America. J. Geophys. Res. 117, D08115 (2012).
Lawler, J. J. et al. Resource management in a changing and uncertain climate. Front. Ecol. Environ. 8, 35–43 (2010).
IPCC Climate Change 2013: The Physical Science Basis (Cambridge Univ. Press, 2013).
Taleb, N. N. The Black Swan: The Impact of the Highly Improbable (Random House, 2007).
Pahl-Wostl, C. A conceptual framework for analysing adaptive capacity and multi-level learning processes in resource governance regimes. Glob. Environ. Change 19, 354–365 (2009).
Argyris, C. & Schön, D. A. Organizational Learning: A Theory of Action Perspective (Addison-Wesley, 1978).
Stroeve, J., Holland, M. M., Meier, W., Scambos, T. & Serreze, M. Arctic sea ice decline: faster than forecast. Geophys. Res. Lett. 34, L09501 (2007).
Miller, M. R. & Duncan, D. C. The northern pintail in North America: status and conservation needs of a struggling population. Wildl. Soc. Bull. 27, 788–800 (1999).
Dawid, A. P. Statistical theory: the prequential approach. J. R. Stat. Soc. A 147, 278–292 (1984).
Williams, B. K., Johnson, F. A. & Wilkins, K. Uncertainty and the adaptive management of waterfowl harvests. J. Wildl. Manage. 60, 223–232 (1996).
Gneiting, T., Balabdaoui, F. & Raftery, A. E. Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. B 69, 243–268 (2007).
Seillier-Moiseiwitsch, F. & Dawid, A. P. On testing the validity of sequential probability forecasts. J. Am. Statist. Assoc. 88, 355–359 (1993).
Stephens, M. A. in Goodness-of-fit Techniques Vol. 68 (eds D’Agostino, R. B. & Stephens, M. A.) Ch. 4, 97–193 (Marcel Dukker, 1986).
Austin, J. E. & Miller, M. R. in The Birds of North America (ed. Poole, A.) (Cornell Laboratory of Ornithology, 1995).
Hestbeck, J. B. Response of northern pintail breeding populations to drought, 1961–92. J. Wildl. Manage. 59, 9–15 (1995).
US Fish and Wildlife Service Adaptive Harvest Management: 2014 Hunting Season (United States Department of Interior, 2014).
Stroeve, J. et al. The Arctic’s rapidly shrinking sea ice cover: a research synthesis. Climatic Change 110, 1005–1027 (2012).
Amstrup, S. C. et al. Greenhouse gas mitigation can reduce sea-ice loss and increase polar bear persistence. Nature 468, 955–958 (2010).
Hunter, C. M. et al. Climate change threatens polar bear populations: a stochastic demographic analysis. Ecology 91, 2883–2897 (2010).
Stroeve, J. C. et al. Trends in Arctic sea ice extent from CMIP5, CMIP3 and observations. Geophys. Res. Lett. 39, L16502 (2012).
Cavalieri, D. J., Parkinson, C. L., Gloersen, P. & Zwally, H. J. Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I-SSMIS Passive Microwave Data (NASA DAAC at the National Snow and Ice Data Center, 1996).
Rayner, N. A. et al. Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century. J. Geophys. Res. 108, 4407 (2003).
Meier, W. N., Stroeve, J. & Fetterer, F. Whither Arctic sea ice? A clear signal of decline regionally, seasonally and extending beyond the satellite record. Ann. Glaciol. 46, 428–434 (2007).
Stroeve, J., Barrett, A., Serreze, M. & Schweiger, A. Using records from submarine, aircraft and satellites to evaluate climate model simulations of Arctic sea ice thickness. Cryosphere 8, 1839–1854 (2014).
Babu, G. J. & Rao, C. R. Goodness-of-fit tests when parameters are estimated. Sankhyā: Indian J. Stat. 66, 63–74 (2004).
Fletcher, R. Practical Methods of Optimization (John Wiley, 1987).
Shanno, D. F. Conditioning of quasi-Newton methods for function minimization. Math. Comput. 24, 647–656 (1970).
Acknowledgements
The authors acknowledge the support of the NOAA Climate Program Office, Modeling, Analysis, Predictions, and Projections (MAPP) Program as part of the CMIP5 Task Force. J.C.S. was supported by NOAA grant NA10OAR4320142. E.M.-M. was supported by an ARC DECRA Fellowship and by the ARC Centre for Excellence in Environmental Decisions.
Author information
Authors and Affiliations
Contributions
M.C.R. and E.M.-M. conceived of the methods; J.C.S. and A.P.B. extracted the sea-ice forecasts from the CMIP5 models; M.C.R. analysed the pintail and sea-ice data and prepared the figures; and M.C.R., E.M.-M. and J.C.S. co-wrote the paper.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Information
Supplementary Information (PDF 954 kb)
Rights and permissions
About this article
Cite this article
Runge, M., Stroeve, J., Barrett, A. et al. Detecting failure of climate predictions. Nature Clim Change 6, 861–864 (2016). https://doi.org/10.1038/nclimate3041
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nclimate3041
This article is cited by
-
A frequent ice-free Arctic is likely to occur before the mid-21st century
npj Climate and Atmospheric Science (2023)
-
Double-Loop Learning in Adaptive Management: The Need, the Challenge, and the Opportunity
Environmental Management (2018)
-
New vigour involving statisticians to overcome ensemble fatigue
Nature Climate Change (2017)
-
The effects of time-varying observation errors on semi-empirical sea-level projections
Climatic Change (2017)