The practical consequences of climate change challenge society to formulate responses that are more suited to achieving long-term objectives, even if those responses have to be made in the face of uncertainty1,2. Such a decision-analytic focus uses the products of climate science as probabilistic predictions about the effects of management policies3. Here we present methods to detect when climate predictions are failing to capture the system dynamics. For a single model, we measure goodness of fit based on the empirical distribution function, and define failure when the distribution of observed values significantly diverges from the modelled distribution. For a set of models, the same statistic can be used to provide relative weights for the individual models, and we define failure when there is no linear weighting of the ensemble models that produces a satisfactory match to the observations. Early detection of failure of a set of predictions is important for improving model predictions and the decisions based on them. We show that these methods would have detected a range shift in northern pintail 20 years before it was actually discovered, and are increasingly giving more weight to those climate models that forecast a September ice-free Arctic by 2055.
This is a preview of subscription content
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
McDonald-Madden, E., Runge, M. C., Martin, T. G. & Possingham, H. Optimal timing for managed relocation of species faced with climate change. Nature Clim. Change 1, 261–265 (2011).
Conroy, M. J., Runge, M. C., Nichols, J. D., Stodola, K. W. & Cooper, R. J. Conservation in the face of climate change: the roles of alternative models, monitoring, and adaptation in confronting and reducing uncertainty. Biol. Conserv. 144, 1204–1213 (2011).
Terando, A., Keller, K. & Easterling, W. E. Probabilistic projections of agro-climate indices in North America. J. Geophys. Res. 117, D08115 (2012).
Lawler, J. J. et al. Resource management in a changing and uncertain climate. Front. Ecol. Environ. 8, 35–43 (2010).
IPCC Climate Change 2013: The Physical Science Basis (Cambridge Univ. Press, 2013).
Taleb, N. N. The Black Swan: The Impact of the Highly Improbable (Random House, 2007).
Pahl-Wostl, C. A conceptual framework for analysing adaptive capacity and multi-level learning processes in resource governance regimes. Glob. Environ. Change 19, 354–365 (2009).
Argyris, C. & Schön, D. A. Organizational Learning: A Theory of Action Perspective (Addison-Wesley, 1978).
Stroeve, J., Holland, M. M., Meier, W., Scambos, T. & Serreze, M. Arctic sea ice decline: faster than forecast. Geophys. Res. Lett. 34, L09501 (2007).
Miller, M. R. & Duncan, D. C. The northern pintail in North America: status and conservation needs of a struggling population. Wildl. Soc. Bull. 27, 788–800 (1999).
Dawid, A. P. Statistical theory: the prequential approach. J. R. Stat. Soc. A 147, 278–292 (1984).
Williams, B. K., Johnson, F. A. & Wilkins, K. Uncertainty and the adaptive management of waterfowl harvests. J. Wildl. Manage. 60, 223–232 (1996).
Gneiting, T., Balabdaoui, F. & Raftery, A. E. Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. B 69, 243–268 (2007).
Seillier-Moiseiwitsch, F. & Dawid, A. P. On testing the validity of sequential probability forecasts. J. Am. Statist. Assoc. 88, 355–359 (1993).
Stephens, M. A. in Goodness-of-fit Techniques Vol. 68 (eds D’Agostino, R. B. & Stephens, M. A.) Ch. 4, 97–193 (Marcel Dukker, 1986).
Austin, J. E. & Miller, M. R. in The Birds of North America (ed. Poole, A.) (Cornell Laboratory of Ornithology, 1995).
Hestbeck, J. B. Response of northern pintail breeding populations to drought, 1961–92. J. Wildl. Manage. 59, 9–15 (1995).
US Fish and Wildlife Service Adaptive Harvest Management: 2014 Hunting Season (United States Department of Interior, 2014).
Stroeve, J. et al. The Arctic’s rapidly shrinking sea ice cover: a research synthesis. Climatic Change 110, 1005–1027 (2012).
Amstrup, S. C. et al. Greenhouse gas mitigation can reduce sea-ice loss and increase polar bear persistence. Nature 468, 955–958 (2010).
Hunter, C. M. et al. Climate change threatens polar bear populations: a stochastic demographic analysis. Ecology 91, 2883–2897 (2010).
Stroeve, J. C. et al. Trends in Arctic sea ice extent from CMIP5, CMIP3 and observations. Geophys. Res. Lett. 39, L16502 (2012).
Cavalieri, D. J., Parkinson, C. L., Gloersen, P. & Zwally, H. J. Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I-SSMIS Passive Microwave Data (NASA DAAC at the National Snow and Ice Data Center, 1996).
Rayner, N. A. et al. Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century. J. Geophys. Res. 108, 4407 (2003).
Meier, W. N., Stroeve, J. & Fetterer, F. Whither Arctic sea ice? A clear signal of decline regionally, seasonally and extending beyond the satellite record. Ann. Glaciol. 46, 428–434 (2007).
Stroeve, J., Barrett, A., Serreze, M. & Schweiger, A. Using records from submarine, aircraft and satellites to evaluate climate model simulations of Arctic sea ice thickness. Cryosphere 8, 1839–1854 (2014).
Babu, G. J. & Rao, C. R. Goodness-of-fit tests when parameters are estimated. Sankhyā: Indian J. Stat. 66, 63–74 (2004).
Fletcher, R. Practical Methods of Optimization (John Wiley, 1987).
Shanno, D. F. Conditioning of quasi-Newton methods for function minimization. Math. Comput. 24, 647–656 (1970).
The authors acknowledge the support of the NOAA Climate Program Office, Modeling, Analysis, Predictions, and Projections (MAPP) Program as part of the CMIP5 Task Force. J.C.S. was supported by NOAA grant NA10OAR4320142. E.M.-M. was supported by an ARC DECRA Fellowship and by the ARC Centre for Excellence in Environmental Decisions.
The authors declare no competing financial interests.
About this article
Cite this article
Runge, M., Stroeve, J., Barrett, A. et al. Detecting failure of climate predictions. Nature Clim Change 6, 861–864 (2016). https://doi.org/10.1038/nclimate3041
Environmental Management (2018)
Nature Climate Change (2017)
Climatic Change (2017)