Perspective | Published:

Taking climate model evaluation to the next level

Abstract

Earth system models are complex and represent a large number of processes, resulting in a persistent spread across climate projections for a given future scenario. Owing to different model performances against observations and the lack of independence among models, there is now evidence that giving equal weight to each available model projection is suboptimal. This Perspective discusses newly developed tools that facilitate a more rapid and comprehensive evaluation of model simulations with observations, process-based emergent constraints that are a promising way to focus evaluation on the observations most relevant to climate projections, and advanced methods for model weighting. These approaches are needed to distil the most credible information on regional climate changes, impacts, and risks for stakeholders and policy-makers.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    IPCC Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) (Cambridge Univ. Press, 2013).

  2. 2.

    Meehl, G. A., Boer, G. J., Covey, C., Latif, M. & Stouffer, R. J. The Coupled Model Intercomparison Project (CMIP). Bull. Am. Meteorol. Soc. 81, 313–318 (2000).

  3. 3.

    Meehl, G. A. et al. THE WCRP CMIP3 multimodel dataset: a new era in climate change research. Bull. Am. Meteorol. Soc. 88, 1383–1394 (2007).

  4. 4.

    Taylor, K. E., Stouffer, R. J. & Meehl, G. A. An overview of CMIP5 and the experiment design. Bull. Am. Meteorol. Soc. 93, 485–498 (2012).

  5. 5.

    Eyring, V. et al. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. 9, 1937–1958 (2016).

  6. 6.

    Stouffer, R. J. et al. CMIP5 scientific gaps and recommendations for CMIP6. Bull. Am. Meteorol. Soc. 98, 95–105 (2017).

  7. 7.

    Flato, G. et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) Ch. 9 (IPCC, Cambridge Univ. Press, 2013).

  8. 8.

    Maraun, D. et al. Towards process-informed bias correction of climate change simulations. Nat. Clim. Change 7, 764–773 (2017).

  9. 9.

    Collins, M. et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) Ch. 12 (IPCC, Cambridge Univ. Press, 2013).

  10. 10.

    Knutti, R. et al. A climate model projection weighting scheme accounting for performance and interdependence. Geophys. Res. Lett. 44, 1909–1918 (2017).

  11. 11.

    Wenzel, S., Eyring, V., Gerber, E. P. & Karpechko, A. Y. Constraining future summer austral jet stream positions in the CMIP5 ensemble by process-oriented multiple diagnostic regression. J. Clim. 29, 673–687 (2016).

  12. 12.

    Sanderson, B. M., Wehner, M. & Knutti, R. Skill and independence weighting for multi-model assessments. Geosci. Model Dev. 10, 2379–2395 (2017).

  13. 13.

    Sanderson, B. M., Knutti, R. & Caldwell, P. Addressing interdependency in a multimodel ensemble by interpolation of model properties. J. Clim. 28, 5150–5170 (2015).

  14. 14.

    Bishop, C. H. & Abramowitz, G. Climate model dependence and the replicate Earth paradigm. Clim. Dynam. 41, 885–900 (2013).

  15. 15.

    Abramowitz, G. & Bishop, C. H. Climate model dependence and the ensemble dependence transformation of CMIP projections. J. Clim. 28, 2332–2348 (2015).

  16. 16.

    Alexander, K. & Easterbrook, S. M. The software architecture of climate models: a graphical comparison of CMIP5 and EMICAR5 configurations. Geosci. Model Dev. 8, 1221–1232 (2015).

  17. 17.

    Knutti, R. The end of model democracy? Climatic Change 102, 395–404 (2010).

  18. 18.

    Eyring, V. et al. ESMValTool (v1.0) — a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP. Geosci. Model Dev. 9, 1747–1802 (2016).

  19. 19.

    Gleckler, P. et al. A more powerful reality test for climate models. Eos https://doi.org/10.1029/2016eo051663 (2016).

  20. 20.

    Lauer, A. et al. Process-level improvements in CMIP5 models and their impact on tropical variability, the Southern Ocean, and monsoons. Earth Syst. Dynam. 9, 33–67 (2018).

  21. 21.

    Ma, C.-C., Mechoso, C. R., Robertson, A. W. & Arakawa, A. Peruvian stratus clouds and the tropical pacific circulation: a coupled ocean-atmosphere GCM study. J. Clim. 9, 1635–1645 (1996).

  22. 22.

    Hourdin, F. et al. Parameterization of convective transport in the boundary layer and its impact on the representation of the diurnal cycle of wind and dust emissions. Atmos. Chem. Phys. 15, 6775–6788 (2015).

  23. 23.

    Richter, I. Climate model biases in the eastern tropical oceans: causes, impacts and ways forward. WIREs Clim. Change 6, 345–358 (2015).

  24. 24.

    Wang, C. Z., Zhang, L. P., Lee, S. K., Wu, L. X. & Mechoso, C. R. A global perspective on CMIP5 climate model biases. Nat. Clim. Change 4, 201–205 (2014).

  25. 25.

    Fyfe, J. C., Gillett, N. P. & Zwiers, F. W. Overestimated global warming over the past 20 years. Nat. Clim. Change 3, 767–769 (2013).

  26. 26.

    Meehl, G. A., Hu, A., Arblaster, J. M., Fasullo, J. & Trenberth, K. E. Externally forced and internally generated decadal climate variability associated with the Interdecadal Pacific Oscillation. J. Clim. 26, 7298–7310 (2013).

  27. 27.

    Fyfe, J. C. et al. Making sense of the early-2000s warming slowdown. Nat. Clim. Change 6, 224–228 (2016).

  28. 28.

    Santer, B. D. et al. Volcanic contribution to decadal changes in tropospheric temperature. Nat. Geosci. 7, 185–189 (2014).

  29. 29.

    Meehl, G. A., Teng, H. & Arblaster, J. M. Climate model simulations of the observed early-2000s hiatus of global warming. Nat. Clim. Change 4, 898–902 (2014).

  30. 30.

    Thoma, M., Greatbatch, R. J., Kadow, C. & Gerdes, R. Decadal hindcasts initialized using observed surface wind stress: evaluation and prediction out to 2024. Geophys. Res. Lett. 42, 6454–6461 (2015).

  31. 31.

    Meehl, G. A., Hu, A. & Teng, H. Initialized decadal prediction for transition to positive phase of the Interdecadal Pacific Oscillation. Nat. Commun. 7, 11718 (2016).

  32. 32.

    Mears, C. A., Santer, B. D., Wentz, F. J., Taylor, K. E. & Wehner, M. F. Relationship between temperature and precipitable water changes over tropical oceans. Geophys. Res. Lett. 34, L24709 (2007).

  33. 33.

    Karl, T. R. et al. Possible artifacts of data biases in the recent global surface warming hiatus. Science 348, 1469–1472 (2015).

  34. 34.

    Mears, C. A. & Wentz, F. J. The effect of diurnal correction on satellite-derived lower tropospheric temperature. Science 309, 1548–1551 (2005).

  35. 35.

    Mauritsen, T. et al. Tuning the climate of a global model. J. Adv. Model. Earth Syst. 4, M00A01 (2012).

  36. 36.

    Hourdin, F. et al. The art and science of climate model tuning. Bull. Am. Meteorol. Soc. 98, 589–602 (2017).

  37. 37.

    Bodas-Salcedo, A. et al. COSP Satellite simulation software for model assessment. Bull. Am. Meteorol. Soc. 92, 1023–1043 (2011).

  38. 38.

    Eyring, V. et al. Towards improved and more routine Earth system model evaluation in CMIP. Earth Syst. Dynam. 7, 813–830 (2016).

  39. 39.

    Phillips, A. S., Deser, C. & Fasullo, J. Evaluating modes of variability in climate models. Eos 95, 453–455 (2014).

  40. 40.

    Luo, Y. Q. et al. A framework for benchmarking land models. Biogeosciences 9, 3857–3874 (2012).

  41. 41.

    Prabhat. et al. TECA: a parallel toolkit for extreme climate analysis. Proc. Comput. Sci. 9, 866–876 (2012).

  42. 42.

    Ferraro, R., Waliser, D. E., Gleckler, P., Taylor, K. E. & Eyring, V. Evolving Obs4MIPs to support phase 6 of the Coupled Model Intercomparison Project (CMIP6). Bull. Am. Meteorol. Soc. 96, ES131–ES133 (2015).

  43. 43.

    Hall, A. & Qu, X. Using the current seasonal cycle to constrain snow albedo feedback in future climate change. Geophys. Res. Lett. 33, L03502 (2006).

  44. 44.

    Allen, M. R. & Ingram, W. J. Constraints on future changes in climate and the hydrologic cycle. Nature 419, 224–232 (2002).

  45. 45.

    Massonnet, F. et al. Constraining projections of summer Arctic sea ice. Cryosphere 6, 1383–1394 (2012).

  46. 46.

    O’Gorman, P. A. Sensitivity of tropical precipitation extremes to climate change. Nat. Geosci. 5, 697–700 (2012).

  47. 47.

    Cox, P. M. et al. Sensitivity of tropical carbon to climate change constrained by carbon dioxide variability. Nature 494, 341–344 (2013).

  48. 48.

    Kidston, J. & Gerber, E. P. Intermodel variability of the poleward shift of the austral jet stream in the CMIP3 integrations linked to biases in 20th century climatology. Geophys. Res. Lett. 37, L09708 (2010).

  49. 49.

    Tsushima, Y. et al. Robustness, uncertainties, and emergent constraints in the radiative responses of stratocumulus cloud regimes to future warming. Clim. Dynam. 46, 3025–3039 (2016).

  50. 50.

    Brient, F. & Bony, S. Interpretation of the positive low-cloud feedback predicted by a climate model under global warming. Clim. Dynam. 40, 2415–2431 (2013).

  51. 51.

    Brient, F. & Schneider, T. Constraints on climate sensitivity from space-based measurements of low-cloud reflection. J. Clim. 29, 5821–5835 (2016).

  52. 52.

    Lipat, B. R., Tselioudis, G., Grise, K. M. & Polvani, L. M. CMIP5 models’ shortwave cloud radiative response and climate sensitivity linked to the climatological Hadley cell extent. Geophys. Res. Lett. 44, 5739–5748 (2017).

  53. 53.

    Sherwood, S. C., Bony, S. & Dufresne, J. L. Spread in model climate sensitivity traced to atmospheric convective mixing. Nature 505, 37–42 (2014).

  54. 54.

    Tian, B. Spread of model climate sensitivity linked to double-Intertropical Convergence Zone bias. Geophys. Res. Lett. 42, 4133–4141 (2015).

  55. 55.

    Cox, P. M., Huntingford, C. & Williamson, M. S. Emergent constraint on equilibrium climate sensitivity from global temperature variability. Nature 553, 319–322 (2018).

  56. 56.

    Klein, S. A. & Hall, A. Emergent constraints for cloud feedbacks. Curr. Clim. Change Rep. 1, 276–287 (2015).

  57. 57.

    Dessler, A. E. & Forster, P. M. An estimate of equilibrium climate sensitivity from interannual variability. J. Geophys. Res. Atmos. 123, 8634–8645 (2018).

  58. 58.

    DeAngelis, A. M., Qu, X., Zelinka, M. D. & Hall, A. An observational radiative constraint on hydrologic cycle intensification. Nature 528, 249–253 (2015).

  59. 59.

    Li, G., Xie, S.-P., He, C. & Chen, Z. Western Pacific emergent constraint lowers projected increase in Indian summer monsoon rainfall. Nat. Clim. Change 7, 708–712 (2017).

  60. 60.

    Wenzel, S., Cox, P. M., Eyring, V. & Friedlingstein, P. Emergent constraints on climate-carbon cycle feedbacks in the CMIP5 Earth system models. J. Geophys. Res. Biogeosci. 119, 794–807 (2014).

  61. 61.

    Wenzel, S., Cox, P. M., Eyring, V. & Friedlingstein, P. Projected land photosynthesis constrained by changes in the seasonal cycle of atmospheric CO2. Nature 538, 499–501 (2016).

  62. 62.

    Kwiatkowski, L. et al. Emergent constraints on projections of declining primary production in the tropical oceans. Nat. Clim. Change 7, 355–358 (2017).

  63. 63.

    Chadburn, S. E. et al. An observation-based constraint on permafrost loss as a function of global warming. Nat. Clim. Change 7, 340–344 (2017).

  64. 64.

    Hoffman, F. M. et al. Causes and implications of persistent atmospheric carbon dioxide biases in Earth system models. J. Geophys. Res. Biogeosci. 119, 141–162 (2014).

  65. 65.

    Donat, M. G., Pitman, A. J. & Angélil, O. Understanding and reducing future uncertainty in mid-latitude daily heat extremes via land surface feedback constraints. Geophys. Res. Lett. 45, 10627–10636 (2018).

  66. 66.

    Qu, X. & Hall, A. What controls the strength of snow-albedo feedback? J. Clim. 20, 3971–3981 (2007).

  67. 67.

    Waugh, D. W. & Eyring, V. Quantitative performance metrics for stratospheric-resolving chemistry-climate models. Atmos. Chem. Phys. 8, 5699–5713 (2008).

  68. 68.

    Karpechko, A. Y., Maraun, D. & Eyring, V. Improving Antarctic total ozone projections by a process-oriented multiple diagnostic ensemble regression. J. Atmos. Sci. 70, 3959–3976 (2013).

  69. 69.

    Räisänen, J., Ruokolainen, L. & Ylhäisi, J. Weighting of model results for improving best estimates of climate change. Clim. Dynam. 35, 407–422 (2010).

  70. 70.

    Lorenz, R. et al. Prospects and caveats of weighting climate models for summer maximum temperature projections over North America. J. Geophys. Res. Atmos. 123, 4509–4526 (2018).

  71. 71.

    Abramowitz, G. et al. Model dependence in multi-model climate ensembles: weighting, sub-selection and out-of-sample testing. Earth Syst. Dynam. Discuss. https://doi.org/10.5194/esd-2018-51 (2018).

  72. 72.

    Herger, N. et al. Selecting a climate model subset to optimise key ensemble properties. Earth Syst. Dynam. 9, 135–151 (2018).

  73. 73.

    Herger, N. et al. Calibrating climate model ensembles for assessing extremes in a changing climate. J. Geophys. Res. Atmos. 123, 5988–6004 (2018).

  74. 74.

    Santer, B. D. et al. Incorporating model quality information in climate change detection and attribution studies. Proc. Natl Acad. Sci. USA 106, 14778–14783 (2009).

  75. 75.

    Caldwell, P. M. et al. Statistical significance of climate sensitivity predictors obtained by data mining. Geophys. Res. Lett. 41, 1803–1808 (2014).

  76. 76.

    Masson, D. & Knutti, R. Climate model genealogy. Geophys. Res. Lett. 38, L08703 (2011).

  77. 77.

    Pennell, C. & Reichler, T. On the effective number of climate models. J. Clim. 24, 2358–2367 (2011).

  78. 78.

    Sunyer, M. A., Madsen, H., Rosbjerg, D. & Arnbjerg-Nielsen, K. A Bayesian approach for uncertainty quantification of extreme precipitation projections including climate model interdependency and nonstationary bias. J. Clim. 27, 7113–7132 (2014).

  79. 79.

    Sanderson, B. M., Knutti, R. & Caldwell, P. A representative democracy to reduce interdependency in a multimodel ensemble. J. Clim. 28, 5171–5194 (2015).

  80. 80.

    Zhang, X. et al. Indices for monitoring changes in extremes based on daily temperature and precipitation data. WIREs Clim. Change 2, 851–870 (2011).

  81. 81.

    Sillmann, J., Kharin, V. V., Zhang, X., Zwiers, F. W. & Bronaugh, D. Climate extremes indices in the CMIP5 multimodel ensemble. Part 1. Model evaluation in the present climate. J. Geophys. Res. Atmos. 118, 1716–1733 (2013).

  82. 82.

    Pendergrass, A. G. & Hartmann, D. L. The atmospheric energy constraint on global-mean precipitation change. J. Clim. 27, 757–768 (2014).

  83. 83.

    Zscheischler, J. et al. Future climate risk from compound events. Nat. Clim. Change 8, 469–477 (2018).

  84. 84.

    Gutowski, W. J. Jr et al. WCRP COordinated Regional Downscaling EXperiment (CORDEX): a diagnostic MIP for CMIP6. Geosci. Model Dev. 9, 4087–4095 (2016).

  85. 85.

    van Oldenborgh, G. J., Otto, F. E. L., Haustein, K. & Cullen, H. Climate change increases the probability of heavy rains like those of storm Desmond in the UK—an event attribution study in near-real time. Hydrol. Earth Syst. Sci. Discuss. 12, 13197–13216 (2015).

  86. 86.

    Ruane, A. C., Goldberg, R. & Chryssanthacopoulos, J. Climate forcing datasets for agricultural modeling: merged products for gap-filling and historical climate series estimation. Agric. For. Meteorol. 200, 233–248 (2015).

  87. 87.

    Elliott, J. et al. The Global Gridded Crop Model Intercomparison: data and modeling protocols for Phase 1 (v1.0). Geosci. Model Dev. 8, 261–277 (2015).

  88. 88.

    Ruane, A. C. et al. The Vulnerability, Impacts, Adaptation and Climate Services Advisory Board (VIACS AB v1.0) contribution to CMIP6. Geosci. Model Dev. 9, 3493–3515 (2016).

  89. 89.

    Warszawski, L. et al. The Inter-Sectoral Impact Model Intercomparison Project (ISI–MIP): project framework. Proc. Natl Acad. Sci. USA 111, 3228–3232 (2014).

  90. 90.

    Ruane, A. C. & McDermid, S. P. Selection of a representative subset of global climate models that captures the profile of regional changes for integrated climate impacts assessment. Earth Perspect. 4, 1 (2017).

  91. 91.

    Stevens, B. & Bony, S. What are climate models missing? Science 340, 1053–1054 (2013).

  92. 92.

    Webb, M. J. et al. The Cloud Feedback Model Intercomparison Project (CFMIP) contribution to CMIP6. Geosci. Model Dev. 10, 359–384 (2017).

  93. 93.

    Mears, C. A., Wentz, F. J., Thorne, P. & Bernie, D. Assessing uncertainty in estimates of atmospheric temperature changes from MSU and AMSU using a Monte-Carlo estimation technique. J. Geophys. Res. Atmos. 116, D08112 (2011).

  94. 94.

    Argo Argo Float Data and Metadata from Global Data Assembly Centre (Argo GDAC) (SEANOE, 2000); https://doi.org/10.17882/42182

  95. 95.

    Malavelle, F. F. et al. Strong constraints on aerosol–cloud interactions from volcanic eruptions. Nature 546, 485–491 (2017).

  96. 96.

    Reichstein, M. et al. Deep learning and process understanding for data-driven Earth system science. Nature (in the press).

  97. 97.

    Fountalis, I., Bracco, A. & Dovrolis, C. ENSO in CMIP5 simulations: network connectivity from the recent past to the twenty-third century. Clim. Dynam. 45, 511–538 (2015).

  98. 98.

    Barz, B., Rodner, E., Guanche Garcia, Y. & Denzler, J. Detecting regions of maximal divergence for spatio-temporal anomaly detection. IEEE Trans. Pattern Anal. Mach. Intell. https://doi.org/10.1109/TPAMI.2018.2823766 (2018).

  99. 99.

    Runge, J. et al. Identifying causal gateways and mediators in complex spatio-temporal systems. Nat. Commun. 6, 8502 (2015).

  100. 100.

    Kageyama, M. et al. The PMIP4 contribution to CMIP6 – Part 1: Overview and over-arching analysis plan. Geosci. Model Dev. 11, 1033–1057 (2018).

  101. 101.

    Haarsma, R. J. et al. High Resolution Model Intercomparison Project (HighResMIP v1.0) for CMIP6. Geosci. Model Dev. 9, 4185–4208 (2016).

  102. 102.

    IPCC Special Report on Global Warming of 1.5 °C (Cambridge Univ. Press, 2018).

Download references

Acknowledgements

The authors acknowledge the Aspen Global Change Institute (AGCI) for hosting a workshop on Earth System Model Evaluation to Improve Process Understanding in August 2017 as part of its traditionally landmark summer interdisciplinary sessions (http://www.agci.org/event/17s2). NASA, the Heising-Simons Foundation, Horizon 2020 European Union’s Framework Programme for Research and Innovation under Grant Agreement No 641816, the Coordinated Research in Earth Systems and Climate: Experiments, kNowledge, Dissemination and Outreach (CRESCENDO) project, the ESA Climate Change Initiative (CCI) Climate Model User Group (CMUG), WCRP and the Department of Energy (DOE) all provided support for the workshop. The viewpoint presented here substantially draws on conclusions from that workshop. Portions of this study were supported by the Regional and Global Model Analysis (RGMA) component of the Earth and Environmental System Modeling Program of the US DOE Office of Biological & Environmental Research (BER) Cooperative Agreement DE-FC02-97ER62402 and Contract No. DE-AC05-00OR22725 and the National Science Foundation. NCAR is sponsored by the National Science Foundation.

Author information

V.E., P.M.C., G.M.F. and P.J.G. were the co-chairs of the AGCI workshop and led the writing of the paper. All authors participated in the AGCI workshop and contributed to discussions and writing of the text.

Competing interests

The authors declare no competing interests.

Correspondence to Veronika Eyring.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark
Fig. 1: Annual mean SST error from the CMIP5 multi-model ensemble.
Fig. 2: Schematic diagram of the workflow for CMIP Evaluation Tools running alongside the ESGF.
Fig. 3: Examples of newly developed physical and biogeochemical emergent constraints since the AR5.
Fig. 4: Model skill and independence weights for CMIP5 models evaluated over the contiguous United States/Canada domain.