Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Evaluation of CMIP5 palaeo-simulations to improve climate projections


Structural differences among models account for much of the uncertainty in projected climate changes, at least until the mid-twenty-first century. Recent observations encompass too limited a range of climate variability to provide a robust test of the ability to simulate climate changes. Past climate changes provide a unique opportunity for out-of-sample evaluation of model performance. Palaeo-evaluation has shown that the large-scale changes seen in twenty-first-century projections, including enhanced land–sea temperature contrast, latitudinal amplification, changes in temperature seasonality and scaling of precipitation with temperature, are likely to be realistic. Although models generally simulate changes in large-scale circulation sufficiently well to shift regional climates in the right direction, they often do not predict the correct magnitude of these changes. Differences in performance are only weakly related to modern-day biases or climate sensitivity, and more sophisticated models are not better at simulating climate changes. Although models correctly capture the broad patterns of climate change, improvements are required to produce reliable regional projections.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Figure 1: Scatter plots showing temperature and precipitation changes in past, present and projected climates.
Figure 2: Comparison of observed and simulated regional climate.
Figure 3: Taylor diagram90 for the LGM and mid-Holocene precipitation and temperature anomalies.
Figure 4: Maps of the p-values of Hotelling's T2 test91 comparing the CMIP3 plus PMIP2 versus CMIP5 ensembles.


  1. 1

    Taylor, K. E., Stouffer, R. J. & Meehl, G. A. An overview of CMIP5 and the experiment design. Bull. Am. Meteorol. Soc. 93, 485–498 (2012).

    Article  Google Scholar 

  2. 2

    Flato, G. et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) 741–866 (IPCC, Cambridge Univ. Press, 2013).

    Google Scholar 

  3. 3

    Kirtman, B. et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) 953–1028 (IPCC, Cambridge Univ. Press, 2013).

    Google Scholar 

  4. 4

    Collins, M. et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) 1029–1136 (IPCC, Cambridge Univ. Press, 2013).

    Google Scholar 

  5. 5

    Hawkins, E. & Sutton, R. The potential to narrow uncertainty in regional climate predictions. Bull. Am. Meteorol. Soc. 90, 1095–1107 (2009).

    Article  Google Scholar 

  6. 6

    Hawkins, E. & Sutton, R. The potential to narrow uncertainty in projections of regional precipitation change. Clim. Dynam. 37, 407–418 (2011).

    Article  Google Scholar 

  7. 7

    Braconnot, P. et al. Evaluation of climate models using palaeoclimatic data. Nature Clim. Change 2, 417–424 (2012).

    Article  Google Scholar 

  8. 8

    MARGO Project Members Constraints on the magnitude and patterns of ocean cooling at the Last Glacial Maximum. Nature Geosci. 2, 127–132 (2009).

  9. 9

    Bartlein, P. J. et al. Pollen-based continental climate reconstructions at 6 and 21 ka: A global synthesis. Clim. Dynam. 37, 775–802 (2011).

    Article  Google Scholar 

  10. 10

    Joussaume, S. et al. Monsoon changes for 6000 years ago: Results of 18 simulations from the Paleoclimate Modelling Intercomparison Project (PMIP). Geophys. Res. Lett. 26, 859–862 (1999).

    Article  Google Scholar 

  11. 11

    Pinot, S. et al. Tropical palaeoclimates at the Last Glacial Maximum: Comparison of Paleoclimate Modelling Intercomparison Project (PMIP) simulations and paleodata. Clim. Dynam. 15, 857–874 (1999).

    Article  Google Scholar 

  12. 12

    Braconnot, P. et al. Results of PMIP2 coupled simulations of the mid-Holocene and Last Glacial Maximum—Part 1: Experiments and large-scale features. Clim. Past 3, 261–277 (2007).

  13. 13

    Harrison, S. P. et al. Model benchmarking with glacial and mid-Holocene climates. Clim. Dynam. 43, 671–688 (2014).

    Article  Google Scholar 

  14. 14

    Joshi, M. M., Lambert, F. H. & Webb, M. J. An explanation for the difference between twentieth and twenty-first century land–sea warming ratio in climate models. Clim. Dynam. 41, 1853–1869 (2013).

    Article  Google Scholar 

  15. 15

    Pithan, F. & Mauritsen, T. Arctic amplification dominated by temperature feedbacks in contemporary climate models. Nature Geosci. 7, 181–184 (2014).

    Article  CAS  Google Scholar 

  16. 16

    Izumi, K., Bartlein, P. J. & Harrison, S. P. Consistent behaviour of the climate system in response to past and future forcing. Geophys. Res. Lett. 40, 1–7 (2013).

    Article  Google Scholar 

  17. 17

    Hargreaves, J. C. & Annan, J. D. Can we trust climate models? WIREs Clim. Change 5, 435–440 (2014).

    Article  Google Scholar 

  18. 18

    Schmidt, G. A. et al. Using paleo-climate comparisons to constrain future projections in CMIP5. Clim. Past 10, 221–250 (2014).

    Article  Google Scholar 

  19. 19

    Lunt, D. J. et al. A multi-model assessment of last interglacial temperatures. Clim. Past 9, 699–717 (2013).

    Article  Google Scholar 

  20. 20

    Hill, D. J. et al. Evaluating the dominant components of warming in Pliocene climate simulations. Clim. Past 10, 79–90 (2014).

    Article  Google Scholar 

  21. 21

    Salzmann, U. et al. Challenges in quantifying Pliocene terrestrial warming revealed by data–model discord. Nature Clim. Change 3, 969–974 (2013).

    Article  CAS  Google Scholar 

  22. 22

    Miller, G. H., Lehman, S. J., Refsnider, K. A., Southon, J. R. & Zhong, Y. Unprecedented recent summer warmth in Arctic Canada. Geophys. Res. Lett. 40, 5745–5751 (2013).

    Article  Google Scholar 

  23. 23

    Izumi, K., Bartlein, P. J. & Harrison, S. P. Energy-balance mechanisms underlying consistent large-scale temperature responses in warm and cold climates. Clim. Dynam. 44, 3111–3127 (2015).

    Article  Google Scholar 

  24. 24

    Allan, R. P. Examination of relationships between clear-sky longwave radiation and aspects of the atmospheric hydrological cycle in climate models, reanalysis, and observations. J. Climate 22, 3127–3145 (2009).

    Article  Google Scholar 

  25. 25

    Trenberth, K. E. & Shea, D. J. Relationships between precipitation and surface temperature. Geophys. Res. Lett. 32, L14703 (2005).

    Article  Google Scholar 

  26. 26

    Lau, W. K. M., Wu, H-T. & Kim, K-M. A canonical response of precipitation characteristics to global warming from CMIP5 models. Geophys. Res. Lett. 40, 3163–3169 (2013).

    Article  Google Scholar 

  27. 27

    Li, G., Harrison, S. P., Bartlein, P. J., Izumi, K. & Prentice, I. C. Precipitation scaling with temperature in warm and cold climates: An analysis of CMIP5 simulations. Geophs. Res. Lett. 40, 4018–4024 (2013).

    Article  Google Scholar 

  28. 28

    Kutzbach, J. E. & Street-Perrott, F. A. Milankovitch forcing of fluctuations in the level of tropical lakes from 18 to 0 kyr BP. Nature 317, 130–134 (1985).

    Article  Google Scholar 

  29. 29

    Wohlfahrt, J., Harrison, S. P. & Braconnot, P. Synergistic feedbacks between ocean and vegetation on mid- and high-latitude climates during the mid-Holocene. Clim. Dynam. 22, 223–238 (2004).

    Article  Google Scholar 

  30. 30

    Marzin, C. & Braconnot, P. The role of the ocean feedback on Asian and African monsoon variations at 6 kyr and 9.5 kyr BP. C. R. Geosci. 341, 643–655 (2009).

    Article  Google Scholar 

  31. 31

    Wang, T., Wang, H. J. & Jiang, D. B. Mid-Holocene East Asian summer climate as simulated by the PMIP2 models. Palaeogeogr. Palaeoclimatol. Palaeoecol. 288, 93–102 (2010).

    Article  Google Scholar 

  32. 32

    Zhao, Y. & Harrison, S. P. Mid-Holocene monsoons: A multi-model analysis of the inter-hemispheric differences in the responses to orbital forcing and ocean feedbacks. Clim. Dynam. 39, 1457–1487 (2012).

    Article  CAS  Google Scholar 

  33. 33

    Jiang, D., Lang, X., Tian, Z. & Ju, L. Mid-Holocene East Asian summer monsoon strengthening: Insights from Paleoclimate Modeling Intercomparison Project (PMIP) simulations. Palaeogeogr. Palaeoclimatol. Palaeoecol. 369, 422–429 (2013).

    Article  Google Scholar 

  34. 34

    Jiang, D., Tian, Z. & Land, X. Mid-Holocene net precipitation changes over China: Model–data comparison. Quat. Sci. Rev. 82, 104–120 (2013).

    Article  Google Scholar 

  35. 35

    Perez-Sanz, A., Li, G., Gonzalez, P. & Harrison, S. P. Evaluation of seasonal climates of northern Africa and the Mediterranean in the CMIP5 simulations. Clim. Past 10, 551–568 (2014).

    Article  Google Scholar 

  36. 36

    Prado, L. F., Wainer, I. & Chiessi, C. M. Mid-Holocene PMIP3/CMIP5 model results: Intercomparison for the South American monsoon system. Holocene 21, 1915–1920 (2013).

    Article  Google Scholar 

  37. 37

    Kutzbach, J. E., Bonan, G. B., Foley, J. A. & Harrison, S. P. Vegetation and soils feedbacks on the response of the African monsoon to orbital forcing in the early to middle Holocene. Nature 384, 623–626 (1996).

    Article  CAS  Google Scholar 

  38. 38

    Claussen, M. & Gayler, V. The greening of the Sahara during the mid-Holocene: Results of an interactive atmosphere–biome model. Glob. Ecol. Biogeogr. Lett. 6, 369–377 (1997).

    Article  Google Scholar 

  39. 39

    Broström, A. et al. Land surface feedbacks and palaeomonsoons in northern Africa. Geophys. Res. Lett. 25, 3615–3618 (1998).

    Article  Google Scholar 

  40. 40

    Braconnot, P., Joussaume, S., Marti, O. & de Noblet, N. Synergistic feedbacks from ocean and vegetation on the African monsoon response to mid-Holocene insolation. Geophys. Res. Lett. 26, 2481–2484 (1999).

    Article  Google Scholar 

  41. 41

    Otto, J., Raddatz, T., Claussen, M., Brovkin, V. & Gayler, V. Separation of atmosphere–ocean–vegetation feedbacks and synergies for mid-Holocene climate. Glob. Biogeochem. Cycles 23, L09701 (2009).

    Google Scholar 

  42. 42

    Claussen, M., Bathiany, S., Brovkin, V. & Kleinen, T. Simulated climate–vegetation interaction in semi-arid regions affected by plant diversity. Nature Geosci. 6, 954–958 (2013).

    Article  CAS  Google Scholar 

  43. 43

    Levis, S., Bonan, G. B. & Bonfils, C. Soil feedback drives the mid-Holocene North African monsoon northward in fully coupled CCSM2 simulations with a dynamic vegetation model. Clim. Dynam. 23, 791–802 (2004).

    Article  Google Scholar 

  44. 44

    Wang, Y. et al. Detecting vegetation–precipitation feedbacks in mid-Holocene North Africa from two climate models. Clim. Past 4, 59–67 (2008).

    Article  CAS  Google Scholar 

  45. 45

    Tian, Z. & Jiang, D. Mid-Holocene ocean and vegetation feedbacks over East Asia. Clim. Past 9, 2153–2171 (2013).

    Article  Google Scholar 

  46. 46

    Prentice, I. C., Liang, X., Medlyn, B. & Wang, Y. Reliable, robust and realistic: The three R's of next-generation land-surface modelling. Atmos. Chem. Phys. Discuss. 14, 24811–24861 (2014).

    Article  Google Scholar 

  47. 47

    Roehrig, R., Bouniol, D., Guichard, F., Hourdin, F. & Redelsperger, J-L. The present and future of the West African monsoon: A process-oriented assessment of CMIP5 simulations along the AMMA transect. J. Climate 26, 6471–6505 (2013).

    Article  Google Scholar 

  48. 48

    Zheng, W. & Braconnot, P. Characterization of model spread in PMIP2 mid-Holocene simulations of the African monsoon. J. Climate 26, 1192–1210 (2013).

    Article  Google Scholar 

  49. 49

    Yu, G. & Harrison, S. P. An evaluation of the simulated water balance of Eurasia and northern Africa at 6000 yr BP using lake status data. Clim. Dynam. 12, 723–735 (1996).

    Article  Google Scholar 

  50. 50

    Wohlfahrt, J. et al. Evaluation of coupled ocean–atmosphere simulations of Northern Hemisphere extratropical climates in the mid-Holocene. Clim. Dynam. 31, 871–890 (2008).

    Article  Google Scholar 

  51. 51

    Mueller, B. & Seneviratne, S. I. Systematic land climate and evapotranspiration biases in CMIP5 simulations. Geophys. Res. Lett. 41, 128–134 (2014).

    Article  CAS  Google Scholar 

  52. 52

    Mauri, A., Davis, B. A. S., Collins, P. M. & Kaplan, J. O. The influence of atmospheric circulation on the mid-Holocene climate of Europe: A data–model comparison. Clim. Past 10, 1925–1938 (2014).

    Article  Google Scholar 

  53. 53

    Masson, V. et al. Mid-Holocene climate in Europe: What can we infer from PMIP model data comparisons? Clim. Dynam. 15, 163–182 (1999).

    Article  Google Scholar 

  54. 54

    Brewer, S., Guiot, J. & Torre, F. Mid-Holocene climate change in Europe: A data–model comparison. Clim. Past 3, 499–512 (2007).

    Article  Google Scholar 

  55. 55

    Davis, B. A. S. & Brewer, S. Orbital forcing and the role of the latitudinal temperature/insolation gradient. Clim. Dynam. 32, 143–165 (2009).

    Article  Google Scholar 

  56. 56

    van Oldenborgh, G. J. et al. Western Europe is warming much faster than expected. Clim. Past 5, 1–12 (2009).

    Article  Google Scholar 

  57. 57

    Berger, A., Brandefelt, J. & Nilsson, J. The sensitivity of the Arctic sea ice to orbitally induced insolation changes: A study of the mid-Holocene Paleoclimate Modelling Intercomparison Project 2 and 3 simulations. Clim. Past 9, 969–982 (2013).

    Article  Google Scholar 

  58. 58

    Goosse, H., Roche, D. M., Mairesse, A. & Berger, M. Modelling past sea ice changes. Quat. Sci. Rev. 79, 191–206 (2013).

    Article  Google Scholar 

  59. 59

    Chavaillaz, Y., Codron, F. & Kageyama, M. Southern westerlies in LGM and future (RCP4.5) climates. Clim. Past 9, 517–524 (2013).

    Article  Google Scholar 

  60. 60

    Rojas, M. Sensitivity of Southern Hemisphere circulation to LGM and 4 × CO2 climates. Geophys. Res. Lett. 40, 965–970 (2013).

    Article  CAS  Google Scholar 

  61. 61

    DiNezio, P. N. & Tierney, J. E. The effect of sea level on glacial Indo-Pacific climate. Nature Geosci. 6, 485–491 (2013).

    Article  CAS  Google Scholar 

  62. 62

    Hargreaves, J. C., Annan, J. D., Ohgaito, R., Paul, A. & Abe-Ouchi, A. Skill and reliability of climate model ensembles at the Last Glacial Maximum and mid-Holocene. Clim. Past 9, 811–823 (2013).

    Article  Google Scholar 

  63. 63

    Gleckler, P. J., Taylor, K. E. & Doutriaux, C, Performance metrics for climate models. J. Geophys. Res. 113, D06104 (2008).

    Article  Google Scholar 

  64. 64

    Annan, J. D. & Hargreaves, J. C. Understanding the CMIP3 multimodel ensemble. J. Climate 24, 4529–4538 (2011).

    Article  Google Scholar 

  65. 65

    Hessler, I. et al. Implication of methodological uncertainties for mid-Holocene sea surface temperature reconstructions. Clim. Past 10, 2237–2252 (2014).

    Article  Google Scholar 

  66. 66

    Kageyama, M. et al. Mid-Holocene and Last Glacial Maximum climate simulations with the IPSL model – Part I: Comparing IPSL CM5A to IPSL CM4. Clim. Dynam. 40, 2447–2468 (2013).

    Article  Google Scholar 

  67. 67

    Kageyama, M. et al. Mid-Holocene and Last Glacial Maximum climate simulations with the IPSL model: Part II: Model–data comparisons. Clim. Dynam. 40, 2469–2495 (2013).

    Article  Google Scholar 

  68. 68

    Knutti, R. The end of model democracy? Clim. Change 102, 395–404 (2010).

    Article  Google Scholar 

  69. 69

    Rauser, F., Gleckler, P. & Marotzke, J. Rethinking the default construction of multi-model climate ensembles. Bull. Am. Meteorol. Soc. (2014).

  70. 70

    PALAEOSENS project members Making sense of palaeoclimate sensitivity. Nature 491, 683–691 (2012).

  71. 71

    Annan, J. D., Hargreaves, J. C., Ohgaito, R., Abe-Ouchi, A. & Emori, S. Efficiently constraining climate sensitivity with ensembles of paleoclimate simulations. SOLA 1, 181–184 (2005).

    Article  Google Scholar 

  72. 72

    Crucifix, M. Does the Last Glacial Maximum constrain climate sensitivity? Geophys. Res. Lett. 33, L18701 (2006).

    Article  Google Scholar 

  73. 73

    Schneider von Deimling, T., Held, H., Ganolpolski, A. & Rahmstorf, S. Climate sensitivity estimated from ensemble simulations of glacial climate. Clim. Dynam. 27, 149–163 (2006).

    Article  Google Scholar 

  74. 74

    Hargreaves, J. C., Annan, J. D., Yoshimori, M. & Abe-Ouchi, A. Can the Last Glacial Maximum constrain climate sensitivity? Geophys. Res. Lett. 39, L24702 (2012).

    Article  Google Scholar 

  75. 75

    Tripati, A. K. et al. Modern and glacial tropical snowlines controlled by sea surface temperature and atmospheric mixing. Nature Geosci. 7, 205–209 (2014).

    Article  CAS  Google Scholar 

  76. 76

    Yoshimori, M., Hargreaves, J. C., Annan, J. D., Yokohata, T. & Abe-Ouchi, A. Dependency of feedbacks on forcing and climate state in physics parameter ensembles. J. Climate 24, 6440–6455 (2011).

    Article  Google Scholar 

  77. 77

    Otto-Bliesner, B. L. et al. Last Glacial Maximum and Holocene climate in CCSM3. J. Climate 19, 2526–2544 (2006).

    Article  Google Scholar 

  78. 78

    Schmittner, A. et al. Climate sensitivity estimated from temperature reconstructions of the Last Glacial Maximum. Science 334, 1385–1388 (2011).

    Article  CAS  Google Scholar 

  79. 79

    Claquin, T. et al. Radiative forcing effect of ice-age dust. Clim. Dynam. 20, 193–202 (2003).

    Article  Google Scholar 

  80. 80

    Hoelzmann, P. et al. Mid-Holocene land-surface conditions in northern Africa and the Arabian peninsula: A data set for the analysis of biogeophysical feedbacks in the climate system. Glob. Biogeochem. Cycles 12, 35–51 (1998).

    Article  CAS  Google Scholar 

  81. 81

    Lézine, A-M., Hély, C., Grenier, C., Braconnot, P. & Krinner, G. Sahara and Sahel vulnerability to climate changes, lessons from Holocene hydrological data. Quat. Sci. Rev. 30, 3001–3012 (2011).

    Article  Google Scholar 

  82. 82

    Elenga, H. et al. Pollen-based reconstruction for Southern Europe and Africa 18,000 years ago. J. Biogeogr. 27, 621–634 (2000).

    Article  Google Scholar 

  83. 83

    Watrin, J., Lézine, A-M. & Hély, C. Plant migration and ecosystems at the time of the “green Sahara”. C. R. Geosci. 341, 656–670 (2009).

    Article  Google Scholar 

  84. 84

    Kuper, R. & Kropelin, S. Climate-controlled Holocene occupation in the Sahara: Motor of Africa's evolution. Science 313, 803–807 (2006).

    Article  CAS  Google Scholar 

  85. 85

    Manning, K. & Timpson, A. The demographic response to Holocene climate change in the Sahara. Quat. Sci. Rev. 101, 28–35 (2014).

    Article  Google Scholar 

  86. 86

    Prentice, I. C., Harrison, S. P. & Bartlein, P. J. Global vegetation and terrestrial carbon cycle changes after the last ice age. New Phytol. 189, 988–998 (2011).

    Article  CAS  Google Scholar 

  87. 87

    Coe, M. T. & Harrison, S. P. The water balance of northern Africa during the mid-Holocene: An evaluation of the 6ka BP PMIP experiments. Clim. Dynam. 19, 155–166 (2002).

    Article  Google Scholar 

  88. 88

    Ward, P. J., Aerts, J. C. J. H., de Moel, H. & Renssen, H. Verification of a coupled climate–hydrological model against Holocene palaeohydrological records. Glob. Planet. Change 57, 283–300 (2007).

    Article  Google Scholar 

  89. 89

    Haywood, A. M. et al. Pliocene Model Intercomparison Project (PlioMIP): Experimental design and boundary conditions (Experiment 1). Geosci. Model Dev. 3, 227–242 (2010).

    Article  Google Scholar 

  90. 90

    Taylor, K. E. Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res. 106, 7183–7192 (2001).

    Article  Google Scholar 

  91. 91

    Wilks, D. S. Statistical Methods in the Atmospheric Sciences 3rd edn (International Geophysics Series 100, Academic, 2011).

    Google Scholar 

  92. 92

    Wilks, D. S. On “field significance” and the false discovery rate. J. Appl. Meteorol. Climatol. 45, 1181–1189 (2006).

    Article  Google Scholar 

  93. 93

    Joussaume, S. & Taylor, K. E. in Proc. 1st Int. AMIP Sci. Conf. 425–430 (WCRP Series Report 92, WMO, 1995).

    Google Scholar 

  94. 94

    Gates, W. L. AMIP: The Atmospheric Model Intercomparison Project. Bull. Am. Meteorol. Soc. 73, 1962–1970 (1992).

    Article  Google Scholar 

  95. 95

    Abe-Ouchi, A. & Harrison, S. P. Constraining the carbon-cycle feedback using palaeodata: The PalaeoCarbon Modelling Intercomparison Project. Eos 90, 140 (2009).

    Article  Google Scholar 

Download references


This paper is a contribution to the ongoing work on the PMIP. We thank all of the modelling groups who have contributed to the CMIP5 archive. We acknowledge financial support from the Centre for Past Climate Change, University of Reading. G.L. was supported by an international postgraduate research scholarship at Macquarie University. P.J.B. and K.I. were supported by the US National Science Foundation paleoclimatology programme. M.K., P.B. and K.I. acknowledge financial support from Labex L-IPSL.

Author information




S.P.H. planned the paper and was responsible for drafting the text; all authors were involved in analysis and interpretation of the data, and contributed to the final version.

Corresponding author

Correspondence to S. P. Harrison.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Table 1

Description of past, present and future simulations. (PDF 130 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Harrison, S., Bartlein, P., Izumi, K. et al. Evaluation of CMIP5 palaeo-simulations to improve climate projections. Nature Clim Change 5, 735–743 (2015).

Download citation

Further reading


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing