Machine learning approaches are increasingly used to extract patterns and insights from the ever-increasing stream of geospatial data, but current approaches may not be optimal when system behaviour is dominated by spatial or temporal context. Here, rather than amending classical machine learning, we argue that these contextual cues should be used as part of deep learning (an approach that is able to extract spatio-temporal features automatically) to gain further process understanding of Earth system science problems, improving the predictive ability of seasonal forecasting and modelling of long-range spatial connections across multiple timescales, for example. The next step will be a hybrid modelling approach, coupling physical process models with the versatility of data-driven machine learning.
Subscribe to Journal
Get full journal access for 1 year
only $3.90 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Howe, L. & Wain, A. Predicting the Future Vol. V, 1–195 (Cambridge Univ. Press, 1993).
Bauer, P., Thorpe, A. & Brunet, G. The quiet revolution of numerical weather prediction. Nature 525, 47–55 (2015).
Hantson, S. et al. The status and challenge of global fire modelling. Biogeosciences 13, 3359–3375 (2016).
Agapiou, A. Remote sensing heritage in a petabyte-scale: satellite data and heritage Earth Engine© applications. Int. J. Digit. Earth 10, 85–102 (2017).
Stockhause, M. & Lautenschlager, M. CMIP6 data citation of evolving data. Data Sci. J. 16, 30 (2017).
Lee, J., Weger, R. C., Sengupta, S. K. & Welch, R. M. A neural network approach to cloud classification. IEEE Trans. Geosci. Remote Sens. 28, 846–855 (1990).
Benediktsson, J. A., Swain, P. H. & Ersoy, O. K. Neural network approaches versus statistical methods in classification of multisource remote sensing data. IEEE Trans. Geosci. Remote Sens. 28, 540–552 (1990).
Camps-Valls, G. & Bruzzone, L. Kernel Methods for Remote Sensing Data Analysis 434 (John Wiley & Sons, Chichester, 2009).
Gómez-Chova, L., Tuia, D., Moser, G. & Camps-Valls, G. Multimodal classification of remote sensing images: a review and future directions. Proc. IEEE 103, 1560–1584 (2015).
Camps-Valls, G., Tuia, D., Bruzzone, L. & Benediktsson, J. A. Advances in hyperspectral image classification: Earth monitoring with statistical learning methods. IEEE Signal Process. Mag. 31, 45–54 (2014). This paper provides a comprehensive overview of machine learning for classification.
Gislason, P. O., Benediktsson, J. A. & Sveinsson, J. R. Random forests for land cover classification. Pattern Recogn. Lett. 27, 294–300 (2006). This paper is one of the first machine learning papers for land-cover classification, a method now operationally used.
Muhlbauer, A., McCoy, I. L. & Wood, R. Climatology of stratocumulus cloud morphologies: microphysical properties and radiative effects. Atmos. Chem. Phys. 14, 6695–6716 (2014).
Grimm, R., Behrens, T., Märker, M. & Elsenbeer, H. Soil organic carbon concentrations and stocks on Barro Colorado Island—digital soil mapping using Random Forests analysis. Geoderma 146, 102–113 (2008).
Hengl, T. et al. SoilGrids250m: global gridded soil information based on machine learning. PLoS ONE 12, e0169748 (2017). This paper describes machine learning used for operational global soil mapping.
Townsend, P. A., Foster, J. R., Chastain, R. A. & Currie, W. S. Application of imaging spectroscopy to mapping canopy nitrogen in the forests of the central Appalachian Mountains using Hyperion and AVIRIS. IEEE Trans. Geosci. Remote Sens. 41, 1347–1354 (2003).
Coops, N. C., Smith, M.-L., Martin, M. E. & Ollinger, S. V. Prediction of eucalypt foliage nitrogen content from satellite-derived hyperspectral data. IEEE Trans. Geosci. Remote Sens. 41, 1338–1346 (2003).
Verrelst, J., Alonso, L., Camps-Valls, G., Delegido, J. & Moreno, J. Retrieval of vegetation biophysical parameters using Gaussian process techniques. IEEE Trans. Geosci. Remote Sens. 50, 1832–1843 (2012).
Papale, D. & Valentini, R. A new assessment of European forests carbon exchanges by eddy fluxes and artificial neural network spatialization. Glob. Change Biol. 9, 525–535 (2003).
Jung, M. et al. Global patterns of land-atmosphere fluxes of carbon dioxide, latent heat, and sensible heat derived from eddy covariance, satellite and meteorological observations. J. Geophys. Res. Biogeo. 116, G00j07 (2011).
Tramontana, G. et al. Predicting carbon dioxide and energy fluxes across global FLUXNET sites with regression algorithms. Biogeosciences 13, 4291–4313 (2016).
Jung, M. et al. Recent decline in the global land evapotranspiration trend due to limited moisture supply. Nature 467, 951–954 (2010). This paper describes the first data-driven machine-learning-based spatio-temporal estimation of global water fluxes on land.
Jung, M. et al. Compensatory water effects link yearly global land CO2 sink changes to temperature. Nature 541, 516–520 (2017).
Bonan, G. B. et al. Improving canopy processes in the Community Land Model version 4 (CLM4) using global flux fields empirically inferred from FLUXNET data. J. Geophys. Res. Biogeosci. 116, G02014 (2011).
Anav, A. et al. Spatiotemporal patterns of terrestrial gross primary production: a review. Rev. Geophys. 53, 785–818 (2015).
Landschützer, P. et al. A neural network-based estimate of the seasonal to inter-annual variability of the Atlantic Ocean carbon sink. Biogeosciences 10, 7793–7815 (2013). This paper describes the first data-driven machine-learning-based carbon fluxes in the ocean.
Kühnlein, M., Appelhans, T., Thies, B. & Nauss, T. Improving the accuracy of rainfall rates from optical satellite sensors with machine learning—a random forests-based approach applied to MSG SEVIRI. Remote Sens. Environ. 141, 129–143 (2014).
Caldwell, P. M. et al. Statistical significance of climate sensitivity predictors obtained by data mining. Geophys. Res. Lett. 41, 1803–1808 (2014).
Reichstein, M. & Beer, C. Soil respiration across scales: the importance of a model-data integration framework for data interpretation. J. Plant Nutr. Soil Sci. 171, 344–354 (2008).
Wright, S. Correlation and causation. J. Agric. Res. 20, 557–585 (1921).
Guttman, N. B. Accepting the standardized precipitation index: a calculation algorithm. J. Am. Water Resour. Assoc. 35, 311–322 (1999).
Vicente-Serrano, S. M., Beguería, S. & López-Moreno, J. I. A multiscalar drought index sensitive to global warming: the standardized precipitation evapotranspiration index. J. Clim. 23, 1696–1718 (2010).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Lore, K. G., Stoecklein, D., Davies, M., Ganapathysubramanian, B. & Sarkar, S. Hierarchical feature extraction for efficient design of microfluidic flow patterns. Proc. Machine Learning Res. 44, 213–225 (2015).
Baldi, P., Sadowski, P. & Whiteson, D. Searching for exotic particles in high-energy physics with deep learning. Nat. Commun. 5, 4308 (2014).
Bhimji, W., Farrell, S. A., Kurth, T., Paganini, M. & Racah, E. Deep neural networks for physics analysis on low-level whole-detector data at the LHC. Preprint at https://arxiv.org/abs/1711.03573 (2017).
Schütt, K. T., Arbabzadah, F., Chmiela, S., Muller, K. R. & Tkatchenko, A. Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8, 13890 (2017).
Alipanahi, B., Delong, A., Weirauch, M. T. & Frey, B. J. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat. Biotechnol. 33, 831–838 (2015).
Prabhat. A look at deep learning for science. O’Reilly Blog https://www.oreilly.com/ideas/a-look-at-deep-learning-for-science (2017).
Zhang, L. P., Zhang, L. F. & Du, B. Deep learning for remote sensing data: a technical tutorial on the state of the art. IEEE Geosci. Remote Sens. Mag. 4, 22–40 (2016).
Ball, J. E., Anderson, D. T. & Chan, C. S. Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community. J. Appl. Remote Sens. 11, 042609 (2017).
Racah, E. et al. ExtremeWeather: a large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events. Adv. Neural Inform. Process. Syst. 30, 3405–3416 (2017).
Liu, Y. et al. Application of deep convolutional neural networks for detecting extreme weather in climate datasets. In ABDA'16-International Conference on Advances in Big Data Analytics 81–88 https://arxiv.org/abs/1605.01156 (2016). This paper is the first approach to detecting extreme weather automatically without any prescribed thresholds, using deep learning.
Zhao, W. Z. & Du, S. H. Learning multiscale and deep representations for classifying remotely sensed imagery. ISPRS J. Photogramm. Remote Sens. 113, 155–165 (2016).
Mathieu, M., Couprie, C. & LeCun, Y. Deep multi-scale video prediction beyond mean square error. Preprint at https://arxiv.org/abs/1511.05440 (2015).
Oh, J., Guo, X., Lee, H., Lewis, R. L. & Singh, S. Action-conditional video prediction using deep networks in Atari games. Adv. Neural Inf. Process. Syst. 28, 2863–2871 (2015).
Shi, X. et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting. Adv. Neural Inf. Process. Syst. 28, 802–810 (2015).
Deng, J. et al. ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2009).
Friedlingstein, P. et al. Uncertainties in CMIP5 climate projections due to carbon cycle feedbacks. J. Clim. 27, 511–526 (2014).
Montavon, G., Samek, W. & Müller, K.-R. Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2017).
Runge, J. et al. Identifying causal gateways and mediators in complex spatio-temporal systems. Nat. Commun. 6, 8502 (2015).
Chalupka, K., Bischoff, T., Perona, P. & Eberhardt, F. in UAI'16 Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence 72–81 (AUAI Press, 2016).
Ghahramani, Z. Probabilistic machine learning and artificial intelligence. Nature 521, 452–459 (2015).
Goodfellow, I. J. et al. Generative Adversarial Nets. Adv. Neural. Inf. Process. Syst. 27, 2672–2680 (2014). This is a fundamental paper on a deep generative modelling approach, allowing possible futures to be modelled from data.
Hansen, M. C. et al. High-resolution global maps of 21st-century forest cover change. Science 342, 850–853 (2013).
Pekel, J.-F., Cottam, A., Gorelick, N. & Belward, A. S. High-resolution mapping of global surface water and its long-term changes. Nature 540, 418–422 (2016).
Karpatne, A. et al. Theory-guided data science: a new paradigm for scientific discovery from data. IEEE Trans. Knowl. Data Eng. 29, 2318–2331 (2017).
Camps-Valls, G. et al. Physics-aware Gaussian processes in remote sensing. Appl. Soft Comput. 68, 69–82 (2018).
Karpatne, A., Watkins, W., Read, J. & Kumar, V. Physics-guided Neural Networks (PGNN): an application in lake temperature modeling. Preprint at https://arxiv.org/abs/1710.11431 (2017).
Luo, Y. Q. et al. A framework for benchmarking land models. Biogeosciences 9, 3857–3874 (2012).
Eyring, V. et al. Towards improved and more routine Earth system model evaluation in CMIP. Earth Syst. Dyn. 7, 813–830 (2016).
Klocke, D., Pincus, R. & Quaas, J. On constraining estimates of climate sensitivity with present-day observations through model weighting. J. Clim. 24, 6092–6099 (2011).
Cox, P. M. et al. Sensitivity of tropical carbon to climate change constrained by carbon dioxide variability. Nature 494, 341–344 (2013).
Beck, H. E. et al. Global-scale regionalization of hydrologic model parameters. Wat. Resour. Res. 52, 3599–3622 (2016).
Schirber, S., Klocke, D., Pincus, R., Quaas, J. & Anderson, J. L. Parameter estimation using data assimilation in an atmospheric general circulation model: from a perfect toward the real world. J. Adv. Model. Earth Syst. 5, 58–70 (2013).
Gentine, P., Pritchard, M., Rasp, S., Reinaudi, G. & Yacalis, G. Could machine learning break the convection parameterization deadlock? Geophys. Res. Lett. 45, 5742–5751 (2018).
Becker, T., Stevens, B. & Hohenegger, C. Imprint of the convective parameterization and sea-surface temperature on large-scale convective self-aggregation. J. Adv. Model. Earth Syst. 9, 1488–1505 (2017).
Siongco, A. C., Hohenegger, C. & Stevens, B. Sensitivity of the summertime tropical Atlantic precipitation distribution to convective parameterization and model resolution in ECHAM6. J. Geophys. Res. Atmos. 122, 2579–2594 (2017).
de Bezenac, E., Pajot, A. & Gallinari, P. Deep learning for physical processes: incorporating prior scientific knowledge. Preprint at https://arxiv.org/abs/1711.07970 (2017).
Brenowitz, N. D. & Bretherton, C. S. Prognostic validation of a neural network unified physics parameterization. Geophys. Res. Lett. 45, 6289–6298 (2018).
Willis, M. J. & von Stosch, M. Simultaneous parameter identification and discrimination of the nonparametric structure of hybrid semi-parametric models. Comput. Chem. Eng. 104, 366–376 (2017).
McGovern, A. et al. Using artificial intelligence to improve real-time decision making for high-impact weather. Bull. Am. Meteorol. Soc. 98, 2073–2090 (2017).
Vandal, T. et al. Generating high resolution climate change projections through single image super-resolution: an abridged version. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18) https://www.ijcai.org/proceedings/2018/0759.pdf (2018).
Verrelst, J. et al. Emulation of leaf, canopy and atmosphere radiative transfer models for fast global sensitivity analysis. Remote Sens. 8, 673 (2016).
Chevallier, F., Chéruy, F., Scott, N. & Chédin, A. A neural network approach for a fast and accurate computation of a longwave radiative budget. J. Appl. Meteorol. 37, 1385–1397 (1998).
Castruccio, S. et al. Statistical emulation of climate model projections based on precomputed GCM runs. J. Clim. 27, 1829–1844 (2014).
Fer, I. et al. Linking big models to big data: efficient ecosystem model calibration through Bayesian model emulation. Biogeosci. Disc. 2018, 1–30 (2018).
Glahn, H. R. & Lowry, D. A. The use of model output statistics (MOS) in objective weather forecasting. J. Appl. Meteorol. 11, 1203–1211 (1972).
Wilks, D. S. Multivariate ensemble model output statistics using empirical copulas. Q. J. R. Meteorol. Soc. 141, 945–952 (2015).
Tewari, A. et al. in Proc. IEEE Conf. on Computer Vision and Pattern Recognition 2549–2559 (IEEE, 2018).
Xie, Y., Franz, E., Chu, M. & Thuerey, N. tempoGAN: a temporally coherent, volumetric GAN for super-resolution fluid flow. Preprint at https://arxiv.org/abs/1801.09710 (2018).
Stewart, R. & Ermon, S. in Proc. Thirty-First AAAI Conf. on Artificial Intelligence (AAAI-17) 2576–2582 (2017).
Gunning, D. Explainable Artificial Intelligence (XAI) https://www.cc.gatech.edu/~alanwags/DLAI2016/(Gunning)%20IJCAI-16%20DLAI%20WS.pdf (2017).
Hu, Z., Ma, X., Liu, Z., Hovy, E. & Xing, E. in Proc. 54th Annual Meeting of the Association for Computational Linguistics Vol. 1, 2410–2420 (Association for Computational Linguistics, 2016).
Pearlmutter, B. A. & Siskind, J. M. Reverse-mode AD in a functional framework: lambda the ultimate backpropagator. ACM Trans. Progr. Lang. Syst. 30, 7 (2008).
Wang, F. & Rompf, T. in ICLR 2018 Workshop https://openreview.net/pdf?id=SJxJtYkPG (2018).
Beer, C. et al. Terrestrial gross carbon dioxide uptake: global distribution and covariation with climate. Science 329, 834–838 (2010).
Forkel, M. et al. Enhanced seasonal CO2 exchange caused by amplified plant productivity in northern ecosystems. Science 351, 696–699 (2016).
Bellprat, O., Kotlarski, S., Lüthi, D. & Schär, C. Objective calibration of regional climate models. J. Geophys. Res. Atmos. 117, D23115 (2012).
Reichstein, M. et al. in AGU Fall Meeting Abstracts 2016AGUFM.B2044A.2007R (AGU, 2016).
Rußwurm, M. & Körner, M. Multi-temporal land cover classification with long short-term memory neural networks. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci 42, 551–558 (2017). This paper describes the first use of the LSTM deep learning model for multi-temporal land-cover classification.
Nash, J. E. & Sutcliffe, J. V. River flow forecasting through conceptual models. Part I—a discussion of principles. J. Hydrol. 10, 282–290 (1970).
Shi, X. et al. Deep learning for precipitation nowcasting: a benchmark and a new model. Adv. Neural. Inf. Process. Syst. 30, 5617–5627 (2017). This paper describes the first approach to data-driven modelling of near-term precipitation using a combination of deep-learning concepts, that is, LSTMs and convolutional neural networks.
Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. Preprint at https://arxiv.org/abs/1611.07004 (2016). This paper is a geoscience-related extension application of ref. 53, in which, for example, remote sensing images are transferred to thematic maps.
Tompson, J., Schlachter, K., Sprechmann, P. & Perlin, K. Accelerating Eulerian fluid simulation with convolutional networks. Proc. Machine Learning Res. 70, 3424–3433 (2017).
University Corporation for Atmospheric Research (UCAR). Short-Term Explicit Prediction (STEP) Program Research Applications Laboratory 2013 Annual Report https://nar.ucar.edu/2013/ral/short-term-explicit-prediction-step-program (NCAR/UCAR, 2013).
Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28, 91–99 (2015).
Zaytar, M. A. & El Amrani, C. Sequence to sequence weather forecasting with long short term memory recurrent neural networks. Int. J. Comput. Appl. 143, 7–11 (2016).
Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning Vol. xxii, 1–775 (MIT Press, Cambridge, 2016).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput 9, 1735–1780 (1997).
Schmidhuber, J. Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015).
Requena-Mesa, C., Reichstein, M., Mahecha, M., Kraft, B. & Denzler, J. Predicting landscapes as seen from space from environmental conditions. In IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 1768–1771 (IEEE, 2018).
We thank D. Frank and L. Maack for proofreading, help with the literature and technical help, and F. Gans for programming help in Julia. This research was supported by a grant by the Alexander von Humboldt Foundation (Max Planck Research Prize) to M.R. G.C.-V. was supported by the European Research Council (ERC) under the ERC Consolidator Grant ERC-CoG-2014 SEDAL (grant agreement 647423).
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Reichstein, M., Camps-Valls, G., Stevens, B. et al. Deep learning and process understanding for data-driven Earth system science. Nature 566, 195–204 (2019). https://doi.org/10.1038/s41586-019-0912-1
Remote Sensing of Environment (2021)
Remote Sensing (2021)
Advanced Materials (2021)
A Spatial-Temporal Interpretable Deep Learning Model for improving interpretability and predictive accuracy of satellite-based PM2.5
Environmental Pollution (2021)
Forecasting PM2.5 using hybrid graph convolution-based model considering dynamic wind-field to offer the benefit of spatial interpretability
Environmental Pollution (2021)