Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Differentiable modelling to unify machine learning and physical models for geosciences

Nature Reviews Earth & Environment volume 4, pages 552–567 (2023)Cite this article

Subjects

Abstract

Process-based modelling offers interpretability and physical consistency in many domains of geosciences but struggles to leverage large datasets efficiently. Machine-learning methods, especially deep networks, have strong predictive skills yet are unable to answer specific scientific questions. In this Perspective, we explore differentiable modelling as a pathway to dissolve the perceived barrier between process-based modelling and machine learning in the geosciences and demonstrate its potential with examples from hydrological modelling. ‘Differentiable’ refers to accurately and efficiently calculating gradients with respect to model variables or parameters, enabling the discovery of high-dimensional unknown relationships. Differentiable modelling involves connecting (flexible amounts of) prior physical knowledge to neural networks, pushing the boundary of physics-informed machine learning. It offers better interpretability, generalizability, and extrapolation capabilities than purely data-driven machine learning, achieving a similar level of accuracy while requiring less training data. Additionally, the performance and efficiency of differentiable models scale well with increasing data volumes. Under data-scarce scenarios, differentiable models have outperformed machine-learning models in producing short-term dynamics and decadal-scale trends owing to the imposed physical constraints. Differentiable modelling approaches are primed to enable geoscientists to ask questions, test hypotheses, and discover unrecognized physical relationships. Future work should address computational challenges, reduce uncertainty, and verify the physical significance of outputs.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Synergies from combining machine learning and process-based modelling.
Fig. 2: Why differentiable models are interpretable and performant.
Fig. 3: An example differentiable hydrological model.
Fig. 4: Differentiable parameter learning.
Fig. 5: Physics-informed neural networks for learning constitutive relationships in unsaturated flow models.

References

  1. Ajami, N. K., Gupta, H., Wagener, T. & Sorooshian, S. Calibration of a semi-distributed hydrologic model for streamflow estimation along a river system. J. Hydrol. 298, 112–135 (2004).

    Article  Google Scholar 

  2. van Griensven, A. & Meixner, T. A global and efficient multi-objective auto-calibration and uncertainty estimation method for water quality catchment models. J. Hydroinform. 9, 277–291 (2007).

    Article  Google Scholar 

  3. Barendrecht, M. H. et al. The value of empirical data for estimating the parameters of a sociohydrological flood risk model. Water Resour. Res. 55, 1312–1336 (2019).

    Article  Google Scholar 

  4. Post, H., Vrugt, J. A., Fox, A., Vereecken, H. & Franssen, H.-J. H. Estimation of Community Land Model parameters for an improved assessment of net carbon fluxes at European sites. J. Geophys. Res. Biogeosci. 122, 661–689 (2017).

    Article  Google Scholar 

  5. Aumont, O., Ethé, C., Tagliabue, A., Bopp, L. & Gehlen, M. PISCES-v2: an ocean biogeochemical model for carbon and ecosystem studies. Geosci. Model. Dev. 8, 2465–2513 (2015).

    Article  Google Scholar 

  6. Ahmed, M. et al. Calibration and validation of APSIM-Wheat and CERES-Wheat for spring wheat under rainfed conditions: models evaluation and application. Comput. Electron. Agric. 123, 384–401 (2016).

    Article  Google Scholar 

  7. Lepore, C., Arnone, E., Noto, L. V., Sivandran, G. & Bras, R. L. Physically based modeling of rainfall-triggered landslides: a case study in the Luquillo forest, Puerto Rico. Hydrol. Earth Syst. Sci. 17, 3371–3387 (2013).

    Article  Google Scholar 

  8. Shirzaei, M. et al. Measuring, modelling and projecting coastal land subsidence. Nat. Rev. Earth Environ. 2, 40–58 (2021).

    Article  Google Scholar 

  9. Biemans, H. et al. Importance of snow and glacier meltwater for agriculture on the Indo-Gangetic Plain. Nat. Sustain. 2, 594–601 (2019).

    Article  Google Scholar 

  10. Li, L. et al. Toward catchment hydro-biogeochemical theories. WIREs Water 8, e1495 (2021).

    Article  Google Scholar 

  11. Steefel, C. I. et al. Reactive transport codes for subsurface environmental simulation. Comput. Geosci. 19, 445–478 (2015).

    Article  Google Scholar 

  12. Li, L. et al. Expanding the role of reactive transport models in critical zone processes. Earth Sci. Rev. 165, 280–301 (2017).

    Article  Google Scholar 

  13. Flato, G. M. Earth system models: an overview. WIREs Clim. Change 2, 783–800 (2011).

    Article  Google Scholar 

  14. Danabasoglu, G. et al. The Community Earth System Model version 2 (CESM2). J. Adv. Modeling Earth Syst. 12, e2019MS001916 (2020).

    Google Scholar 

  15. Eyring, V. et al. Overview of the Coupled Model Intercomparison Project phase 6 (CMIP6) experimental design and organization. Geosci. Model. Dev. 9, 1937–1958 (2016).

    Article  Google Scholar 

  16. Calvin, K. et al. GCAM v5.1: representing the linkages between energy, water, land, climate, and economic systems. Geosci. Model. Dev. 12, 677–698 (2019).

    Article  Google Scholar 

  17. ISIMIP. The Inter-Sectoral Impact Model Intercomparison Project (ISIMIP). https://www.isimip.org/ (2022).

  18. Lange, S. Trend-preserving bias adjustment and statistical downscaling with ISIMIP3BASD (v1.0). Geosci. Model. Dev. 12, 3055–3070 (2019).

    Article  Google Scholar 

  19. Weyant, J. et al. in Climate Change 1995: Social and Economic Dimensions of Climate Change (eds Bruce, J. P., Lee, H. & Haites, E. F.) 367–396 (IPCC, Cambridge Univ. Press, 1996).

  20. IPCC. Climate Change 2021: The Physical Science Basis (eds Masson-Delmotte, V. et al.) (Cambridge Univ. Press, 2021).

  21. Clark, M. P. et al. Improving the representation of hydrologic processes in Earth system models. Water Resour. Res. 51, 5929–5956 (2015).

    Article  Google Scholar 

  22. Geary, W. L. et al. A guide to ecosystem models and their environmental applications. Nat. Ecol. Evol. 4, 1459–1471 (2020).

    Article  Google Scholar 

  23. Fatichi, S. et al. An overview of current applications, challenges, and future trends in distributed process-based models in hydrology. J. Hydrol. 537, 45–60 (2016).

    Article  Google Scholar 

  24. Wagener, T. et al. On doing hydrology with dragons: realizing the value of perceptual models and knowledge accumulation. WIREs Water 8, e1550 (2021).

    Article  Google Scholar 

  25. Hood, R. R. et al. The Chesapeake Bay program modeling system: overview and recommendations for future development. Ecol. Model. 456, 109635 (2021).

    Article  Google Scholar 

  26. Fan, Y. et al. Hillslope hydrology in global change research and Earth system modeling. Water Resour. Res. 55, 1737–1772 (2019).

    Article  Google Scholar 

  27. van Kampenhout, L. et al. Improving the representation of polar snow and firn in the Community Earth System Model. J. Adv. Modeling Earth Syst. 9, 2583–2600 (2017).

    Article  Google Scholar 

  28. Medlyn, B. E. et al. Using ecosystem experiments to improve vegetation models. Nat. Clim. Change 5, 528–534 (2015).

    Article  Google Scholar 

  29. Nearing, G. S. et al. What role does hydrological science play in the age of machine learning? Water Resour. Res. 57, e2020WR028091 (2021).

    Article  Google Scholar 

  30. Shen, C. et al. HESS Opinions: incubating deep-learning-powered hydrologic science advances as a community. Hydrol. Earth Syst. Sci. 22, 5639–5656 (2018).

    Article  Google Scholar 

  31. Hunt, R. J., Fienen, M. N. & White, J. T. Revisiting ‘An exercise in groundwater model calibration and prediction’ after 30 years: insights and new directions. Groundwater 58, 168–182 (2020).

    Article  Google Scholar 

  32. Addor, N. & Melsen, L. A. Legacy, rather than adequacy, drives the selection of hydrological models. Water Resour. Res. 55, 378–390 (2019).

    Article  Google Scholar 

  33. Clark, M. P., Kavetski, D. & Fenicia, F. Pursuing the method of multiple working hypotheses for hydrological modeling. Water Resour. Res. 47, WR009827 (2011).

    Article  Google Scholar 

  34. Jakeman, A. J. & Hornberger, G. M. How much complexity is warranted in a rainfall-runoff model? Water Resour. Res. 29, 2637–2649 (1993).

    Article  Google Scholar 

  35. Wagener, T., Wheater, H. S. & Gupta, H. V. in Calibration of Watershed Models 29–47 (Wiley, 2003).

  36. Young, P., Jakeman, A. & McMurtrie, R. An instrumental variable method for model order identification. Automatica 16, 281–294 (1980).

    Article  Google Scholar 

  37. Shen, C. A transdisciplinary review of deep learning research and its relevance for water resources scientists. Water Resour. Res. 54, 8558–8593 (2018).

    Article  Google Scholar 

  38. Abbott, B. W. et al. Human domination of the global water cycle absent from depictions and perceptions. Nat. Geosci. 12, 533–540 (2019).

    Article  Google Scholar 

  39. Lemordant, L., Gentine, P., Swann, A. S., Cook, B. I. & Scheff, J. Critical impact of vegetation physiology on the continental hydrologic cycle in response to increasing CO2. Proc. Natl Acad. Sci. USA 115, 4093–4098 (2018).

    Article  Google Scholar 

  40. Trancoso, R., Larsen, J. R., McVicar, T. R., Phinn, S. R. & McAlpine, C. A. CO2–vegetation feedbacks and other climate changes implicated in reducing base flow. Geophys. Res. Lett. 44, 2310–2318 (2017).

    Article  Google Scholar 

  41. Yu, D. et al. Socio-hydrology: an interplay of design and self-organization in a multilevel world. Ecol. Soc. 25, 22 (2020).

    Article  Google Scholar 

  42. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  43. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).

    Article  Google Scholar 

  44. Yin, H., Guo, Z., Zhang, X., Chen, J. & Zhang, Y. RR-Former: rainfall-runoff modeling based on transformer. J. Hydrol. 609, 127781 (2022).

    Article  Google Scholar 

  45. Amanambu, A. C., Mossa, J. & Chen, Y.-H. Hydrological drought forecasting using a deep transformer model. Water 14, 3611 (2022).

    Article  Google Scholar 

  46. Sun, A. Y., Jiang, P., Mudunuru, M. K. & Chen, X. Explore spatio-temporal learning of large sample hydrology using graph neural networks. Water Resour. Res. 57, e2021WR030394 (2021).

    Article  Google Scholar 

  47. Krizhevsky, A., Sutskever, I. & Hinton, G. E. in Advances in Neural Information Processing Systems vol. 25 (eds Bartlett, P. et al.) 1097–1105 (Curran Associates, 2012).

  48. Lecun, Y. & Bengio, Y. Convolutional networks for images, speech, and time-series. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 255–258 (MIT Press, 1995).

  49. Khaki, S. & Wang, L. Crop yield prediction using deep neural networks. Front. Plant Sci. 10, 00621 (2019).

    Article  Google Scholar 

  50. Wang, A. X., Tran, C., Desai, N., Lobell, D. & Ermon, S. Deep transfer learning for crop yield prediction with remote sensing data. In Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies, 1–5 (ACM, 2018).

  51. Pan, B. et al. Improving seasonal forecast using probabilistic deep learning. J. Adv. Model. Earth Syst. 14, e2021MS002766 (2022).

    Article  Google Scholar 

  52. Shi, X. et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In Proceedings of the 28th International Conference on Neural Information Processing Systems Vol. 1, 802–810 (MIT Press, 2015).

  53. Bhowmik, M., Singh, M., Rao, S. & Paul, S. DeepClouds.ai: deep learning enabled computationally cheap direct numerical simulations. Preprint at https://doi.org/10.48550/arXiv.2208.08956 (2022).

  54. Lin, G.-Y., Chen, H.-W., Chen, B.-J. & Yang, Y.-C. Characterization of temporal PM2.5, nitrate, and sulfate using deep learning techniques. Atmos. Pollut. Res. 13, 101260 (2022).

    Article  Google Scholar 

  55. Varadharajan, C. et al. Can machine learning accelerate process understanding and decision-relevant predictions of river water quality? Hydrol. Process. 36, e14565 (2022).

    Article  Google Scholar 

  56. Jia, X. et al. Physics-guided recurrent graph model for predicting flow and temperature in river networks. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), 612–620 (Society for Industrial and Applied Mathematics, 2021).

  57. Rahmani, F. et al. Exploring the exceptional performance of a deep learning stream temperature model and the value of streamflow data. Environ. Res. Lett. https://doi.org/10.1088/1748-9326/abd501 (2021).

    Article  Google Scholar 

  58. Rahmani, F., Shen, C., Oliver, S., Lawson, K. & Appling, A. Deep learning approaches for improving prediction of daily stream temperature in data-scarce, unmonitored, and dammed basins. Hydrol. Process. 35, e14400 (2021).

    Article  Google Scholar 

  59. Read, J. S. et al. Process-guided deep learning predictions of lake water temperature. Water Resour. Res. 55, 9173–9190 (2019).

    Article  Google Scholar 

  60. Zhi, W. et al. From hydrometeorology to river water quality: can a deep learning model predict dissolved oxygen at the continental scale? Environ. Sci. Technol. 55, 2357–2368 (2021).

    Article  Google Scholar 

  61. Zhi, W., Ouyang, W., Shen, C. & Li, L. Temperature outweighs light and flow as the predominant driver of dissolved oxygen in US rivers. Nat. Water 1, 249–260 (2023).

    Article  Google Scholar 

  62. He, M., Wu, S., Huang, B., Kang, C. & Gui, F. Prediction of total nitrogen and phosphorus in surface water by deep learning methods based on multi-scale feature extraction. Water 14, 1643 (2022).

    Article  Google Scholar 

  63. Hrnjica, B., Mehr, A. D., Jakupović, E., Crnkić, A. & Hasanagić, R. Application of deep learning neural networks for nitrate prediction in the Klokot River, Bosnia and Herzegovina. In 7th International Conference on Control, Instrumentation and Automation (ICCIA) 1–6 (IEEE, 2021).

  64. Xiong, R. et al. Predicting dynamic riverine nitrogen export in unmonitored watersheds: leveraging insights of AI from data-rich regions. Environ. Sci. Technol. 56, 10530–10542 (2022).

    Article  Google Scholar 

  65. Shen, C., Chen, X. & Laloy, E. Editorial: broadening the use of machine learning in hydrology. Front. Water, https://doi.org/10.3389/frwa.2021.681023 (2021).

  66. Fang, K., Shen, C., Kifer, D. & Yang, X. Prolongation of SMAP to spatiotemporally seamless coverage of continental U.S. using a deep learning neural network. Geophys. Res. Lett. 44, 11030–11039 (2017).

    Article  Google Scholar 

  67. Fang, K., Pan, M. & Shen, C. The value of SMAP for long-term soil moisture estimation with the help of deep learning. IEEE Trans. Geosci. Remote. Sens. 57, 2221–2233 (2019).

    Article  Google Scholar 

  68. Fang, K. & Shen, C. Near-real-time forecast of satellite-based soil moisture using long short-term memory with an adaptive data integration kernel. J. Hydrometeorol. 21, 399–413 (2020).

    Article  Google Scholar 

  69. Feng, D., Fang, K. & Shen, C. Enhancing streamflow forecast and extracting insights using long-short term memory networks with data integration at continental scales. Water Resour. Res. 56, e2019WR026793 (2020).

    Article  Google Scholar 

  70. Kratzert, F. et al. Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrol. Earth Syst. Sci. 23, 5089–5110 (2019).

    Article  Google Scholar 

  71. Xiang, Z. & Demir, I. Distributed long-term hourly streamflow predictions using deep learning — a case study for State of Iowa. Environ. Model. Softw. 131, 104761 (2020).

    Article  Google Scholar 

  72. Alemohammad, S. H. et al. Water, energy, and carbon with artificial neural networks (WECANN): a statistically based estimate of global surface turbulent fluxes and gross primary productivity using solar-induced fluorescence. Biogeosciences 14, 4101–4124 (2017).

    Article  Google Scholar 

  73. Jung, M. et al. The FLUXCOM ensemble of global land-atmosphere energy fluxes. Sci. Data 6, 74 (2019).

    Article  Google Scholar 

  74. Zhao, W. L. et al. Physics-constrained machine learning of evapotranspiration. Geophys. Res. Lett. 46, 14496–14507 (2019).

    Article  Google Scholar 

  75. Afzaal, H., Farooque, A. A., Abbas, F., Acharya, B. & Esau, T. Groundwater estimation from major physical hydrology components using artificial neural networks and deep learning. Water 12, 5 (2020).

    Article  Google Scholar 

  76. Meyal, A. Y. et al. Automated cloud based long short-term memory neural network based SWE prediction. Front. Water 2, 574917 (2020).

    Article  Google Scholar 

  77. McDonnell, J. J. & Beven, K. Debates — the future of hydrological sciences: a (common) path forward? A call to action aimed at understanding velocities, celerities and residence time distributions of the headwater hydrograph. Water Resour. Res. 50, 5342–5350 (2014).

    Article  Google Scholar 

  78. Appling, A. P., Oliver, S. K., Read, J. S., Sadler, J. M. & Zwart, J. Machine learning for understanding inland water quantity, quality, and ecology. earthArXiv preprint at https://doi.org/10.1016/B978-0-12-819166-8.00121-3 (2022).

    Article  Google Scholar 

  79. Fang, K., Kifer, D., Lawson, K., Feng, D. & Shen, C. The data synergy effects of time-series deep learning models in hydrology. Water Resour. Res. 58, e2021WR029583 (2022).

    Article  Google Scholar 

  80. McGovern, A., Ebert-Uphoff, I., Gagne, D. J. & Bostrom, A. Why we need to focus on developing ethical, responsible, and trustworthy artificial intelligence approaches for environmental science. Environ. Data Sci. 1, e6 (2022).

    Article  Google Scholar 

  81. Schölkopf, B. in Probabilistic and Causal Inference: The Works of Judea Pearl vol. 36, 765–804 (Association for Computing Machinery, 2022).

  82. Bach, S. et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10, e0130140 (2015).

    Article  Google Scholar 

  83. Montavon, G., Samek, W. & Müller, K.-R. Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2018).

    Article  Google Scholar 

  84. Toms, B. A., Barnes, E. A. & Ebert-Uphoff, I. Physically interpretable neural networks for the geosciences: applications to Earth system variability. J. Adv. Modeling Earth Syst. 12, e2019MS002002 (2020).

    Google Scholar 

  85. Fleming, S. W., Watson, J. R., Ellenson, A., Cannon, A. J. & Vesselinov, V. C. Machine learning in Earth and environmental science requires education and research policy reforms. Nat. Geosci. 14, 878–880 (2021).

    Article  Google Scholar 

  86. Hornik, K. Approximation capabilities of multilayer feedforward networks. Neural Netw. 4, 251–257 (1991).

    Article  Google Scholar 

  87. Hornik, K., Stinchcombe, M. & White, H. Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989).

    Article  Google Scholar 

  88. Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https://arxiv.org/abs/2303.12712 (2023).

  89. Deb, K., Pratap, A., Agarwal, S. & Meyarivan, T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evolut. Comput. 6, 182–197 (2002).

    Article  Google Scholar 

  90. Duan, Q., Sorooshian, S. & Gupta, V. Effective and efficient global optimization for conceptual rainfall-runoff models. Water Resour. Res. 28, 1015–1031 (1992).

    Article  Google Scholar 

  91. Zitzler, E., Laumanns, M. & Thiele, L. SPEA2: Improving the Strength Pareto Evolutionary Algorithm. TIK Report vol. 103 (ETH Zurich, 2001); https://www.research-collection.ethz.ch/handle/20.500.11850/145755.

  92. Liu, S. et al. A hybrid approach of support vector regression with genetic algorithm optimization for aquaculture water quality prediction. Math. Comput. Model. 58, 458–465 (2013).

    Article  Google Scholar 

  93. Zambrano-Bigiarini, M. & Rojas, R. A model-independent particle swarm optimisation software for model calibration. Environ. Model. Softw. 43, 5–25 (2013).

    Article  Google Scholar 

  94. Baydin, A. G., Pearlmutter, B. A., Radul, A. A. & Siskind, J. M. Automatic differentiation in machine learning: a survey. J. Mach. Learn. Res. 18, 1–43 (2018).

    Google Scholar 

  95. Innes, M. et al. A differentiable programming system to bridge machine learning and scientific computing. Preprint at https://arxiv.org/abs/1907.07587 (2019).

  96. Goodfellow, I., Bengio, Y. & Courville, A. in Deep Learning (ed. Goodfellow, I.) Ch. 4 (MIT Press, 2016).

  97. Paszke, A. et al. Automatic differentiation in PyTorch. In 31st Conference on Neural Information Processing Systems (NIPS 2017) (NIPS, 2017).

  98. Bradbury, J. et al. JAX: Autograd and XLA. Astrophysics Source Code Library record ascl:2111.002 (Astrophysics Source Code Library, 2021).

  99. Bezanson, J., Edelman, A., Karpinski, S. & Shah, V. B. Julia: a fresh approach to numerical computing. SIAM Rev. 59, 65–98 (2017).

    Article  Google Scholar 

  100. Abadi, M. et al. Tensorflow: a system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) 265–283 (USENIX Association, 2016).

  101. Errico, R. M. What is an adjoint model? Bull. Am. Meteorol. Soc. 78, 2577–2592 (1997).

    Article  Google Scholar 

  102. Johnson, S. G. Notes on Adjoint Methods for 18.335 (2021).

  103. Pal, A., Edelman, A. & Rackauckas, C. Mixing implicit and explicit deep learning with skip DEQs and infinite time neural ODEs (continuous DEQs). Preprint at https://doi.org/10.48550/arXiv.2201.12240 (2022).

  104. Ghattas, O. & Willcox, K. Learning physics-based models from data: perspectives from inverse problems and model reduction. Acta Numerica 30, 445–554 (2021).

    Article  Google Scholar 

  105. Baker, N. et al. Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence. https://www.osti.gov/biblio/1478744 (2019).

  106. Rackauckas, C. et al. Universal differential equations for scientific machine learning. Preprint at https://doi.org/10.48550/arXiv.2001.04385 (2021).

  107. Feng, D., Liu, J., Lawson, K. & Shen, C. Differentiable, learnable, regionalized process-based models with multiphysical outputs can approach state-of-the-art hydrologic prediction accuracy. Water Resour. Res. 58, e2022WR032404 (2022).

    Article  Google Scholar 

  108. Huang, D. Z., Xu, K., Farhat, C. & Darve, E. Learning constitutive relations from indirect observations using deep neural networks. J. Comput. Phys. 416, 109491 (2020).

    Article  Google Scholar 

  109. Tartakovsky, A. M., Marrero, C. O., Perdikaris, P., Tartakovsky, G. D. & Barajas‐Solano, D. Physics-informed deep neural networks for learning parameters and constitutive relationships in subsurface flow problems. Water Resour. Res. 56, e2019WR026731 (2020).

    Article  Google Scholar 

  110. Padarian, J., McBratney, A. B. & Minasny, B. Game theory interpretation of digital soil mapping convolutional neural networks. Soil 6, 389–397 (2020).

    Article  Google Scholar 

  111. Udrescu, S.-M. & Tegmark, M. AI Feynman: a physics-inspired method for symbolic regression. Sci. Adv. 6, eaay2631 (2020).

    Article  Google Scholar 

  112. Ma, Y., Tsao, D. & Shum, H.-Y. On the principles of parsimony and self-consistency for the emergence of intelligence. Front. Inf. Technol. Electron. Eng. 23, 1298–1323 (2022).

    Article  Google Scholar 

  113. Haber, E. & Ruthotto, L. Stable architectures for deep neural networks. Inverse Probl. 34, 014004 (2018).

    Article  Google Scholar 

  114. Chen, R. T. Q., Rubanova, Y., Bettencourt, J. & Duvenaud, D. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 6572–6583 (Curran Associates, 2018).

  115. Mahecha, M. D. et al. Earth system data cubes unravel global multivariate dynamics. Earth Syst. Dyn. 11, 201–234 (2020).

    Article  Google Scholar 

  116. Myneni, R., Knyazikhin, Y. & Park, T. MCD15A2H MODIS/Terra+Aqua Leaf Area Index/FPAR 8-day L4 Global 500m SIN Grid V006 (2015); https://doi.org/10.5067/MODIS/MCD15A2H.006.

  117. About SMOS — Soil Moisture and Ocean Salinity Mission (ESA, 2022); https://earth.esa.int/eogateway/missions/smos.

  118. O’Neill, P. E. et al. SMAP enhanced L3 radiometer global and polar grid daily 9 km EASE-grid soil moisture, Version 5 (SPL3SMP_E) (2021); https://doi.org/10.5067/4DQ54OUIJ9DL.

  119. Lin, Y.-S. et al. Optimal stomatal behaviour around the world. Nat. Clim. Change 5, 459–464 (2015).

    Article  Google Scholar 

  120. Feng, D., Lawson, K. & Shen, C. Mitigating prediction error of deep learning streamflow models in large data-sparse regions with ensemble modeling and soft data. Geophys. Res. Lett. 48, e2021GL092999 (2021).

    Article  Google Scholar 

  121. Feng, D., Beck, H., Lawson, K. & Shen, C. The suitability of differentiable, learnable hydrologic models for ungauged regions and climate change impact assessment. Hydrology and Earth System Sciences Discussions 1–28 (European Geoscience Union, 2022); https://doi.org/10.5194/hess-2022-245.

  122. Wagener, T. et al. The future of hydrology: an evolving science for a changing world. Water Resour. Res. 46, 1–10 (2010).

    Article  Google Scholar 

  123. Liu, J., Hughes, D., Rahmani, F., Lawson, K. & Shen, C. Evaluating a global soil moisture dataset from a multitask model (GSM3 v1.0) with potential applications for crop threats. Geosci. Model. Dev. 16, 1553–1567 (2023).

    Article  Google Scholar 

  124. Tsai, W.-P. et al. From calibration to parameter learning: harnessing the scaling effects of big data in geoscientific modeling. Nat. Commun. 12, 5988 (2021).

    Article  Google Scholar 

  125. Jiang, S., Zheng, Y. & Solomatine, D. Improving AI system awareness of geoscience knowledge: Symbiotic integration of physical approaches and deep learning. Geophys. Res. Lett. 47, e2020GL088229 (2020).

    Article  Google Scholar 

  126. Beven, K. A manifesto for the equifinality thesis. J. Hydrol. 320, 18–36 (2006).

    Article  Google Scholar 

  127. Pokhrel, P., Gupta, H. V. & Wagener, T. A spatial regularization approach to parameter estimation for a distributed watershed model. Water Resour. Res. 44, WR006615 (2008).

    Article  Google Scholar 

  128. Wagener, T., McIntyre, N., Lees, M. J., Wheater, H. S. & Gupta, H. V. Towards reduced uncertainty in conceptual rainfall-runoff modelling: dynamic identifiability analysis. Hydrol. Process. 17, 455–476 (2003).

    Article  Google Scholar 

  129. Onken, D. & Ruthotto, L. Discretize-optimize vs. optimize-discretize for time-series regression and continuous normalizing flows. Preprint at https://arxiv.org/abs/2005.13420 (2020).

  130. Mitusch, S. K., Funke, S. W. & Kuchta, M. Hybrid FEM-NN models: combining artificial neural networks with the finite element method. J. Comput. Phys. 446, 110651 (2021).

    Article  Google Scholar 

  131. Farrell, P. E., Ham, D. A., Funke, S. W. & Rognes, M. E. Automated derivation of the adjoint of high-level transient finite element programs. SIAM J. Sci. Comput. 35, C369–C393 (2013).

    Article  Google Scholar 

  132. Fisher, M. & Andersson, E. Developments in 4D-Var and Kalman Filtering. https://www.ecmwf.int/sites/default/files/elibrary/2001/9409-developments-4d-var-and-kalman-filtering.pdf (2001).

  133. Neupauer, R. M. & Wilson, J. L. Adjoint-derived location and travel time probabilities for a multidimensional groundwater system. Water Resour. Res. 37, 1657–1668 (2001).

    Article  Google Scholar 

  134. Clark, M. P. & Kavetski, D. Ancient numerical daemons of conceptual hydrological modeling: 1. Fidelity and efficiency of time stepping schemes. Water Resour. Res. 46, W10510 (2010).

    Article  Google Scholar 

  135. Kavetski, D. & Clark, M. P. Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction. Water Resour. Res. 46, W10511 (2010).

    Article  Google Scholar 

  136. Höge, M., Scheidegger, A., Baity-Jesi, M., Albert, C. & Fenicia, F. Improving hydrologic models for predictions and process understanding using neural ODEs. Hydrol. Earth Syst. Sci. 26, 5085–5102 (2022).

    Article  Google Scholar 

  137. Aboelyazeed, D. et al. A differentiable ecosystem modeling framework for large-scale inverse problems: demonstration with photosynthesis simulations. Biogeosci. Discuss. https://doi.org/10.5194/bg-2022-211 (2022).

    Article  Google Scholar 

  138. Bao, T. et al. Partial differential equation driven dynamic graph networks for predicting stream water temperature. in 2021 IEEE International Conference on Data Mining (ICDM) 11–20 (IEEE, 2021); https://doi.org/10.1109/ICDM51629.2021.00011.

  139. Bindas, T. et al. Improving large-basin streamflow simulation using a modular, differentiable, learnable graph model for routing. Preprint at https://doi.org/10.1002/essoar.10512512.1 (2023).

  140. Forghani, M. et al. Application of deep learning to large scale riverine flow velocity estimation. Stoch. Environ. Res. Risk Assess. 35, 1069–1088 (2021).

    Article  Google Scholar 

  141. Forghani, M. et al. Variational encoder geostatistical analysis (VEGAS) with an application to large scale riverine bathymetry. Adv. Water Resour. 170, 104323 (2022).

    Article  Google Scholar 

  142. Asher, M. J., Croke, B. F. W., Jakeman, A. J. & Peeters, L. J. M. A review of surrogate models and their application to groundwater modeling. Water Resour. Res. 51, 5957–5973 (2015).

    Article  Google Scholar 

  143. Blechschmidt, J. & Ernst, O. G. Three ways to solve partial differential equations with neural networks — a review. GAMM-Mitteilungen 44, e202100006 (2021).

    Article  Google Scholar 

  144. Lu, L., Meng, X., Mao, Z. & Karniadakis, G. E. DeepXDE: a deep learning library for solving differential equations. SIAM Rev. 63, 208–228 (2021).

    Article  Google Scholar 

  145. Takamoto, M. et al. PDEBENCH: an extensive benchmark for scientific machine learning. Preprint at https://arxiv.org/abs/2210.07182 (2022).

  146. Maxwell, R. M., Condon, L. E. & Melchior, P. A physics-informed, machine learning emulator of a 2D surface water model: what temporal networks and simulation-based inference can help us learn about hydrologic processes. Water 13, 3633 (2021).

    Article  Google Scholar 

  147. Liu, X., Song, Y. & Shen, C. Bathymetry inversion using a deep-learning-based surrogate for shallow water equations solvers. Preprint at https://doi.org/10.48550/arXiv.2203.02821 (2022).

  148. Raissi, M., Perdikaris, P. & Karniadakis, G. E. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019).

    Article  Google Scholar 

  149. He, Q., Barajas-Solano, D., Tartakovsky, G. & Tartakovsky, A. M. Physics-informed neural networks for multiphysics data assimilation with application to subsurface transport. Adv. Water Resour. 141, 103610 (2020).

    Article  Google Scholar 

  150. Wang, N., Zhang, D., Chang, H. & Li, H. Deep learning of subsurface flow via theory-guided neural network. J. Hydrol. 584, 124700 (2020).

    Article  Google Scholar 

  151. Brown, T. B. et al. Language models are few-shot learners. In Proc. of the 34th International Conference on Neural Information Processing Systems, 1877–1901 (2020).

  152. Kraft, B., Jung, M., Körner, M. & Reichstein, M. in The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences vol. XLIII-B2-2020, 1537–1544 (Copernicus, 2020).

  153. Kraft, B., Jung, M., Körner, M., Koirala, S. & Reichstein, M. Towards hybrid modeling of the global hydrological cycle. Hydrol. Earth Syst. Sci. 26, 1579–1614 (2022).

    Article  Google Scholar 

  154. Liu, J., Rahmani, F., Lawson, K. & Shen, C. A multiscale deep learning model for soil moisture integrating satellite and in situ data. Geophys. Res. Lett. 49, e2021GL096847 (2022).

    Google Scholar 

  155. Karniadakis, G. E. et al. Physics-informed machine learning. Nat. Rev. Phys. 3, 422–440 (2021).

    Article  Google Scholar 

  156. Karpatne, A. et al. Theory-guided data science: a new paradigm for scientific discovery from data. IEEE Trans. Knowl. Data Eng. 29, 2318–2331 (2017).

    Article  Google Scholar 

  157. Khandelwal, A. et al. Physics guided machine learning methods for hydrology. https://arxiv.org/abs/2012.02854 (2020).

  158. Pawar, S., San, O., Aksoylu, B., Rasheed, A. & Kvamsdal, T. Physics guided machine learning using simplified theories. Phys. Fluids 33, 011701 (2021).

    Article  Google Scholar 

  159. Reichstein, M. et al. Deep learning and process understanding for data-driven Earth system science. Nature 566, 195–204 (2019).

    Article  Google Scholar 

  160. Bennett, A. & Nijssen, B. Deep learned process parameterizations provide better representations of turbulent heat fluxes in hydrologic models. Water Resour. Res. 57, e2020WR029328 (2021).

    Article  Google Scholar 

  161. Schaap, M. G., Leij, F. J. & van Genuchten, M. Th. Rosetta: a computer program for estimating soil hydraulic parameters with hierarchical pedotransfer functions. J. Hydrol. 251, 163–176 (2001).

    Article  Google Scholar 

  162. Rasp, S., Pritchard, M. S. & Gentine, P. Deep learning to represent subgrid processes in climate models. Proc. Natl Acad. Sci. USA 115, 9684–9689 (2018).

    Article  Google Scholar 

  163. Zhu, Y. et al. Physics-informed deep-learning parameterization of ocean vertical mixing improves climate simulations. Natl Sci. Rev. 9, nwac044 (2022).

    Article  Google Scholar 

  164. Koppa, A., Rains, D., Hulsman, P., Poyatos, R. & Miralles, D. G. A deep learning-based hybrid model of global terrestrial evaporation. Nat. Commun. 13, 1912 (2022).

    Article  Google Scholar 

  165. Liu, B. et al. Physics-guided long short-term memory network for streamflow and flood simulations in the Lancang–Mekong river basin. Water 14, 1429 (2022).

    Article  Google Scholar 

  166. Li, D., Marshall, L., Liang, Z., Sharma, A. & Zhou, Y. Bayesian LSTM with stochastic variational inference for estimating model uncertainty in process-based hydrological models. Water Resour. Res. 57, e2021WR029772 (2021).

    Article  Google Scholar 

  167. Frame, J. M. et al. Post-processing the national water model with long short-term memory networks for streamflow predictions and model diagnostics. J. Am. Water Resour. Assoc. 57, 885–905 (2021).

    Article  Google Scholar 

  168. Sun, A. Y., Jiang, P., Yang, Z.-L., Xie, Y. & Chen, X. A graph neural network (GNN) approach to basin-scale river network learning: the role of physics-based connectivity and data fusion. Hydrol. Earth Syst. Sci. Discuss. 26, 5163–5184 (2022).

    Article  Google Scholar 

  169. Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowl. Syst. 06, 107–116 (1998).

    Article  Google Scholar 

  170. Hochreiter, S., Bengio, Y., Frasconi, P. & Jürgen S. in A Field Guide to Dynamical Recurrent Neural Networks (eds Kremer, S. C. & Kolen, J. F.) 237–244 (IEEE, 2001).

  171. Kochkov, D. et al. Machine learning–accelerated computational fluid dynamics. Proc. Natl Acad. Sci. USA 118, e2101784118 (2021).

    Article  Google Scholar 

  172. Fang, K., Kifer, D., Lawson, K. & Shen, C. Evaluating the potential and challenges of an uncertainty quantification method for long short-term memory models for soil moisture predictions. Water Resour. Res. 56, e2020WR028095 (2020).

    Article  Google Scholar 

  173. Tabas, S. S. & Samadi, S. Variational Bayesian dropout with a Gaussian prior for recurrent neural networks application in rainfall–runoff modeling. Environ. Res. Lett. 17, 065012 (2022).

    Article  Google Scholar 

  174. Krapu, C. & Borsuk, M. A differentiable hydrology approach for modeling with time-varying parameters. Water Resour. Res. 58, e2021WR031377 (2022).

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful for the discussion at the HydroML symposium, University Park, PA, May 2022, https://bit.ly/3g3DQNX, sponsored by National Science Foundation EAR #2015680 and Penn State Institute for Computational and Data Sciences. C.S., Y.S. and T.B. were supported by National Science Foundation Award EAR-2221880, Office of Science, US Department of Energy under award DE-SC0016605, and Cooperative Institute for Research to Operations in Hydrology (CIROH), award number A22-0307-S003, respectively. P.G. acknowledges funding from the National Science Foundational Science and Technology Center, Learning the Earth with Artificial intelligence and Physics (LEAP), award #2019625, and USMILE European Research Council grant. M. Wernimont at the US Geological Survey (USGS) greatly improved the presentation of Figs. 1 and 2. A.P.A. was supported by the USGS Water Mission Area, Water Availability and Use Science Program. Any use of trade, firm or product names is for descriptive purposes only and does not imply endorsement by the US Government.

Author information

Authors and Affiliations

  1. Civil and Environmental Engineering, The Pennsylvania State University, University Park, PA, USA

    Chaopeng Shen, Li Li, Xiaofeng Liu, Dapeng Feng, Doaa Aboelyazeed, Farshid Rahmani, Yalan Song, Tadd Bindas & Kathryn Lawson

  2. US Geological Survey, Reston, VA, USA

    Alison P. Appling

  3. National Science Foundation Science and Technology Center for Learning the Earth with Artificial Intelligence and Physics (LEAP), Columbia University, New York, NY, USA

    Pierre Gentine

  4. Energy Geoscience Divisions, Earth and Environmental Sciences Area, Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    Toshiyuki Bandai

  5. Hydrology and Atmospheric Sciences, The University of Arizona, Tucson, AZ, USA

    Hoshin Gupta

  6. Civil and Environmental Engineering, University of Illinois, Urbana Champaign, IL, USA

    Alexandre Tartakovsky & Praveen Kumar

  7. Eawag: Swiss Federal Institute of Aquatic Science and Technology, Dübendorf, Switzerland

    Marco Baity-Jesi, Fabrizio Fenicia & Marvin Höge

  8. Computer Science and Engineering, The Pennsylvania State University, University Park, PA, USA

    Daniel Kifer

  9. Department of Natural Resources and the Environment, University of Connecticut, Storrs, CT, USA

    Wei Ren

  10. Southern University of Science and Technology, Shenzhen, Guangdong Province, China

    Yi Zheng

  11. Department of Environmental Health and Engineering, Johns Hopkins University, Baltimore, MD, USA

    Ciaran J. Harman

  12. Global Institute for Water Security, University of Saskatchewan, Canmore, Alberta, Canada

    Martyn Clark

  13. US Army Engineer Research and Development Center, Vicksburg, MS, USA

    Matthew Farthing

  14. Prairie Research Institute, University of Illinois, Urbana Champaign, IL, USA

    Praveen Kumar

  15. Physical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

    Hylke E. Beck

  16. Climate and Ecosystem Sciences Divisions, Earth and Environmental Sciences Area, Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    Dipankar Dwivedi

  17. Department of Earth System Science, Stanford University, Stanford, CA, USA

    Kuai Fang

  18. Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology, Cambridge, MA, USA

    Chris Rackauckas

  19. Department of Biological and Agricultural Engineering, Texas A&M University, College Station, TX, USA

    Binayak Mohanty

  20. Civil and Environmental Engineering, University of Nebraska-Lincoln, Lincoln, NE, USA

    Tirthankar Roy

  21. Earth and Environmental Sciences Division, Los Alamos National Laboratory, New Mexico, NM, USA

    Chonggang Xu

Authors
  1. Chaopeng Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alison P. Appling
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pierre Gentine
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Toshiyuki Bandai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hoshin Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alexandre Tartakovsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Marco Baity-Jesi
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Fabrizio Fenicia
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Daniel Kifer
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Li Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Xiaofeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Wei Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Yi Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Ciaran J. Harman
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Martyn Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Matthew Farthing
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Dapeng Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Praveen Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Doaa Aboelyazeed
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Farshid Rahmani
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Yalan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Hylke E. Beck
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Tadd Bindas
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Dipankar Dwivedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Kuai Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Marvin Höge
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Chris Rackauckas
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Binayak Mohanty
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. Tirthankar Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  30. Chonggang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  31. Kathryn Lawson
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.S. researched data for the article. C.S., A.P.A., P.G., T.B., H.G., Y.Z., A.T., M.B.-J., F.F., D.K., L.L., X.L. and W.R. contributed substantially to discussion of the content. C.S. wrote the initial article. All authors reviewed and/or edited the manuscript before submission.

Corresponding author

Correspondence to Chaopeng Shen.

Ethics declarations

Competing interests

K.L. and C.S. have financial interests in HydroSapient, Inc.

Peer review

Peer review information

Nature Reviews Earth & Environment thanks A. McGovern, G. Varando and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shen, C., Appling, A.P., Gentine, P. et al. Differentiable modelling to unify machine learning and physical models for geosciences. Nat Rev Earth Environ 4, 552–567 (2023). https://doi.org/10.1038/s43017-023-00450-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s43017-023-00450-9

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing