Predictive simulations of complex systems are essential for applications ranging from weather forecasting to drug design. The veracity of these predictions hinges on their capacity to capture effective system dynamics. Massively parallel simulations predict the system dynamics by resolving all spatiotemporal scales, often at a cost that prevents experimentation, while their findings may not allow for generalization. On the other hand, reduced-order models are fast but limited by the frequently adopted linearization of the system dynamics and the utilization of heuristic closures. Here we present a novel systematic framework that bridges large-scale simulations and reduced-order models to learn the effective dynamics of diverse, complex systems. The framework forms algorithmic alloys between nonlinear machine learning algorithms and the equation-free approach for modelling complex systems. Learning the effective dynamics deploys autoencoders to formulate a mapping between fine- and coarse-grained representations and evolves the latent space dynamics using recurrent neural networks. The algorithm is validated on benchmark problems, and we find that it outperforms state-of-the-art reduced-order models in terms of predictability, and large-scale simulations in terms of cost. Learning the effective dynamics is applicable to systems ranging from chemistry to fluid mechanics and reduces the computational effort by up to two orders of magnitude while maintaining the prediction accuracy of the full system dynamics. We argue that learning the effective dynamics provides a potent novel modality for accurately predicting complex systems.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Numerical Bifurcation Analysis of PDEs From Lattice Boltzmann Model Simulations: a Parsimonious Machine Learning Approach
Journal of Scientific Computing Open Access 24 June 2022
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
All the data analysed in this paper were produced with open-source software described in the code availability statement. Reference data and the scripts used to produce the data figures, as well as instructions to launch training and inference (evaluation of trained models) scripts for LED, are available on the GitHub repository: https://github.com/cselab/LED.
Simulations of the KS equation have been performed with a spectral fourth-order solver for stiff PDEs developed in Python. Simulation of the FHN equation has been performed with an LB method developed in Python. The LED software is implemented in Python, utilizing the PyTorch library for the neural networks. All codes are made readily available in the GitHub repository: https://github.com/cselab/LED. Direct numerical simulations were performed with the flow solver CubismUP 2D: https://github.com/cselab/CubismUP_2D.
Rackovsky, S. & Scheraga, H. A. The structure of protein dynamic space. Proc. Natl Acad. Sci. USA 117, 19938–19942 (2020).
Gilmour, D., Rembold, M. & Leptin, M. From morphogen to morphogenesis and back. Nature 541, 311–320 (2017).
Robinson, P. A., Rennie, C. J., Rowe, D. L., O’Connor, S. C. & Gordon, E. Multiscale brain modelling. Philos. Trans. R. Soc. B 360, 1043–1050 (2005).
Council, N. R. A National Strategy for Advancing Climate Modeling (National Academies Press, 2012).
Mahadevan, A. The impact of submesoscale physics on primary productivity of plankton. Annu. Rev. Mar. Sci. 8, 161–184 (2016).
Bellomo, N. & Dogbe, C. On the modeling of traffic and crowds: a survey of models, speculations, and perspectives. SIAM Rev. 53, 409–463 (2011).
Lee, E. H., Hsin, J., Sotomayor, M., Comellas, G. & Schulten, K. Discovery through the computational microscope. Structure 17, 1295–1306 (2009).
Springel, V. et al. Simulations of the formation, evolution and clustering of galaxies and quasars. Nature 435, 629–636 (2005).
Car, R. & Parrinello, M. Unified approach for molecular dynamics and density-functional theory. Phys. Rev. Lett. 55, 2471–2474 (1985).
Kevrekidis, I. G. et al. Equation-free, coarse-grained multiscale computation: enabling microscopic simulators to perform system-level analysis. Commun. Math. Sci. 1, 715–762 (2003).
Weinan, E. & Engquist, B. et al. The heterogeneous multiscale methods. Commun. Math. Sci. 1, 87–132 (2003).
Kevrekidis, I. G., Gear, C. W. & Hummer, G. Equation-free: the computer-aided analysis of complex multiscale systems. AIChE J. 50, 1346–1355 (2004).
Laing, C. R., Frewen, T. & Kevrekidis, I. G. Reduced models for binocular rivalry. J. Comput. Neurosci. 28, 459–476 (2010).
Bar-Sinai, Y., Hoyer, S., Hickey, J. & Brenner, M. P. Learning data-driven discretizations for partial differential equations. Proc. Natl Acad. Sci. USA 116, 15344–15349 (2019).
Weinan, E., Li, X. & Vanden-Eijnden, E. in Multiscale Modelling and Simulation (eds Attinger, S. & Koumoutsakos, P.) 3–21 (Springer, 2004).
Weinan, E., Engquist, B., Li, X., Ren, W. & Vanden-Eijnden, E. Heterogeneous multiscale methods: a review. Commun. Comput. Phys. 2, 367–450 (2007).
Tao, M., Owhadi, H. & Marsden, J. E. Nonintrusive and structure preserving multiscale integration of stiff ODEs, SDEs, and Hamiltonian systems with hidden slow dynamics via flow averaging. Multiscale Model. Simul. 8, 1269–1324 (2010).
Linot, A. J. & Graham, M. D. Deep learning to discover and predict dynamics on an inertial manifold. Phys. Rev. E 101, 062209 (2020).
Robinson, J. C. Inertial manifolds for the Kuramoto–Sivashinsky equation. Phys. Lett. A 184, 190–193 (1994).
Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021).
Brunton, S. L., Noack, B. R. & Koumoutsakos, P. Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52, 477–508 (2019).
Lusch, B., Kutz, J. N. & Brunton, S. L. Deep learning for universal linear embeddings of nonlinear dynamics. Nat. Commun. 9, 4950 (2018).
Geneva, N. & Zabaras, N. Modeling the dynamics of PDE systems with physics-constrained deep auto-regressive networks. J. Comput. Phys. 403, 109056 (2020).
Milano, M. & Koumoutsakos, P. Neural network modeling for near wall turbulent flow. J. Comput. Phys. 182, 1–26 (2002).
Wehmeyer, C. & Noé, F. Time-lagged autoencoders: deep learning of slow collective variables for molecular kinetics. J. Chem. Phys. 148, 241703 (2018).
Bhatia, H. et al. Machine-learning-based dynamic-importance sampling for adaptive multiscale simulations. Nat. Mach. Intell. 3, 401–409 (2021).
Chung, J. et al. A recurrent latent variable model for sequential data. Adv. Neural Inf. Process. Syst. 28, 2980–2988 (2015).
Vlachas, P. R. et al. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Netw. 126, 191–217 (2020).
Gonzalez, F. J. & Balajewicz, M. Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems. Preprint at https://arxiv.org/abs/1808.01346 (2018).
Maulik, R., Lusch, B. & Balaprakash, P. Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders. Phys. Fluids 33, 037106 (2021).
Hasegawa, K., Fukami, K., Murata, T. & Fukagata, K. Machine-learning-based reduced-order modeling for unsteady flows around bluff bodies of various shapes. Theor. Comput. Fluid Dyn. 34, 367–383 (2020).
Lee, S., Kooshkbaghi, M., Spiliotis, K., Siettos, C. I. & Kevrekidis, I. G. Coarse-scale PDEs from fine-scale observations via machine learning. Chaos 30, 013141 (2020).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Bishop, C. M. Mixture Density Networks Technical Report NCRG/97/004 (Neural Computing Research Group, Aston University, 1994).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Werbos, P. J. Generalization of backpropagation with application to a recurrent gas market model. Neural Netw. 1, 339–356 (1988).
Hernández, C. X., Wayment-Steele, H. K., Sultan, M. M., Husic, B. E. & Pande, V. S. Variational encoding of complex dynamics. Phys. Rev. E 97, 062412 (2018).
Sultan, M. M., Wayment-Steele, H. K. & Pande, V. S. Transferable neural networks for enhanced sampling of protein dynamics. J. Chem. Theory Comput. 14, 1887–1894 (2018).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proc. (eds Bengio, Y. & LeCun, Y.) 1-15 (2015).
Vlachas, P. R., Zavadlav, J., Praprotnik, M. & Koumoutsakos, P. Accelerated simulations of molecular systems through learning of their effective dynamics. J. Chem. Theory Comput. 18, 538–549 (2021).
FitzHugh, R. Impulses and physiological states in theoretical models of nerve membrane. Biophys. J. 1, 445 (1961).
Nagumo, J., Arimoto, S. & Yoshizawa, S. An active pulse transmission line simulating nerve axon. Proc. IRE 50, 2061–2070 (1962).
Karlin, I. V., Ansumali, S., Frouzakis, C. E. & Chikatamarla, S. S. Elements of the lattice Boltzmann method I: Linear advection equation. Commun. Comput. Phys. 1, 616–655 (2006).
Pathak, J., Hunt, B., Girvan, M., Lu, Z. & Ott, E. Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach. Phys. Rev. Lett.120, 024102 (2018).
Brunton, S. L., Proctor, J. L. & Kutz, J. N. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl Acad. Sci. USA 113, 3932–3937 (2016).
Kuramoto, Y. Diffusion-induced chaos in reaction systems. Prog. Theor. Phys. Suppl. 64, 346–367 (1978).
Sivashinsky, G. I. Nonlinear analysis of hydrodynamic instability in laminar flames—I. Derivation of basic equations. Acta Astronaut. 4, 1177–1206 (1977).
Cvitanović, P., Davidchack, R. L. & Siminos, E. On the state space geometry of the Kuramoto–Sivashinsky flow in a periodic domain. SIAM J. Appl. Dyn. Syst. 9, 1–33 (2010).
Kassam, A. & Trefethen, L. Fourth-order time-stepping for stiff PDEs. SIAM J. Sci. Comput. 26, 1214–1233 (2005).
Zdravkovich, M. Flow Around Circular Cylinders Volume 1: Fundamentals (Oxford University Press, 1997).
Rossinelli, D. et al. MRAG-I2D: multi-resolution adapted grids for remeshed vortex methods on multicore architectures. J. Comput. Phys. 288, 1–18 (2015).
Bost, C., Cottet, G.-H. & Maitre, E. Convergence analysis of a penalization method for the three-dimensional motion of a rigid body in an incompressible viscous fluid. SIAM J. Numer. Anal. 48, 1313–1337 (2010).
Taira, K. et al. Modal analysis of fluid flows: applications and outlook. AIAA J. 58, 998–1022 (2020).
We thank N.Kallikounis (ETH Zurich) for helpful discussions on the LB method, P. Weber and M. Chatzimanolakis (ETH Zurich) for help with the simulations of the flow past a cylinder and Y. Kevrekidis (Johns Hopkins University) and K. Spiliotis (University of Rostock) for providing code to reproduce data for the FHN equation. We acknowledge the infrastructure and support of CSCS, providing the necessary computational resources under project s929.
The authors declare no competing interests.
Peer review information
Nature Machine Intelligence thanks Harsh Bhatia and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Vlachas, P.R., Arampatzis, G., Uhler, C. et al. Multiscale simulations of complex systems by learning their effective dynamics. Nat Mach Intell 4, 359–366 (2022). https://doi.org/10.1038/s42256-022-00464-w
This article is cited by
Journal of Mathematical Biology (2023)
Inferring the Physics of Structural Evolution of Multicomponent Polymers via Machine-Learning-Accelerated Method
Chinese Journal of Polymer Science (2023)
Nature Machine Intelligence (2022)
Numerical Bifurcation Analysis of PDEs From Lattice Boltzmann Model Simulations: a Parsimonious Machine Learning Approach
Journal of Scientific Computing (2022)