Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Physics-enhanced deep surrogates for partial differential equations

Nature Machine Intelligence volume 5, pages 1458–1465 (2023)Cite this article

Subjects

Abstract

Many physics and engineering applications demand partial differential equations (PDE) property evaluations that are traditionally computed with resource-intensive high-fidelity numerical solvers. Data-driven surrogate models provide an efficient alternative but come with a substantial cost of training. Emerging applications would benefit from surrogates with an improved accuracy–cost tradeoff when studied at scale. Here we present a ‘physics-enhanced deep-surrogate’ (PEDS) approach towards developing fast surrogate models for complex physical systems, which is described by PDEs. Specifically, a combination of a low-fidelity, explainable physics simulator and a neural network generator is proposed, which is trained end-to-end to globally match the output of an expensive high-fidelity numerical solver. Experiments on three exemplar test cases, diffusion, reaction–diffusion and electromagnetic scattering models, show that a PEDS surrogate can be up to three times more accurate than an ensemble of feedforward neural networks with limited data (approximately 103 training points), and reduces the training data need by at least a factor of 100 to achieve a target error of 5%. Experiments reveal that PEDS provides a general, data-driven strategy to bridge the gap between a vast array of simplified physical models with corresponding brute-force numerical solvers modelling complex systems, offering accuracy, speed and data efficiency, as well as physical insights into the process.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Diagram of PEDS.
Fig. 2: Example geometry and the corresponding solutions for the diffusion and the reaction–diffusion equations.
Fig. 3: PEDS’s fractional error on the test set outperforms that of the NN baseline.

Similar content being viewed by others

Data availability

Datasets to reproduce the findings for the five surrogate models with about 1,000 data points are available on GitHub52. The full dataset and the raw data to create the figure plots in the main text and Supplementary Information can be found on Zenodo53.

Code availability

The code used for these findings can be found on GitHub52.

References

  1. Baker, N. et al. Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence. Technical Report (USDOE Office of Science, 2019).

  2. Benner, P., Gugercin, S. & Willcox, K. A survey of projection-based model reduction methods for parametric dynamical systems. SIAM Rev. 57, 483–531 (2015).

    Article  MathSciNet  Google Scholar 

  3. Willard, J., Jia, X., Xu, S., Steinbach, M. & Kumar, V. Integrating scientific knowledge with machine learning for engineering and environmental systems. ACM Comput. Surv. 55, 1–37 (2022).

  4. Hoffmann, J. et al. Machine learning in a data-limited regime: augmenting experiments with synthetic data uncovers order in crumpled sheets. Sci. Adv. 5, eaau6792 (2019).

    Article  Google Scholar 

  5. Pant, P., Doshi, R., Bahl, P. & Barati Farimani, A. Deep learning for reduced order modelling and efficient temporal evolution of fluid simulations. Phys. Fluids 33, 107101 (2021).

    Article  Google Scholar 

  6. Pestourie, R. et al. Inverse design of large-area metasurfaces. Opt. Expr. 26, 33732–33747 (2018).

    Article  Google Scholar 

  7. Boyd, J. P. Chebyshev and Fourier Spectral Methods 2nd edn (Dover Publications, Inc., 2001).

  8. Pestourie, R., Mroueh, Y., Nguyen, T. V., Das, P. & Johnson, S. G. Active learning of deep surrogates for PDEs: application to metasurface design. npj Comput. Mat. 6, 164 (2020).

    Article  Google Scholar 

  9. Lu, L., Pestourie, R., Johnson, S. G. & Romano, G. Multifidelity deep neural operators for efficient learning of partial differential equations with application to fast inverse design of nanoscale heat transport. Phys. Rev. Research 4, 023210 (2022).

  10. Pestourie, R. Assume Your Neighbor Is Your Equal: Inverse Design in Nanophotonics. PhD thesis, Harvard Univ. (2020).

  11. Bayati, E. et al. Inverse designed extended depth of focus meta-optics for broadband imaging in the visible. Nanophotonics 11, 2531–2540 (2021).

  12. Li, Z. et al. Inverse design enables large-scale high-performance meta-optics reshaping virtual reality. Nat. Commun. 13, 2409 (2022).

  13. Potton, R. J. Reciprocity in optics. Rep. Progr. Phys. 67, 717 (2004).

    Article  Google Scholar 

  14. Li, B. et al. Trustworthy AI: from principles to practices. ACM Comput. Surv. 55, 1–46 (2023).

  15. Oskooi, A. F., Kottke, C. & Johnson, S. G. Accurate finite-difference time-domain simulation of anisotropic media by subpixel smoothing. Optics Lett. 34, 2778–2780 (2009).

    Article  Google Scholar 

  16. Ferziger, J. H., Perić, M. & Street, R. L. Computational Methods for Fluid Dynamics Vol. 3 (Springer, 2002).

  17. Romano, G. OpenBTE: a solver for ab-initio phonon transport in multidimensional structures. Preprint at https://arxiv.org/abs/2106.02764 (2021).

  18. Cranmer, M. et al. Discovering symbolic models from deep learning with inductive biases. Adv. Neural Info. Proc. Syst. 33, 17429–17442 (2020).

    Google Scholar 

  19. Rackauckas, C. et al. Universal differential equations for scientific machine learning. Preprint at https://arxiv.org/abs/2001.04385 (2020).

  20. Hoerl, A. E. & Kennard, R. W. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12, 55–67 (1970).

    Article  Google Scholar 

  21. Karniadakis, G. E. et al. Physics-informed machine learning. Nat. Rev. Phys. 3, 422–440 (2021).

    Article  Google Scholar 

  22. Lu, L. et al. Physics-informed neural networks with hard constraints for inverse design. SIAM J. Sci. Comput. 43, B1105–B1132 (2021).

  23. Shin, Y., Darbon, J. & Karniadakis, G. E. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs. Commun. Comput. Phys. 28, 2042–2074 (2020).

  24. Kochkov, D. et al. Machine learning–accelerated computational fluid dynamics. Proc. Natl Acad. Sci. USA 118, e2101784118 (2021).

    Article  MathSciNet  Google Scholar 

  25. Lu, L., Jin, P., Pang, G., Zhang, Z. & Karniadakis, G. E. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nat. Mach. Intell. 3, 218–229 (2021).

    Article  Google Scholar 

  26. Li, Z. et al. Fourier neural operator for parametric partial differential equations. Preprint at https://arxiv.org/abs/2010.08895 (2020).

  27. Koziel, S., Cheng, Q. S. & Bandler, J. W. Space mapping. IEEE Microw. Mag. 9, 105–122 (2008).

    Article  Google Scholar 

  28. Bakr, M. H., Bandler, J. W., Ismail, M. A., Rayas-Sánchez, J. E. & Zhang, Q.-J. Neural space-mapping optimization for em-based design. IEEE Trans. Microw. Theory Tech. 48, 2307–2315 (2000).

    Article  Google Scholar 

  29. Feng, F. et al. Coarse-and fine-mesh space mapping for em optimization incorporating mesh deformation. IEEE Microw. Wirel. Compon. Lett. 29, 510–512 (2019).

    Article  Google Scholar 

  30. Lu, L. et al. Extraction of mechanical properties of materials through deep learning from instrumented indentation. Proc. Natl Acad. Sci. USA 117, 7052–7062 (2020).

    Article  Google Scholar 

  31. Koziel, S., Bandler, J. W. & Madsen, K. A space-mapping framework for engineering optimization-theory and implementation. IEEE Trans. Microw. Theory Tech. 54, 3721–3730 (2006).

    Article  Google Scholar 

  32. Levine, M. & Stuart, A. A framework for machine learning of model error in dynamical systems. Commun. Am. Math. Soc. 2, 283–344 (2022).

    Article  MathSciNet  Google Scholar 

  33. Ren, P. et al. Physics-informed deep super-resolution for spatiotemporal data. Preprint at https://arxiv.org/abs/2208.01462 (2022).

  34. Drygala, C., Winhart, B., di Mare, F. & Gottschalk, H. Generative modeling of turbulence. Phys. Fluids 34, 035114 (2022).

    Article  Google Scholar 

  35. Geneva, N. & Zabaras, N. Multi-fidelity generative deep learning turbulent flows. Found. Data Sci. 2, 391–428 (2020).

  36. Hesthaven, J. S., Pagliantini, C. & Rozza, G. Reduced basis methods for time-dependent problems. Acta Numerica 31, 265–345 (2022).

    Article  MathSciNet  Google Scholar 

  37. Weinberg, S. The Quantum Theory of Fields Vol. 2 (Cambridge Univ. Press, 1995).

  38. Hou, T. Y., Hwang, F.-N., Liu, P. & Yao, C.-C. An iteratively adaptive multi-scale finite element method for elliptic PDEs with rough coefficients. J. Comput. Phys. 336, 375–400 (2017).

    Article  MathSciNet  Google Scholar 

  39. Molesky, S. et al. Inverse design in nanophotonics. Nat. Photon. 12, 659–670 (2018).

    Article  Google Scholar 

  40. Pérez-Arancibia, C., Pestourie, R. & Johnson, S. G. Sideways adiabaticity: beyond ray optics for slowly varying metasurfaces. Optics Expr. 26, 30202–30230 (2018).

    Article  Google Scholar 

  41. Jin, J.-M. The Finite Element Method in Electromagnetics (John Wiley & Sons, 2015).

  42. Svanberg, K. A class of globally convergent optimization methods based on conservative convex separable approximations. SIAM J. Optim.12, 555–573 (2002).

    Article  MathSciNet  Google Scholar 

  43. Liu, D. C. & Nocedal, J. On the limited memory BFGS method for large scale optimization. Math. Program. 45, 503–528 (1989).

    Article  MathSciNet  Google Scholar 

  44. Huber, P. J. in Breakthroughs in Statistics: Methodology and Distribution (eds Kotz, S. & Johnson, N. L.) 492–518 (Springer, 1992).

  45. Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proc. 31st International Conference on Neural Information Processing Systems (eds von Luxburg, U. et al.) 6405–6416 (Curran Associates Inc., 2017).

  46. Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).

  47. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  48. Innes, M. Flux: elegant machine learning with Julia. J. Open Source Softw. 3, 602 (2018).

  49. Innes, M. Don’t unroll adjoint: differentiating SSA-form programs. Preprint at https://arxiv.org/abs/1810.07951 (2018).

  50. Champagne II, N. J., Berryman, J. G. & Buettner, H. M. FDFD: a 3D finite-difference frequency-domain code for electromagnetic induction tomography. J. Comput. Phys. 170, 830–848 (2001).

    Article  MathSciNet  Google Scholar 

  51. Sacks, Z. S., Kingsland, D. M., Lee, R. & Lee, J.-F. A perfectly matched anisotropic absorber for use as an absorbing boundary condition. IEEE Trans. Antennas Propag. 43, 1460–1463 (1995).

    Article  Google Scholar 

  52. Pestourie, R. & Das, P. payel79/peds: publish. Zenodo https://doi.org/10.5281/zenodo.8342595 (2023).

  53. Pestourie, R. & Das, P. PEDS datasets and figure data. Zenodo https://doi.org/10.5281/zenodo.10011958 (2023).

Download references

Acknowledgements

R.P. was supported by the US Army Research Office through the Institute for Soldier Nanotechnologies (grant award no. W911NF-18-2-0048) and the MIT-IBM Watson AI Laboratory. We thank M. Dost for her suggestions in proof reading.

Author information

Authors and Affiliations

  1. School of Computational Science and Engineering, Georgia Institute of Technology, Atlanta, GA, USA

    Raphaël Pestourie

  2. IBM Research AI, IBM Thomas J Watson Research Center, Yorktown Heights, NY, USA

    Youssef Mroueh & Payel Das

  3. MIT-IBM Watson AI Laboratory, Cambridge, MA, USA

    Youssef Mroueh

  4. Massachusetts Institute of Technology, Cambridge, MA, USA

    Chris Rackauckas & Steven G. Johnson

Authors
  1. Raphaël Pestourie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Youssef Mroueh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chris Rackauckas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Payel Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Steven G. Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

R.P., Y.M., C.R., P.D. and S.G.J. designed the study, contributed to the machine-learning approach and analysed results. R.P. led the code development, software implementation and numerical experiments. R.P. and S.G.J. were responsible for the physical ideas and interpretation. All authors contributed to the algorithmic ideas and writing.

Corresponding authors

Correspondence to Raphaël Pestourie or Payel Das.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Bamdad Hosseini and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Eighteen sections of Supplementary Discussion, Figs. 1–4 and Tables 1–4.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pestourie, R., Mroueh, Y., Rackauckas, C. et al. Physics-enhanced deep surrogates for partial differential equations. Nat Mach Intell 5, 1458–1465 (2023). https://doi.org/10.1038/s42256-023-00761-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-023-00761-y

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics