Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations

Nature Machine Intelligence (2024)Cite this article

Subjects

A preprint version of the article is available at arXiv.

Abstract

One of the most promising applications of machine learning in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of machine-learning-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE-solving literature. Out of all of the articles that report using ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) make a comparison with a weak baseline. Second, we find evidence that reporting biases are widespread, especially outcome reporting and publication biases. We conclude that ML-for-PDE-solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to under-reporting of negative results. To a large extent, these issues seem to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms to reduce perverse incentives for doing so.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The cumulative effects of weak baselines and reporting biases on samples A and B.

Similar content being viewed by others

Data availability

The lists of authors and articles generated during the systematic review and the categorizations of every article in the random samples are publicly available at https://doi.org/10.17605/OSF.IO/GQ5B3 (ref. 124).

Code availability

The code required to reproduce the results in Table 2 is available on GitHub at https://github.com/nickmcgreivy/WeakBaselinesMLPDE/ (ref. 125), and Code Ocean at https://codeocean.com/capsule/9605539/tree/v1 (ref. 126) and https://codeocean.com/capsule/0799002/tree/v1 (ref. 127).

References

  1. Randall, D. & Welser, C. The Irreproducibility Crisis of Modern Science: Causes, Consequences, and the Road to Reform (National Association of Scholars, 2018).

  2. Ritchie, S. Science fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (Vintage, 2020).

  3. Munafò, M. R. et al. A manifesto for reproducible science. Nat. Human Behav. 1, 1–9 (2017).

    Article  Google Scholar 

  4. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).

    Article  Google Scholar 

  5. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).

    Article  Google Scholar 

  6. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712 (2011).

    Article  Google Scholar 

  7. Begley, C. G. & Ellis, L. M. Raise standards for preclinical cancer research. Nature 483, 531–533 (2012).

    Article  Google Scholar 

  8. Gelman, A. & Loken, E. The Garden of Forking Paths: Why Multiple Comparisons Can be a Problem, Even When There is no ‘Fishing Expedition’ or ‘p-Hacking’ and the Research Hypothesis was Posited Ahead of Time Vol. 348, 1–17 (Department of Statistics, 2013).

  9. Baker, M. Reproducibility crisis. Nature 533, 353–66 (2016).

    Google Scholar 

  10. Karagiorgi, G., Kasieczka, G., Kravitz, S., Nachman, B. & Shih, D. Machine learning in the search for new fundamental physics. Nat. Rev. Phys. 4, 399–412 (2022).

    Article  Google Scholar 

  11. Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. H. M. & Ahsan, M. J. Machine learning in drug discovery: a review. Artif. Intell. Rev. 55, 1947–1999 (2022).

    Article  Google Scholar 

  12. Mater, A. C. & Coote, M. L. Deep learning in chemistry. J. Chem. Inform. Model. 59, 2545–2559 (2019).

    Article  Google Scholar 

  13. Carleo, G. et al. Machine learning and the physical sciences. Rev. Modern Phys. 91, 045002 (2019).

    Article  Google Scholar 

  14. Rajkomar, A., Dean, J. & Kohane, I. Machine learning in medicine. New England J. Med. 380, 1347–1358 (2019).

    Article  Google Scholar 

  15. Grimmer, J., Roberts, M. E. & Stewart, B. M. Machine learning for social science: an agnostic approach. Ann. Rev. Pol. Sci. 24, 395–419 (2021).

    Article  Google Scholar 

  16. Jumper, J. et al. Highly accurate protein structure prediction with alphafold. Nature 596, 583–589 (2021).

    Article  Google Scholar 

  17. Hutson, M. Artificial intelligence faces reproducibility crisis. Science 359, 725–726 (2018).

    Article  Google Scholar 

  18. Gundersen, O.E., Coakley, K., Kirkpatrick, C. & Gil, Y. Sources of irreproducibility in machine learning: a review. Preprint at https://arxiv.org/abs/2204.07610 (2022).

  19. Sculley, D., Snoek, J., Wiltschko, A. & Rahimi, A. Winner’s Curse? On Pace, Progress, and Empirical Rigor (ICLR, 2018).

  20. Armstrong, T. G., Moffat, A., Webber, W. & Zobel, J. Improvements that don’t add up: ad-hoc retrieval results since 1998. In Proc. 18th ACM Conference on Information and Knowledge Management 601–610 (ACM, 2009).

  21. Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4, 100804 (2023).

  22. Kapoor, S. et al. Reforms: reporting standards for machine learning based science. Preprint at https://arxiv.org/abs/2308.07832 (2023).

  23. DeMasi, O., Kording, K. & Recht, B. Meaningless comparisons lead to false optimism in medical machine learning. PloS ONE 12, e0184604 (2017).

    Article  Google Scholar 

  24. Roberts, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 3, 199–217 (2021).

    Article  Google Scholar 

  25. Wynants, L. et al. Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal. BMJ 369, m1328 (2020).

  26. Whalen, S., Schreiber, J., Noble, W. S. & Pollard, K. S. Navigating the pitfalls of applying machine learning in genomics. Nat. Rev. Genet. 23, 169–181 (2022).

    Article  Google Scholar 

  27. Artrith, N. et al. Best practices in machine learning for chemistry. Nat. Chem. 13, 505–508 (2021).

    Article  Google Scholar 

  28. Thuerey, N. et al. Physics-based deep learning. Preprint at https://arxiv.org/abs/2109.05237 (2021).

  29. Brunton, S. L. & Kutz, J. N. Promising directions of machine learning for partial differential equations. Nat. Comput. Sci. 4, 483–494 (2024).

  30. Vinuesa, R. & Brunton, S. L. Enhancing computational fluid dynamics with machine learning. Nat. Comput. Sci. 2, 358–366 (2022).

    Article  Google Scholar 

  31. Karniadakis, G. E. et al. Physics-informed machine learning. Nat. Rev. Phys. 3, 422–440 (2021).

    Article  Google Scholar 

  32. Cuomo, S. et al. Scientific machine learning through physics–informed neural networks: where we are and what’s next. J. Sci. Comput. 92, 88 (2022).

    Article  MathSciNet  Google Scholar 

  33. Duraisamy, K., Iaccarino, G. & Xiao, H. Turbulence modeling in the age of data. Ann. Rev. Fluid Mech. 51, 357–377 (2019).

    Article  MathSciNet  Google Scholar 

  34. Durran, D. R. Numerical Methods for Wave Equations in Geophysical Fluid Dynamics Vol. 32 (Springer, 2013).

  35. LeVeque, R. J. Numerical Methods for Conservation Laws Vol. 214 (Birkhäuser, 1992).

  36. Mishra, S. A machine learning framework for data driven acceleration of computations of differential equations. Math. Eng. https://doi.org/10.3934/Mine.2018.1.118 (2018).

  37. Kochkov, D. et al. Machine learning–accelerated computational fluid dynamics. Proc. Natl Acad. Sci. USA 118, e2101784118 (2021).

    Article  MathSciNet  Google Scholar 

  38. Kadapa, C. Machine learning for computational science and engineering—a brief introduction and some critical questions. Preprint at https://arxiv.org/abs/2112.12054 (2021).

  39. Ross, A., Li, Z., Perezhogin, P., Fernandez-Granda, C. & Zanna, L. Benchmarking of machine learning ocean subgrid parameterizations in an idealized model. J. Adv. Model. Earth Syst. 15, e2022MS003258 (2023).

    Article  Google Scholar 

  40. Lippe, P., Veeling, B., Perdikaris, P., Turner, R. & Brandstetter, J. PDE-refiner: achieving accurate long rollouts with neural PDE solvers. In 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

  41. Vlachas, P. R. et al. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Netw. 126, 191–217 (2020).

    Article  Google Scholar 

  42. Raissi, M., Perdikaris, P. & Karniadakis, G. E. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019).

    Article  MathSciNet  Google Scholar 

  43. Grossmann, T. G., Komorowska, U. J., Latz, J. & Schönlieb, C.-B. Can physics-informed neural networks beat the finite element method? IMA J. Appl. Math. 89, 143–174 (2024).

  44. de la Mata, F. F., Gijón, A., Molina-Solana, M. & Gómez-Romero, J. Physics-informed neural networks for data-driven simulation: advantages, limitations, and opportunities. Phys. A 610, 128415 (2023).

    Article  MathSciNet  Google Scholar 

  45. Chuang, P.-Y. & Barba, L. A. Experience report of physics-informed neural networks in fluid simulations: pitfalls and frustration. Preprint at https://arxiv.org/abs/2205.14249 (2022).

  46. Chuang, P.-Y. & Barba, L. A. Predictive limitations of physics-informed neural networks in vortex shedding. Preprint at https://arxiv.org/abs/2306.00230 (2023).

  47. Wang, S., Yu, X. & Perdikaris, P. When and why pinns fail to train: a neural tangent kernel perspective. J. Comput. Phys. 449, 110768 (2022).

    Article  MathSciNet  Google Scholar 

  48. Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R. & Mahoney, M. W. Characterizing possible failure modes in physics-informed neural networks. In 35th Conference on Neural Information Processing Systems Vol. 34, 26548–26560 (NeurIPS 2021).

  49. Basir, S. & Senocak, I. Critical investigation of failure modes in physics-informed neural networks. In AiAA SCITECH 2022 Forum 2353 (ARC, 2022).

  50. Karnakov, P., Litvinov, S. & Koumoutsakos, P. Solving inverse problems in physics by optimizing a discrete loss: fast and accurate learning without neural networks. Proc. Natl Acad. Sci. Nexus 3, pgae005 (2024).

  51. Gundersen, O. E. The fundamental principles of reproducibility. Phil. Trans. R. Soc. A 379, 20200210 (2021).

    Article  Google Scholar 

  52. Aromataris, E. & Pearson, A. The systematic review: an overview. Am. J. Nursing 114, 53–58 (2014).

    Article  Google Scholar 

  53. Magiera, J., Ray, D., Hesthaven, J. S. & Rohde, C. Constraint-aware neural networks for riemann problems. J. Comput. Phys. 409, 109345 (2020).

    Article  MathSciNet  Google Scholar 

  54. Bezgin, D. A., Schmidt, S. J. & Adams, N. A. A data-driven physics-informed finite-volume scheme for nonclassical undercompressive shocks. J. Comput. Phys. 437, 110324 (2021).

    Article  MathSciNet  Google Scholar 

  55. Dresdner, G. et al. Learning to correct spectral methods for simulating turbulent flows. Transact. Mach. Learn Res. https://openreview.net/pdf?id=vIc8P32GRG (2023).

  56. Toshev, A. P., Galletti, G., Brandstetter, J., Adami, S. & Adams, N. A. E(3) equivariant graph neural networks for particle-based fluid mechanics. In ICLR 2023 Workshop on Physics for Machine Learning https://openreview.net/forum?id=5ByoWjLmUa (2023).

  57. Discacciati, N., Hesthaven, J. S. & Ray, D. Controlling oscillations in high-order discontinuous galerkin schemes using artificial viscosity tuned by neural networks. J. Comput. Phys. 409, 109304 (2020).

    Article  MathSciNet  Google Scholar 

  58. Dong, S. & Yang, J. On computing the hyperparameter of extreme learning machines: algorithm and application to computational pdes, and comparison with classical and high-order finite elements. J. Comput. Phys. 463, 111290 (2022).

    Article  MathSciNet  Google Scholar 

  59. Bar-Sinai, Y., Hoyer, S., Hickey, J. & Brenner, M. P. Learning data-driven discretizations for partial differential equations. Proc. Natl Acad. Sci. USA 116, 15344–15349 (2019).

    Article  MathSciNet  Google Scholar 

  60. Li, Z. et al. Fourier neural operator for parametric partial differential equations. In Proc. of ICLR 2021 https://openreview.net/pdf/53c47f849d1cd4d21b865caf7d774e07a5c42aa4.pdf (2021).

  61. Li, Z. et al. Physics-informed neural operator for learning partial differential equations. ACM/JMS J. Data Sci. https://doi.org/10.1145/3648506 (2024).

  62. Richards, G. C. & Onakpoya, I. J. in Catalogue of Bias (Catalogue of Bias, 2019); https://catalogofbias.org/biases/reporting-biases/

  63. Thornton, A. & Lee, P. Publication bias in meta-analysis: its causes and consequences. J. Clin. Epidemiol. 53, 207–216 (2000).

    Article  Google Scholar 

  64. Boutron, I. & Ravaud, P. Misrepresentation and distortion of research in biomedical literature. Proc. Natl Acad. Sci. USA 115, 2613–2619 (2018).

    Article  Google Scholar 

  65. Thomas, E. T. & Heneghan, C. Catalogue of bias: selective outcome reporting bias. BMJ Evid.-Based Med. 27, 370–372 (2022).

    Article  Google Scholar 

  66. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T. & Jennions, M. D. The extent and consequences of p-hacking in science. PLoS Biol. 13, e1002106 (2015).

    Article  Google Scholar 

  67. Saini, P. et al. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ 349, g6501 (2014).

  68. Altman, D. G., Moher, D. & Schulz, K. F. Harms of outcome switching in reports of randomised trials: consort perspective. BMJ 356, j396 (2017).

  69. Erasmus, A., Holman, B. & Ioannidis, J. P. A. Data-dredging bias. BMJ Evid.-Based Med. 27, 209–211 (2022).

    Article  Google Scholar 

  70. De Vries, Y. A. et al. The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychol. Med. 48, 2453–2455 (2018).

    Article  Google Scholar 

  71. Fanelli, D. ‘Positive’ results increase down the hierarchy of the sciences. PloS ONE 5, e10068 (2010).

    Article  Google Scholar 

  72. Wang, S., Wang, H. & Perdikaris, P. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Sci. Adv. 7, eabi8605 (2021).

    Article  Google Scholar 

  73. Um, K., Brand, R., Fei, Y. R., Holl, P. & Thuerey, N. Solver-in-the-loop: learning from differentiable physics to interact with iterative PDE-solvers. In 34th Conference on Neural Information Processing Systems 33, 6111–6122 (NeurIPS, 2020).

  74. Brandstetter, J., Worrall, D. & Welling, M. Message passing neural PDE solvers. In Proc. of ICLR 2022 https://openreview.net/pdf?id=vSix3HPYKSU (2022).

  75. Dong, S. & Li, Z. Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Comput. Methods Appl. Mech. Eng. 387, 114129 (2021).

    Article  MathSciNet  Google Scholar 

  76. Shang, Y., Wang, F. & Sun, J. Deep Petrov–Galerkin method for solving partial differential equations. Preperint at https://arxiv.org/abs/2201.12995 (2022).

  77. Li, Z., Huang, D. Z., Liu, B. & Anandkumar, A. Fourier neural operator with learned deformations for PDEs on general geometries. J. Mach. Learn. Res. https://www.jmlr.org/papers/volume24/23-0064/23-0064.pdf (2023).

  78. Zhuang, J., Kochkov, D., Bar-Sinai, Y., Brenner, M. P. & Hoyer, S. Learned discretizations for passive scalar advection in a two-dimensional turbulent flow. Phys. Rev. Fluids 6, 064605 (2021).

    Article  Google Scholar 

  79. Kube, R., Churchill, R.M. & Sturdevant, B. Machine learning accelerated particle-in-cell plasma simulations. In Proc. of Fourth Workshop on Machine Learning and the Physical Sciences (NeurIPS 2021) https://ml4physicalsciences.github.io/2021/files/NeurIPS_ML4PS_2021_26.pdf (2021).

  80. Stevens, B. & Colonius, T. FiniteNet: a fully convolutional LSTM network architecture for time-dependent partial differential equations. Preprint at https://arxiv.org/abs/2002.03014 (2020).

  81. Alguacil, A., Bauerheim, M., Jacob, M. C. & Moreau, S. Predicting the propagation of acoustic waves using deep convolutional neural networks. J. Sound Vib. 512, 116285 (2021).

    Article  Google Scholar 

  82. Alguacil, A., Bauerheim, M., Jacob, M. C. & Moreau, S. Deep learning surrogate for the temporal propagation and scattering of acoustic waves. AIAA J. 60, 5890–5906 (2022).

    Article  Google Scholar 

  83. Bezgin, D. A., Schmidt, S. J. & Adams, N. A. WENO3-NN: a maximum-order three-point data-driven weighted essentially non-oscillatory scheme. J. Comput. Phys. 452, 110920 (2022).

    Article  MathSciNet  Google Scholar 

  84. Xiao, X., Zhou, Y., Wang, H. & Yang, X. A novel cnn-based poisson solver for fluid simulation. IEEE Trans. Vis. Comput. Graphics 26, 1454–1465 (2018).

    Article  Google Scholar 

  85. Sanchez-Gonzalez, A. et al. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning 8459–8468 (PMLR, 2020).

  86. Klimesch, J., Holl, P. & Thuerey, N. Simulating liquids with graph networks. Preprint at https://arxiv.org/abs/2203.07895 (2022).

  87. Wang, S., Wang, H. & Perdikaris, P. Improved architectures and training algorithms for deep operator networks. J. Sci. Comput. 92, 35 (2022).

    Article  MathSciNet  Google Scholar 

  88. Gupta, J. K. & Brandstetter, J. Towards multi-spatiotemporal-scale generalized PDE modeling. Transact. Mach. Lean. Res. https://openreview.net/pdf?id=dPSTDbGtBY (2023).

  89. McGreivy, N. & Hakim, A. Invariant preservation in machine learned PDE solvers via error correction. Preprint at https://arxiv.org/abs/2303.16110 (2023).

  90. Stachenfeld, K. et al. Learned simulators for turbulence. In Proc. of International Conference on Learning Representations (ICLR) 2022 https://openreview.net/pdf?id=msRBojTz-Nh (2022).

  91. Cheng, L., Illarramendi, E. A., Bogopolsky, G., Bauerheim, M. & Cuenot, B. Using neural networks to solve the 2D Poisson equation for electric field computation in plasma fluid simulations. Preprint at https://arxiv.org/abs/2109.13076 (2021).

  92. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).

    Article  Google Scholar 

  93. Wicherts, J. M. et al. Degrees of freedom in planning, running, analyzing, and reporting psychological studies: a checklist to avoid p-hacking. Front. Psychol. 7, 1832 (2016).

  94. Serra-Garcia, M. & Gneezy, U. Nonreplicable publications are cited more than replicable ones. Sci. Adv. 7, eabd1705 (2021).

    Article  Google Scholar 

  95. Borji, A. Negative results in computer vision: a perspective. Image Vis. Comput. 69, 1–8 (2018).

    Article  Google Scholar 

  96. Smith, J. J., Amershi, S., Barocas, S., Wallach, H. & Wortman Vaughan, J. Real ML: recognizing, exploring, and articulating limitations of machine learning research. In 2022 ACM Conference on Fairness, Accountability, and Transparency 587–597 (ACM, 2022).

  97. Gundersen, O. E. The case against registered reports. AI Magazine 42, 88–92 (2021).

    Article  Google Scholar 

  98. Schooler, J. W. Metascience could rescue the ‘replication crisis’. Nature 515, 9–9 (2014).

    Article  Google Scholar 

  99. Ray, D. & Hesthaven, J. S. An artificial neural network as a troubled-cell indicator. J. Comput. Phys. 367, 166–191 (2018).

    Article  MathSciNet  Google Scholar 

  100. Wang, S. & Perdikaris, P. Long-time integration of parametric evolution equations with physics-informed deeponets. J. Comput. Phys. 475, 111855 (2023).

    Article  MathSciNet  Google Scholar 

  101. Ovadia, O., Kahana, A., Turkel, E. & Dekel, S. Beyond the Courant–Friedrichs–Lewy condition: numerical methods for the wave problem using deep learning. J. Comput. Phys. 442, 110493 (2021).

    Article  MathSciNet  Google Scholar 

  102. Li, Z. et al. Learning dissipative dynamics in chaotic systems. In Proc. of 36th Conference on Neural Information Processing Systems (NeurIPS) https://openreview.net/pdf?id=1C36tFZn7sR (2022).

  103. Ni, N. & Dong, S. Numerical computation of partial differential equations by hidden-layer concatenated extreme learning machine. J. Sci. Comput. 95, 35 (2023).

    Article  MathSciNet  Google Scholar 

  104. Dong, S. & Li, Z. A modified batch intrinsic plasticity method for pre-training the random coefficients of extreme learning machines. J. Comput. Phys. 445, 110585 (2021).

    Article  MathSciNet  Google Scholar 

  105. Mueller, M., Greif, R., Jenko, F. & Thuerey, N. Leveraging thestochastic predictions of Bayesian neural networks for fluid simulations. In Machine Learning and the Physical Sciences workshop, NeurIPS 2022 https://ml4physicalsciences.github.io/2022/files/NeurIPS_ML4PS_2022_107.pdf (2022).

  106. Wang, S., Bhouri, M. A. & Perdikaris, P. Fast PDE-constrained optimization via self-supervised operator learning. Preprint at https://arxiv.org/abs/2110.13297 (2021).

  107. Schwander, L., Ray, D. & Hesthaven, J. S. Controlling oscillations in spectral methods by local artificial viscosity governed by neural networks. J. Comput. Phys. 431, 110144 (2021).

    Article  MathSciNet  Google Scholar 

  108. Donon, B. et al. Deep statistical solvers. In 34th Conference on Neural Information Processing Systems Vol. 33, 7910–7921 (NeurIPS 2020).

  109. Wan, Z. Y., Zepeda-Núñez, L., Boral, A. & Sha, F. Evolve smoothly, fit consistently: learning smooth latent dynamics for advection-dominated systems. In Proc. of International Conference on Learning Representaton (ICLR) https://openreview.net/pdf?id=Z4s73sJYQM (2023).

  110. Di Leoni, P. C., Lu, L., Meneveau, C., Karniadakis, G. E. & Zaki, T. A. Neural operator prediction of linear instability waves in high-speed boundary layers. J. Comput. Phys. 474, 111793 (2023).

    Article  MathSciNet  Google Scholar 

  111. Kovachki, N. et al. Neural operator: learning maps between function spaces with applications to PDEs. J. of Mach. Learn. Res. https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf (2023).

  112. Holl, P., Koltun, V., Um, K. & Thuerey, N. phiflow: a differentiable PDE solving framework for deep learning via physical simulations. In Workshop on Differentiable Vision, Graphics, and Physics in Machine Learning at NeurIPS (NeurIPS, 2020).

  113. Nemmen, R., Duarte, R. & Navarro, J. P. The first AI simulation of a black hole. Proc. Intl Astron. Union 15, 329–333 (2020).

    Article  Google Scholar 

  114. Wandel, N., Weinmann, M. & Klein, R. Learning incompressible fluid dynamics from scratch–towards fast, differentiable fluid models that generalize. In Proc. of International Conference on Learning Representations (ICLR) 2021 https://openreview.net/pdf/a304e46c25faf8b1991a7580669d779b3c3e2cd6.pdf (2021).

  115. Haridas, A., Vadlamani, N. R. & Minamoto, Y. Deep neural networks to correct sub-precision errors in CFD. Appl. Energy Combustion Sci. 12, 100081 (2022).

    Article  Google Scholar 

  116. Guennebaud, G. et al. Eigen v 3 (Eigen, 2010); http://eigen.tuxfamily.org

  117. Falgout, R. D. & Yang, U. M. hypre: A library of high performance preconditioners. In International Conference on Computational Science 632–641 (Springer, 2002).

  118. Mayr, M., Wall, W. A. & Gee, M. W. Adaptive time stepping for fluid–structure interaction solvers. Finite Elements in Anal. Design 141, 55–69 (2018).

    Article  MathSciNet  Google Scholar 

  119. Reynolds, D. R., Woodward, C. S., Gardner, D. J. & Hindmarsh, A. C. Arkode: a library of high order implicit/explicit methods for multi-rate problems. In SIAM Conference on Parallel Processing for Scientific Computing (2014).

  120. Gottlieb, S., Shu, C.-W. & Tadmor, E. Strong stability-preserving high-order time discretization methods. SIAM Rev. 43, 89–112 (2001).

    Article  MathSciNet  Google Scholar 

  121. Hakim, A., Hammett, G., Shi, E. & Mandell, N. Discontinuous galerkin schemes for a class of hamiltonian evolution equations with applications to plasma fluid and kinetic problems. Preprint at https://arxiv.org/abs/1908.01814 (2019).

  122. Cockburn, B. & Shu, C.-W. Tvb runge-kutta local projection discontinuous galerkin finite element method for conservation laws. ii. General framework. Math. Comput. 52, 411–435 (1989).

    MathSciNet  Google Scholar 

  123. Qin, T., Beatson, A., Oktay, D., McGreivy, N. & Adams, R. P. Meta-PDE: learning to solve pdes quickly without a mesh. Preprint at https://arxiv.org/abs/2211.01604 (2022).

  124. McGreivy, N. & Hakim, A. Data Created During Random Sampling From and Systematic Review of ML-for-PDE Solving Research (OSF, 2024); https://doi.org/10.17605/OSF.IO/GQ5B3

  125. McGreivy, N. nickmcgreivy/WeakBaselinesMLPDE: First release (v1.0) (Zenodo, 2024); https://doi.org/10.5281/zenodo.12682908

  126. McGreivy, N. & Hakim, A. Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Code Ocean https://doi.org/10.24433/CO.8345217.v1 (2024).

  127. McGreivy, N. & Hakim, A. (GPU Code) Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Code Ocean https://doi.org/10.24433/CO.7508588.v1 (2024).

  128. Lu, L., Jin, P., Pang, G., Zhang, Z. & Karniadakis, G. E. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nat. Mach, Intell. 3, 218–229 (2021).

    Article  Google Scholar 

  129. Tompson, J., Schlachter, K., Sprechmann, P. & Perlin, K. Accelerating eulerian fluid simulation with convolutional networks. In International Conference on Machine Learning, 3424–3433 (PMLR, 2017).

  130. Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A. & Battaglia, P. W. Learning mesh-based simulation with graph networks. In Proc. of International Conference on Learning Representations (ICLR) 2021 https://openreview.net/pdf?id=roNqYL0_XP (2021).

  131. Kim, B. et al. Deep fluids: a generative network for parameterized fluid simulations. In Computer Graphics Forum Vol. 38, 59–70 (Wiley, 2019).

  132. Cai, S., Wang, Z., Lu, L., Zaki, T. A. & Karniadakis, G. E. DeepM&Mnet: inferring the electroconvection multiphysics fields based on operator approximation by neural networks. J. Comput. Phys. 436, 110296 (2021).

    Article  MathSciNet  Google Scholar 

  133. De Avila Belbute-Peres, F., Economon, T. & Kolter, Z. Combining differentiable PDE solvers and graph neural networks for fluid flow prediction. In International Conference on Machine Learning 2402–2411 (PMLR, 2020).

  134. Yang, C., Yang, X. & Xiao, X. Data-driven projection method in fluid simulation. Comput. Animation Virtual Worlds 27, 415–424 (2016).

    Article  Google Scholar 

  135. Hsieh, J.-T., Zhao, S., Eismann, S., Mirabella, L. & Ermon, S. Learning neural PDE solvers with convergence guarantees. In Proc. of International Conference on Learning Representations (ICLR) 2021 https://openreview.net/pdf?id=rklaWn0qK7 (2021).

  136. Mao, Z., Lu, L., Marxen, O., Zaki, T. A. & Karniadakis, G. E. DeepM&Mnet for hypersonics: predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators. J. Comput. Phys. 447, 110698 (2021).

    Article  MathSciNet  Google Scholar 

  137. Greenfeld, D., Galun, M., Basri, R., Yavneh, I. & Kimmel, R. Learning to optimize multigrid PDE solvers. In International Conference on Machine Learning 2415–2423 (PMLR, 2019).

  138. Ray, D. & Hesthaven, J. S. Detecting troubled-cells on two-dimensional unstructured grids using a neural network. J. Comput. Phys. 397, 108845 (2019).

    Article  MathSciNet  Google Scholar 

  139. Shan, T. et al. Study on a fast solver for poisson’s equation based on deep learning technique. IEEE Trans. Antennas Propagation 68, 6725–6733 (2020).

    Article  Google Scholar 

  140. Luz, I., Galun, M., Maron, H., Basri, R. & Yavneh, I. Learning algebraic multigrid using graph neural networks. In International Conference on Machine Learning 6489–6499 (PMLR, 2020).

  141. Pathak, J. et al. Using machine learning to augment coarse-grid computational fluid dynamics simulations. Preprint at https://arxiv.org/abs/2010.00072 (2020).

  142. Di Leoni, P. C., Lu, L., Meneveau, C., Karniadakis, G. & Zaki, T. A. Neural operator prediction of linear instability waves in high-speed boundary layers. J. Comput. Phys. https://doi.org/10.1016/j.jcp.2022.111793 (2023).

  143. Stevens, B. & Colonius, T. Enhancement of shock-capturing methods via machine learning. Theor. Comput. Fluid Dyn. 34, 483–496 (2020).

    Article  MathSciNet  Google Scholar 

  144. Illarramendi, E. A. et al. Towards an hybrid computational strategy based on deep learning for incompressible flows. In AIAA Aviation 2020 Forum 3058 (AIAA, 2020).

  145. Han, X., Gao, H., Pfaff, T., Wang, J.-X. & Liu, L.-P. Predicting physics in mesh-reduced space with temporal attention. In Proc. of the International Conference on Learning Responses (ICLR) 2022 https://openreview.net/pdf?id=XctLdNfCmP (2022).

  146. Özbay, A. G. et al. Poisson CNN: convolutional neural networks for the solution of the poisson equation on a cartesian mesh. Data-Centric Eng. 2, e6 (2021).

    Article  Google Scholar 

  147. Li, Z. & Farimani, A. B. Graph neural network-accelerated Lagrangian fluid simulation. Comput. Graphics 103, 201–211 (2022).

    Article  Google Scholar 

  148. Peng, W., Yuan, Z. & Wang, J. Attention-enhanced neural network models for turbulence simulation. Phys. Fluids, 34, 025111 (2022).

  149. Chen, L.-W., Cakal, B. A., Hu, X. & Thuerey, N. Numerical investigation of minimum drag profiles in laminar flow using deep learning surrogates. J. Fluid Mech. 919, A34 (2021).

    Article  MathSciNet  Google Scholar 

  150. Wandel, N., Weinmann, M. & Klein, R. Teaching the incompressible navier–stokes equations to fast neural surrogate models in three dimensions. Phys. Fluids, 33, 047117 (2021).

  151. List, B., Chen, L.-W. & Thuerey, N. Learned turbulence modelling with differentiable fluid solvers: physics-based loss functions and optimisation horizons. J. Fluid Mech. 949, A25 (2022).

    Article  MathSciNet  Google Scholar 

  152. Wen, X., Don, W. S., Gao, Z. & Hesthaven, J. S. An edge detector based on artificial neural network with application to hybrid compact-weno finite difference scheme. J. Sci. Comput. 83, 1–21 (2020).

    Article  MathSciNet  Google Scholar 

  153. de Lara, F. M. & Ferrer, E. Accelerating high order discontinuous galerkin solvers using neural networks: 1D Burgers’ equation. Comput. Fluids 235, 105274 (2022).

    Article  MathSciNet  Google Scholar 

  154. Zhao, Q., Lindell, D. B. & Wetzstein, G. Learning to solve PDE-constrained inverse problems with graph networks. In Proc. of the 39th International Conference on Machine Learning (eds Chaudhuri, K et al.) 162, 26895–26910 (2022).

  155. Illarramendi, E. A., Bauerheim, M. & Cuenot, B. Performance and accuracy assessments of an incompressible fluid solver coupled with a deep convolutional neural network. Data-Centric Engineering 3, e2 (2022).

    Article  Google Scholar 

  156. Holloway, I., Wood, A. & Alekseenko, A. Acceleration of boltzmann collision integral calculation using machine learning. Mathematics 9, 1384 (2021).

    Article  Google Scholar 

  157. Azulay, Y. & Treister, E. Multigrid-augmented deep learning preconditioners for the helmholtz equation. SIAM J. Sci. Comput. 45, S127–S151 (2022).

    Article  MathSciNet  Google Scholar 

  158. Wu, T., Maruyama, T. & Leskovec, J. Learning to accelerate partial differential equations via latent global evolution. In 36th Conference on Neural Information Processing Systems Vol. 35, 2240–2253 (NeurIPS 2022).

  159. Liu, X.-Y., Sun, H., Zhu, M., Lu, L. & Wang, J.-X. Predicting parametric spatiotemporal dynamics by multi-resolution PDE structure-preserved deep learning. Preprint at https://arxiv.org/abs/2205.03990 (2022).

  160. Zhang, E. et al. A hybrid iterative numerical transferable solver (hints) for pdes based on deep operator network and relaxation methods. Preprint at https://arxiv.org/abs/2208.13273 (2022).

  161. Duarte, R., Nemmen, R. & Navarro, J. P. Black hole weather forecasting with deep learning: a pilot study. Monthly Notices R. Astron. Soc. 512, 5848–5861 (2022).

    Article  Google Scholar 

  162. Huang, X. et al. LordNet: an efficient neural network for learning to solve parametric partial differential equations without simulated data. Neural Netw. https://doi.org/10.1016/j.neunet.2024.106354 (2024).

  163. Ranade, R., Hill, C., He, H., Maleki, A. & Pathak, J. A latent space solver for PDE generalization. In ICLR 2021 SimDL Workshop https://simdl.github.io/files/22.pdf (2021).

  164. Chen, R., Jin, X. & Li, H. A machine learning based solver for pressure poisson equations. Theor. Appl. Mech. Lett. 12, 100362 (2022).

    Article  Google Scholar 

  165. Ranade, R. et al. A composable autoencoder-based iterative algorithm for accelerating numerical simulations. Preprint at https://arxiv.org/abs/2110.03780 (2021).

  166. Peng, W., Yuan, Z., Li, Z. & Wang, J. Linear attention coupled fourier neural operator for simulation of three-dimensional turbulence. Phys. Fluids, 35, 015106 (2023).

  167. de Lara, F. M. & Ferrer, E. Accelerating high order discontinuous galerkin solvers using neural networks: 3D compressible Navier–Stokes equations. J. Comput. Phys. 489, 112253, (2023).

  168. Ranade, R., Hill, C., Ghule, L. & Pathak, J. A composable machine-learning approach for steady-state simulations on high-resolution grids. In 36th Conference on Neural Information Processing Systems Vol. 35, 17386–17401 (NeurIPS, 2022).

  169. Fang, D. & Tan, J. Immersed boundary-physics informed machine learning approach for fluid–solid coupling. Ocean Eng. 263, 112360 (2022).

    Article  Google Scholar 

  170. Shukla, K. et al. Deep neural operators can serve as accurate surrogates for shape optimization. Eng. Appl. Artif. Intell. https://doi.org/10.1016/j.engappai.2023.107615 (2024).

  171. Zhang, T., Innanen, K. & Trad, D. Learning the elastic wave equation with fourier neural operators. Geoconvention 2022, 1–5 (2022).

    Google Scholar 

  172. Bezgin, D. A., Buhendwa, A. B. & Adams, N. A. A fully-differentiable compressible high-order computational fluid dynamics solver. Preprint at https://arxiv.org/abs/2112.04979 (2021).

  173. Yang, Y., Gao, A. F., Azizzadenesheli, K., Clayton, R. W. & Ross, Z. E. Rapid seismic waveform modeling and inversion with neural operators. IEEE Trans. Geosci. Remote Sensing 61, 1–12 (2023).

    Google Scholar 

  174. Tang, J., Azevedo, V. C., Cordonnier, G. & Solenthaler, B. Neural green’s function for laplacian systems. Comput. Graphics 107, 186–196 (2022).

    Article  Google Scholar 

  175. Nastorg, M. et al. DS-GPS: a deep statistical graph poisson solver (for faster CFD simulations). In Machine Learning and the Physical Sciences workshop, NeurIPS 2022 https://inria.hal.science/hal-03861311/file/DS_GPS.pdf (2022).

  176. Gopakumar, V. et al. Fourier neural operator for plasma modelling. Preprint at https://arxiv.org/abs/2302.06542 (2023).

  177. Shit, S. et al. Semi-implicit neural solver for time-dependent partial differential equations. Preprint at https://arxiv.org/abs/2109.01467 (2021).

  178. Su, X., Walters, R., Aslangil, D. & Yu, R. Forecasting variable-density 3D turbulent flow. In ICLR 2021 SimDL Workshop (ICLR, 2021).

  179. Jeon, J., Lee, J., Eivazi, H., Vinuesa, R. & Kim, S. J. Residual-based physics-informed transfer learning: a hybrid method for accelerating long-term CFD simulations via deep learning. Int. J. Heat Mass Transf. https://doi.org/10.1016/j.ijheatmasstransfer.2023.124900 (2024).

  180. Dai, Y., An, Y. & Li, Z. Fournetflows: an efficient model for steady airfoil flows prediction. Preprint at https://arxiv.org/abs/2207.04358 (2022).

  181. Sun, J., Dong, S. & Wang, F. Local randomized neural networks with discontinuous Galerkin methods for partial differential equations. J. Comput. App. Math. https://doi.org/10.1016/j.cam.2024.115830 (2024).

  182. Shao, T., Shan, T., Li, M., Yang, F. & Xu, S. A poisson’s equation solver based on neural network precondtioned CG method. In 2022 International Applied Computational Electromagnetics Society Symposium (ACES-China) 1–2 (IEEE, 2022).

Download references

Acknowledgements

N.M. was supported via DOE contract DE-AC02-09CH11466 for the Princeton Plasma Physics Laboratory. A.H. was supported by the Partnership for Multiscale Gyrokinetic Turbulence (MGK) and the High-Fidelity Boundary Plasma Simulation (HBPS) projects, part of the U.S. Department of Energy (DOE) Scientific Discovery Through Advanced Computing (SciDAC) program, and the DOE’s ARPA-E BETHE program, via DOE contract DE-AC02-09CH11466 for the Princeton Plasma Physics Laboratory. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Department of Astrophysical Sciences, Princeton University, Princeton, NJ, USA

    Nick McGreivy

  2. Princeton Plasma Physics Laboratory, Princeton, NJ, USA

    Nick McGreivy & Ammar Hakim

Authors
  1. Nick McGreivy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ammar Hakim
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

N.M. conceptualized the systematic review, searched for papers matching the inclusion criteria, evaluated rules 1 and 2 for each article, conceptualized and performed analyses to measure the effect of reporting biases, and wrote the code and the manuscript. A.H. designed strong baselines, evaluated rule 2 for each PDE, provided instructions for implementing the code, edited the manuscript and supervised the research.

Corresponding author

Correspondence to Nick McGreivy.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Petros Koumoutsakos, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Section 1.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

McGreivy, N., Hakim, A. Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Nat Mach Intell (2024). https://doi.org/10.1038/s42256-024-00897-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s42256-024-00897-5

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics