Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Deep recurrent optical flow learning for particle image velocimetry data

Nature Machine Intelligence volume 3, pages 641–651 (2021)Cite this article

Subjects

Abstract

A wide range of problems in applied physics and engineering involve learning physical displacement fields from data. In this paper we propose a deep neural network-based approach for learning displacement fields in an end-to-end manner, focusing on the specific case of particle image velocimetry (PIV), a key approach in experimental fluid dynamics that is of crucial importance in diverse applications such as automotive, aerospace and biomedical engineering. The current state of the art in PIV data processing involves traditional handcrafted models that are subject to limitations including the substantial manual effort required and difficulties in generalizing across conditions. By contrast, the deep learning-based approach introduced in this paper, which is based on a recent optical flow learning architecture known as recurrent all-pairs field transforms, is general, largely automated and provides high spatial resolution. Extensive experiments, including benchmark examples where true gold standards are available for comparison, demonstrate that the proposed approach achieves state-of-the-art accuracy and generalization to new data, relative to both classical approaches and previously proposed optical flow learning schemes.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Set-up and general workflow of PIV experiments and their analysis.
Fig. 2: Main components of RAFT-PIV and comparisons of the particle image datasets.
Fig. 3: Problem Class II—test case cylinder.
Fig. 4: Problem Class III—test case transitional boundary layer.
Fig. 5: Problem Class III—test case turbulent wavy channel flow.

Similar content being viewed by others

Data availability

The public PIV particle image database (Problem Class I) can be found at https://github.com/shengzesnail/PIV_dataset. The Problem Class 2 dataset can be downloaded from Zenodo (https://doi.org/10.5281/zenodo.4432496)56.

Code availability

A reference implementation of RAFT can be found at https://github.com/princeton-vl/RAFT. A Code Ocean compute capsule, which contains a pre-built compute environment and the source code, is available at https://codeocean.com/capsule/7226151/tree/v1 (https://doi.org/10.24433/CO.4413978.v1)57.

References

  1. Braun, M., Schröder, W. & Klaas, M. High-speed tomographic PIV measurements in a DISI engine. Exp. Fluids 60, 146 (2019).

    Article  Google Scholar 

  2. Stella, A., Guj, G., Kompenhans, J., Raffel, M. & Richard, H. Application of particle image velocimetry to combusting flows: design considerations and uncertainty assessment. Exp. Fluids 30, 167–180 (2001).

    Article  Google Scholar 

  3. Pielhop, K., Schmidt, C., Zholtovski, S., Klaas, M. & Schröder, W. Experimental investigation of the fluid–structure interaction in an elastic 180 curved vessel at laminar oscillating flow. Exp. Fluids 55, 1816 (2014).

    Article  Google Scholar 

  4. Saaid, H. et al. In vitro volumetric lagrangian particle tracking and 4D pressure field in a left ventricle model. In Summer Biomechanics, Bioengineering and Biotransport Conference (SB3C, 2019).

  5. Li, W. et al. Analysis of a drag reduced flat plate turbulent boundary layer via uniform momentum zones. Aerospace Sci. Technol. 96, 105552 (2020).

    Article  Google Scholar 

  6. Herbst, S. L., Kähler, C. J. & Hain, R. Sd7003 airfoil in large-scale free stream turbulence. In 35th AIAA Applied Aerodynamics Conference 3748 (AIAA, 2017).

  7. Nobach, H. & Honkanen, M. Two-dimensional gaussian regression for sub-pixel displacement estimation in particle image velocimetry or particle position estimation in particle tracking velocimetry. Exp. Fluids 38, 511–515 (2005).

    Article  Google Scholar 

  8. Scarano, F. & Riethmuller, M. L. Iterative multigrid approach in PIV image processing with discrete window offset. Exp. Fluids 26, 513–523 (1999).

    Article  Google Scholar 

  9. Westerweel, J. & Scarano, F. Universal outlier detection for PIV data. Exp. Fluids 39, 1096–1100 (2005).

    Article  Google Scholar 

  10. Karri, S., Charonko, J. & Vlachos, P. P. Robust wall gradient estimation using radial basis functions and proper orthogonal decomposition (POD) for particle image velocimetry (PIV) measured fields. Measurement Sci. Technol. 20, 045401 (2009).

    Article  Google Scholar 

  11. Teed, Z. & Deng, J. RAFT: recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision 402–419 (Springer, 2020).

  12. Dosovitskiy, A. et al. Flownet: learning optical flow with convolutional networks. In Proc. IEEE International Conference on Computer Vision 2758–2766 (IEEE, 2015).

  13. Ilg, E. et al. Flownet 2.0: evolution of optical flow estimation with deep networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 2462–2470 (2017).

  14. Hui, T.-W., Tang, X. & Change Loy, C. Liteflownet: a lightweight convolutional neural network for optical flow estimation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 8981–8989 (IEEE, 2018).

  15. Sun, D., Yang, X., Liu, M.-Y. & Kautz, J. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 8934–8943 (IEEE, 2018).

  16. Hur, J. & Roth, S. Iterative residual refinement for joint optical flow and occlusion estimation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 5754–5763 (IEEE, 2019).

  17. Yang, G. & Ramanan, D. Volumetric correspondence networks for optical flow. In Advances in Neural Information Processing Systems 794–805 (NIPS, 2019).

  18. Rabault, J., Kolaas, J. & Jensen, A. Performing particle image velocimetry using artificial neural networks: a proof-of-concept. Meas. Sci. Technol. 28, 125301 (2017).

    Article  Google Scholar 

  19. Lee, Y., Yang, H. & Yin, Z. PIV-DCNN: cascaded deep convolutional neural networks for particle image velocimetry. Exp. Fluids 58, 171 (2017).

    Article  Google Scholar 

  20. Cai, S., Liang, J., Gao, Q., Xu, C. & Wei, R. Particle image velocimetry based on a deep learning motion estimator. IEEE Trans. Instrum. Meas. 69, 3538–3554 (2019).

    Article  Google Scholar 

  21. Cai, S., Zhou, S., Xu, C. & Gao, Q. Dense motion estimation of particle images via a convolutional neural network. Exp. Fluids 60, 73 (2019).

    Article  Google Scholar 

  22. Lagemann, C., Lagemann, K., Schröder, W. & Klaas, M. Deep artificial neural network architectures in PIV applications. In 13th International Symposium on Particle Image Velocimetry (ISPIV, 2019).

  23. Horn, B. K. & Schunck, B. G. Determining optical flow. In Techniques and Applications of Image Understanding Vol. 281, 319–331 (1981).

  24. Bruhn, A., Weickert, J. & Schnörr, C. Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. Int. J. Comput. Vision 61, 211–231 (2005).

    Article  Google Scholar 

  25. Mayer, N. et al. What makes good synthetic training data for learning disparity and optical flow estimation? Int. J. Comput. Vision 126, 942–960 (2018).

    Article  Google Scholar 

  26. Perry, B. A. & Mueller, M. E. Joint probability density function models for multiscalar turbulent mixing. Combust. Flame 193, 344–362 (2018).

    Article  Google Scholar 

  27. Zhang, C., Duan, L. & Choudhari, M. M. Direct numerical simulation database for supersonic and hypersonic turbulent boundary layers. AIAA J. 56, 4297–4311 (2018).

    Article  Google Scholar 

  28. Schlatter, P. et al. Turbulent boundary layers up to Reθ = 2500 studied through simulation and experiment. Phys. Fluids 21, 051702 (2009).

    Article  Google Scholar 

  29. Zhang, M. & Piggott, M. D. Unsupervised learning of particle image velocimetry. In International Conference on High Performance Computing 102–115 (Springer, 2020).

  30. Huang, H., Fiedler, H. & Wang, J. Limitation and improvement of PIV. Exp. Fluids 15, 263–273 (1993).

    Article  Google Scholar 

  31. Soria, J., Cater, J. & Kostas, J. High resolution multigrid cross-correlation digital PIV measurements of a turbulent starting jet using half frame image shift film recording. Optics Laser Technol. 31, 3–12 (1999).

    Article  Google Scholar 

  32. Zaki, T. A. From streaks to spots and on to turbulence: exploring the dynamics of boundary layer transition. Flow Turbul. Combust. 91, 451–473 (2013).

    Article  Google Scholar 

  33. Li, Y. et al. A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. J. Turbul. 91, N31 (2008).

    Article  Google Scholar 

  34. Perlman, E., Burns, R., Li, Y. & Meneveau, C. Data exploration of turbulence simulations using a database cluster. In Proc. 2007 ACM/IEEE Conference on Supercomputing 1–11 (IEEE, 2007).

  35. Albers, M. et al. Drag reduction and energy saving by spanwise traveling transversal surface waves for flat plate flow. Flow Turbul. Combust. 105, 1–33 (2020).

  36. Jonschkowski, R. et al. What matters in unsupervised optical flow. In European Conference on Computer Vision 557–572 (Springer, 2020).

  37. Rubbert, A., Albers, M. & Schröder, W. Streamline segment statistics propagation in inhomogeneous turbulence. Phys. Rev. Fluids 4, 034605 (2019).

    Article  Google Scholar 

  38. Charonko, J. J. & Vlachos, P. P. Estimation of uncertainty bounds for individual particle image velocimetry measurements from cross-correlation peak ratio. Meas. Sci. Technol. 24, 065301 (2013).

    Article  Google Scholar 

  39. Xue, Z., Charonko, J. J. & Vlachos, P. P. Particle image velocimetry correlation signal-to-noise ratio metrics and measurement uncertainty quantification. Meas. Sci.Technol. 25, 115301 (2014).

    Article  Google Scholar 

  40. Michaelis, D., Neal, D. R. & Wieneke, B. Peak-locking reduction for particle image velocimetry. Meas. Sci. Technol. 27, 104005 (2016).

    Article  Google Scholar 

  41. Cai, S. et al. Artificial intelligence velocimetry and microaneurysm-on-a-chip for three-dimensional analysis of blood flow in physiology and disease. Proc. Natl Acad. Sci. USA 118, e2100697118 (2021).

    Article  Google Scholar 

  42. Blundell, C., Cornebise, J., Kavukcuoglu, K. & Wierstra, D. Weight uncertainty in neural network. In Proc. 32nd International Conference on Machine Learning Vol. 37, 1613–1622 (ICML, 2015).

  43. Morrell, M. C., Hickmann, K. & Wilson, B. Particle image velocimetry analysis with simultaneous uncertainty quantification using bayesian neural networks. Measurement Science and Technology (2021).

  44. Siam, M., Valipour, S., Jagersand, M. & Ray, N. Convolutional gated recurrent networks for video segmentation. In 2017 IEEE International Conference on Image Processing 3090–3094 (IEEE, 2017).

  45. Paszke, A. et al. Automatic differentiation in pytorch. In 31st Conference on Neural Information Processing Systems (NIPS 2017).

  46. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  47. Marquardt, P., Klaas, M. & Schröder, W. Experimental investigation of isoenergetic film-cooling flows with shock interaction. AIAA J. 57, 3910–3923 (2019).

    Article  Google Scholar 

  48. Astarita, T. & Cardone, G. Analysis of interpolation schemes for image deformation methods in PIV. Exp. Fluids 38, 233–243 (2005).

    Article  Google Scholar 

  49. Gallivan, K., Grimme, G. & Van Dooren, P. A rational Lanczos algorithm for model reduction. Numer. Algorithms 12, 33–63 (1996).

    Article  MathSciNet  Google Scholar 

  50. Schrijer, F. & Scarano, F. Effect of predictor–corrector filtering on the stability and spatial resolution of iterative PIV interrogation. Exp. Fluids 45, 927–941 (2008).

    Article  Google Scholar 

  51. Raffel, M. et al. Particle Image Velocimetry: A Practical Guide (Springer, 2018).

  52. Héas, P., Mémin, E., Heitz, D. & Mininni, P. D. Power laws and inverse motion modelling: application to turbulence measurements from satellite images. Tellus A 64, 10962 (2012).

    Article  Google Scholar 

  53. Lee, M. & Moser, R. D. Direct numerical simulation of turbulent channel flow up to Reτ ≈ 5200. J. Fluid Mech. 774, 395–415 (2015).

    Article  Google Scholar 

  54. Resseguier, V., Mémin, E. & Chapron, B. Geophysical flows under location uncertainty. Part II. Quasi-geostrophy and efficient ensemble spreading. Geophys. Astrophys. Fluid Dyn. 111, 177–208 (2017).

    Article  MathSciNet  Google Scholar 

  55. Scharnowski, S. & Kähler, C. J. On the loss-of-correlation due to PIV image noise. Exp. Fluids 57, 1–12 (2016).

    Article  Google Scholar 

  56. Lagemann, C., Lagemann, K., Mukherjee, S. & Schröder, W. Dataset—Deep Recurrent Optical Flow Learning for Particle Image Velocimetry Data (Zenodo, 2021).

  57. Lagemann, C., Lagemann, K., Mukherjee, S. & Schröder, W. Recurrent All-Pairs Field Transforms for Particle Image Velocimetry Data (CodeOcean, 2021).

Download references

Acknowledgements

We thank P. Marquardt for his support and many fruitful discussions, M. Albers for providing DNS data and M. Klaas for proofreading parts of the manuscript. The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time on the GCS Supercomputers HAWK at Höchstleistungsrechenzentrum Stuttgart (www.hlrs.de) and Juwels at the Forschungszentrum Jülich (www.fz-juelich.de).

Author information

Authors and Affiliations

  1. Institute of Aerodynamics Aachen, RWTH Aachen University, Aachen, Germany

    Christian Lagemann & Wolfgang Schröder

  2. Statistics and Machine Learning, DZNE, Bonn, Germany

    Kai Lagemann & Sach Mukherjee

  3. University of Cambridge, Cambridge, UK

    Sach Mukherjee

  4. JARA Center for Simulation and Data Science, RWTH Aachen University, Aachen, Germany

    Wolfgang Schröder

Authors
  1. Christian Lagemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kai Lagemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sach Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wolfgang Schröder
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.L. contributed the original idea, algorithms, the realistic dataset, ablation studies and wrote the majority of the manuscript. K.L. contributed to the implementation and helped writing the paper. Review and editing were carried out by C.L., K.L., S.M. and W.S. Supervision was carried out by S.M. and W.S.

Corresponding author

Correspondence to Christian Lagemann.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Machine Intelligence thanks Koji Fukagata, Romit Maulik and the other, anonymous, reviewer(s) for their contrubition to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–16, Discussion and Tables 1–6.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lagemann, C., Lagemann, K., Mukherjee, S. et al. Deep recurrent optical flow learning for particle image velocimetry data. Nat Mach Intell 3, 641–651 (2021). https://doi.org/10.1038/s42256-021-00369-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00369-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics