Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Recurrent graph optimal transport for learning 3D flow motion in particle tracking

Nature Machine Intelligence volume 5, pages 505–517 (2023)Cite this article

Subjects

Abstract

Flow visualization technologies such as particle tracking velocimetry are broadly used for studying three-dimensional turbulent flow in natural and industrial processes. Despite the advances in three-dimensional acquisition techniques, it is challenging to develop motion estimation algorithms in particle tracking due to large particle displacements, dense particle distributions and high computational cost. We present an end-to-end solution called graph optimal transport (GotFlow3D) to learn the three-dimensional fluid flow motion from consecutive particle images. The proposed model uses a graph neural network to extract geometric features and to further enrich the particle representations. The extracted deep features are subsequently used to correspond particles between consecutive frames, and the flow motion is then iteratively updated with a recurrent neural network approach. Experimental evaluations—including assessments on numerical experiments and validations on real-world experiments—demonstrate that GotFlow3D achieves state-of-the-art performance compared with recently developed scene flow learners and particle tracking algorithms. We believe that the high accuracy, robustness and generalization ability of our method can provide deeper insight into the complex dynamics of many physical and biological systems.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of GotFlow3D for learning flow motion from particle sets.
Fig. 2: Flow estimation performance in FluidFlow3D-norm dataset.
Fig. 3: Robustness assessment for GotFlow3D.
Fig. 4: GotFlow3D applied to particle tracking velocimetry for CylinderFlow and DeformationFlow datasets.

Similar content being viewed by others

Data availability

The synthetic dataset FluidFlow3D is available at https://github.com/JiamingSkGrey/FluidFlow3D-family. The open-source dataset CylinderFlow is available at https://doi.org/10.15454/GLNRHK. The open-source dataset DeformationFlow is available at https://github.com/FranckLab/SerialTrack.

Code availability

The source code is available at https://github.com/JiamingSkGrey/GotFlow3D.

References

  1. Kemp, M. Leonardo da Vinci’s laboratory: studies in flow. Nature 571, 322–324 (2019).

    Article  Google Scholar 

  2. Dabiri, D. & Pecora, C. Particle Tracking Velocimetry (IOP, 2020).

  3. Kopitca, A., Latifi, K. & Zhou, Q. Programmable assembly of particles on a Chladni plate. Sci. Adv. 7, eabi7716 (2021).

    Article  Google Scholar 

  4. Ferdowsi, B., Ortiz, C. P., Houssais, M. & Jerolmack, D. J. River-bed armouring as a granular segregation phenomenon. Nat. Commun. 8, 1363 (2017).

    Article  Google Scholar 

  5. Hu, D. L., Chan, B. & Bush, J. W. The hydrodynamics of water strider locomotion. Nature 424, 663–666 (2003).

    Article  Google Scholar 

  6. He, B., Doubrovinski, K., Polyakov, O. & Wieschaus, E. Apical constriction drives tissue-scale hydrodynamic flow to mediate cell elongation. Nature 508, 392–396 (2014).

    Article  Google Scholar 

  7. Mestre, H. et al. Cerebrospinal fluid influx drives acute ischemic tissue swelling. Science 367, eaax7171 (2020).

    Article  Google Scholar 

  8. Zhang, Z., Hwang, M., Kilbaugh, T. J., Sridharan, A. & Katz, J. Cerebral microcirculation mapped by echo particle tracking velocimetry quantifies the intracranial pressure and detects ischemia. Nat. Commun. 13, 1–15 (2022).

    Google Scholar 

  9. Guo, M. et al. Probing the stochastic, motor-driven properties of the cytoplasm using force spectrum microscopy. Cell 158, 822–832 (2014).

    Article  Google Scholar 

  10. Peng, Y., Liu, Z. & Cheng, X. Imaging the emergence of bacterial turbulence: phase diagram and transition kinetics. Sci. Adv. 7, eabd1240 (2021).

    Article  Google Scholar 

  11. Schuerle, S. et al. Synthetic and living micropropellers for convection-enhanced nanoparticle transport. Sci. Adv. 5, eaav4803 (2019).

    Article  Google Scholar 

  12. Punzmann, H., Francois, N., Xia, H., Falkovich, G. & Shats, M. Generation and reversal of surface flows by propagating waves. Nat. Phys. 10, 658–663 (2014).

    Article  Google Scholar 

  13. Huang, P. Y. et al. Imaging atomic rearrangements in two-dimensional silica glass: watching silica’s dance. Science 342, 224–227 (2013).

    Article  Google Scholar 

  14. Pereira, F., Stüer, H., Graff, E. C. & Gharib, M. Two-frame 3D particle tracking. Meas. Sci. Technol. 17, 1680 (2006).

    Article  Google Scholar 

  15. Leggett, S. E. et al. Mechanophenotyping of 3D multicellular clusters using displacement arrays of rendered tractions. Proc. Natl Acad. Sci. USA 117, 5655–5663 (2020).

    Article  Google Scholar 

  16. Raffel, M. et al. Particle Image Velocimetry: A Practical Guide (Springer, 2018).

  17. Cierpka, C., Lütke, B. & Kähler, C. J. Higher order multi-frame particle tracking velocimetry. Exp. Fluids 54, 1533 (2013).

    Article  Google Scholar 

  18. Brunton, S. L., Noack, B. R. & Koumoutsakos, P. Machine learning for fluid mechanics. Ann. Rev. Fluid Mech. 52, 477–508 (2020).

    Article  MathSciNet  Google Scholar 

  19. Cai, S., Zhou, S., Xu, C. & Gao, Q. Dense motion estimation of particle images via a convolutional neural network. Exp. Fluids 60, 73 (2019).

    Article  Google Scholar 

  20. Cai, S., Liang, J., Gao, Q., Xu, C. & Wei, R. Particle image velocimetry based on a deep learning motion estimator. IEEE Trans. Instrum. Meas. 69, 3538–3554 (2019).

    Article  Google Scholar 

  21. Liang, J., Cai, S., Xu, C. & Chu, J. Filtering enhanced tomographic PIV reconstruction based on deep neural networks. IET Cyber-Syst. Robot. 2, 43–52 (2020).

    Article  Google Scholar 

  22. Lagemann, C., Lagemann, K., Mukherjee, S. & Schröder, W. Deep recurrent optical flow learning for particle image velocimetry data. Nat. Mach. Intell. 3, 641–651 (2021).

    Article  Google Scholar 

  23. Gim, Y., Jang, D. K., Sohn, D. K., Kim, H. & Ko, H. S. Three-dimensional particle tracking velocimetry using shallow neural network for real-time analysis. Exp. Fluids 61, 26 (2020).

    Article  Google Scholar 

  24. Mallery, K., Shao, S. & Hong, J. Dense particle tracking using a learned predictive model. Exp. Fluids 61, 223 (2020).

    Article  Google Scholar 

  25. Qi, C. R., Su, H., Mo, K. & Guibas, L. J. Pointnet: deep learning on point sets for 3D classification and segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 652–660 (IEEE, 2017).

  26. Liu, X., Qi, C. R. & Guibas, L. J. FlowNet3D: learning scene flow in 3D point clouds. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 529–537 (IEEE, CVF, 2019).

  27. Puy, G., Boulch, A. & Marlet, R. FLOT: scene flow on point clouds guided by optimal transport. In European Conference on Computer Vision 527–544 (Springer, 2020).

  28. Wu, W., Wang, Z. Y., Li, Z., Liu, W. & Fuxin, L. PointPWC-Net: cost volume on point clouds for (self-) supervised scene flow estimation. In European Conference on Computer Vision 88–107 (Springer, 2020).

  29. Wei, Y., Wang, Z., Rao, Y., Lu, J. & Zhou, J. PV-RAFT: point-voxel correlation fields for scene flow estimation of point clouds. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 6954–6963 (IEEE, CVF, 2021).

  30. Liang, J., Cai, S., Xu, C., Chen, T. & Chu, J. DeepPTV: particle tracking velocimetry for complex flow motion via deep neural networks.IEEE Trans. Instrum. Meas. 71, 9594090 (2021).

    Google Scholar 

  31. Wu, Z. et al. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32, 4–24 (2020).

    Article  MathSciNet  Google Scholar 

  32. Khojasteh, A. R., Laizet, S., Heitz, D. & Yang, Y. Lagrangian and Eulerian dataset of the wake downstream of a smooth cylinder at a Reynolds number equal to 3900. Data Brief 40, 107725 (2022).

    Article  Google Scholar 

  33. Patel, M., Leggett, S. E., Landauer, A. K., Wong, I. Y. & Franck, C. Rapid, topology-based particle tracking for high-resolution measurements of large complex 3D motion fields. Sci. Rep. 8, 5581 (2018).

    Article  Google Scholar 

  34. Yang, J. et al. SerialTrack: ScalE and rotation invariant augmented Lagrangian particle tracking. SoftwareX 19, 101204 (2022).

    Article  Google Scholar 

  35. Teed, Z. & Deng, J. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision 402–419 (Springer, 2020).

  36. Sanchez-Gonzalez, A., et al. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning 8459–8468 (PMLR, 2020).

  37. Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A. & Battaglia, P. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations 2837 (ICLR, 2021).

  38. Schanz, D., Gesemann, S. & Schröder, A. Shake-The-Box: Lagrangian particle tracking at high particle image densities. Exp. Fluids 57, 70 (2016).

    Article  Google Scholar 

  39. Tan, S., Salibindla, A., Masuk, A. U. M. & Ni, R. Introducing OpenLPT: new method of removing ghost particles and high-concentration particle shadow tracking. Exp. Fluids 61, 47 (2020).

    Article  Google Scholar 

  40. Cornic, P. et al. Double-frame tomographic PTV at high seeding densities. Exp. Fluids 61, 23 (2020).

    Article  Google Scholar 

  41. Qi, C. R., Yi, L., Su, H. & Guibas, L. J. PointNet++: deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems Vol. 30 (NeurIPS, 2017).

  42. Wang, Y. et al. Dynamic graph CNN for learning on point clouds. ACM Trans. Graphics 38, 1–12 (2019).

    Google Scholar 

  43. Peyré, G. et al. Computational optimal transport: with applications to data science. Found. Trends Mach. Learn. 11, 355–607 (2019).

    Article  Google Scholar 

  44. Cuturi, M. Sinkhorn distances: lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems Vol. 26 (NeurIPS, 2013).

  45. Li, Y. et al. A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. J. Turbulence 9, N31 (2008).

  46. Ethier, C. R. & Steinman, D. Exact fully 3D Navier–Stokes solutions for benchmarking. Int. J. Numer. Methods Fluids 19, 369–375 (1994).

    Article  Google Scholar 

  47. Maas, H., Gruen, A. & Papantoniou, D. Particle tracking velocimetry in three-dimensional flows. Exp. Fluids 15, 133–146 (1993).

    Article  Google Scholar 

  48. Bhattacharya, S. & Vlachos, P. P. Volumetric particle tracking velocimetry (PTV) uncertainty quantification. Exp. Fluids 61, 197 (2020).

    Article  Google Scholar 

  49. Atkinson, C. & Soria, J. An efficient simultaneous reconstruction technique for tomographic particle image velocimetry. Exp. Fluids 47, 553–568 (2009).

    Article  Google Scholar 

  50. Scimone, M. T. et al. Modular approach for resolving and mapping complex neural and other cellular structures and their associated deformation fields in three dimensions. Nat. Protocols 13, 3042–3064 (2018).

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by: the National Key R&D Program of China (grant no. 2019YFB1705800); the National Natural Science Foundation of China (grant no. 61973270); and the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (grant no. 61621002). S.C. gratefully acknowledges the support of Zhejiang University Education Foundation Qizhen Scholar Foundation.

Author information

Authors and Affiliations

  1. Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou, China

    Jiaming Liang, Chao Xu & Shengze Cai

  2. State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou, China

    Jiaming Liang, Chao Xu & Shengze Cai

  3. Zhejiang Provincial Engineering Research Center for Intelligent Mobile Unmanned Systems Technology and Huzhou Key Lab for Autonomous Systems, Huzhou Institute of Zhejiang University, Huzhou, China

    Chao Xu

Authors
  1. Jiaming Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengze Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.L., C.X. and S.C. designed research, and edited and reviewed the paper. J.L. wrote the original draft and performed the research (implemented the algorithms and generated synthetic datasets). J.L. and S.C. analysed data. C.X. and S.C. supervised research.

Corresponding authors

Correspondence to Chao Xu or Shengze Cai.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Steven Wereley and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Robustness assessments for different flow learning methods with respect to (a) displacement ratio ρ, (b) noise from particles correspondence, (c) noise from particles localization.

The Acc Strict, Acc Relax, and Outliers are demonstrated as the evaluation metrics here. The proposed GotFlow3D outperforms the others in all the assessments.

Extended Data Table 1 Configuration of various flow cases in the synthetic dataset
Full size table
Extended Data Table 2 Comparison between the state-of-the-art scene flow learning methods and GotFlow3D on different flow cases in the synthetic FluidFlow3D-norm dataset
Full size table

Supplementary information

Supplementary Information

Ablation study on different modules in GotFlow3D, additional results on CylinderFlow dataset and GotFlow3D.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, J., Xu, C. & Cai, S. Recurrent graph optimal transport for learning 3D flow motion in particle tracking. Nat Mach Intell 5, 505–517 (2023). https://doi.org/10.1038/s42256-023-00648-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-023-00648-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics