Flow visualization technologies such as particle tracking velocimetry are broadly used for studying three-dimensional turbulent flow in natural and industrial processes. Despite the advances in three-dimensional acquisition techniques, it is challenging to develop motion estimation algorithms in particle tracking due to large particle displacements, dense particle distributions and high computational cost. We present an end-to-end solution called graph optimal transport (GotFlow3D) to learn the three-dimensional fluid flow motion from consecutive particle images. The proposed model uses a graph neural network to extract geometric features and to further enrich the particle representations. The extracted deep features are subsequently used to correspond particles between consecutive frames, and the flow motion is then iteratively updated with a recurrent neural network approach. Experimental evaluations—including assessments on numerical experiments and validations on real-world experiments—demonstrate that GotFlow3D achieves state-of-the-art performance compared with recently developed scene flow learners and particle tracking algorithms. We believe that the high accuracy, robustness and generalization ability of our method can provide deeper insight into the complex dynamics of many physical and biological systems.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
The synthetic dataset FluidFlow3D is available at https://github.com/JiamingSkGrey/FluidFlow3D-family. The open-source dataset CylinderFlow is available at https://doi.org/10.15454/GLNRHK. The open-source dataset DeformationFlow is available at https://github.com/FranckLab/SerialTrack.
The source code is available at https://github.com/JiamingSkGrey/GotFlow3D.
Kemp, M. Leonardo da Vinci’s laboratory: studies in flow. Nature 571, 322–324 (2019).
Dabiri, D. & Pecora, C. Particle Tracking Velocimetry (IOP, 2020).
Kopitca, A., Latifi, K. & Zhou, Q. Programmable assembly of particles on a Chladni plate. Sci. Adv. 7, eabi7716 (2021).
Ferdowsi, B., Ortiz, C. P., Houssais, M. & Jerolmack, D. J. River-bed armouring as a granular segregation phenomenon. Nat. Commun. 8, 1363 (2017).
Hu, D. L., Chan, B. & Bush, J. W. The hydrodynamics of water strider locomotion. Nature 424, 663–666 (2003).
He, B., Doubrovinski, K., Polyakov, O. & Wieschaus, E. Apical constriction drives tissue-scale hydrodynamic flow to mediate cell elongation. Nature 508, 392–396 (2014).
Mestre, H. et al. Cerebrospinal fluid influx drives acute ischemic tissue swelling. Science 367, eaax7171 (2020).
Zhang, Z., Hwang, M., Kilbaugh, T. J., Sridharan, A. & Katz, J. Cerebral microcirculation mapped by echo particle tracking velocimetry quantifies the intracranial pressure and detects ischemia. Nat. Commun. 13, 1–15 (2022).
Guo, M. et al. Probing the stochastic, motor-driven properties of the cytoplasm using force spectrum microscopy. Cell 158, 822–832 (2014).
Peng, Y., Liu, Z. & Cheng, X. Imaging the emergence of bacterial turbulence: phase diagram and transition kinetics. Sci. Adv. 7, eabd1240 (2021).
Schuerle, S. et al. Synthetic and living micropropellers for convection-enhanced nanoparticle transport. Sci. Adv. 5, eaav4803 (2019).
Punzmann, H., Francois, N., Xia, H., Falkovich, G. & Shats, M. Generation and reversal of surface flows by propagating waves. Nat. Phys. 10, 658–663 (2014).
Huang, P. Y. et al. Imaging atomic rearrangements in two-dimensional silica glass: watching silica’s dance. Science 342, 224–227 (2013).
Pereira, F., Stüer, H., Graff, E. C. & Gharib, M. Two-frame 3D particle tracking. Meas. Sci. Technol. 17, 1680 (2006).
Leggett, S. E. et al. Mechanophenotyping of 3D multicellular clusters using displacement arrays of rendered tractions. Proc. Natl Acad. Sci. USA 117, 5655–5663 (2020).
Raffel, M. et al. Particle Image Velocimetry: A Practical Guide (Springer, 2018).
Cierpka, C., Lütke, B. & Kähler, C. J. Higher order multi-frame particle tracking velocimetry. Exp. Fluids 54, 1533 (2013).
Brunton, S. L., Noack, B. R. & Koumoutsakos, P. Machine learning for fluid mechanics. Ann. Rev. Fluid Mech. 52, 477–508 (2020).
Cai, S., Zhou, S., Xu, C. & Gao, Q. Dense motion estimation of particle images via a convolutional neural network. Exp. Fluids 60, 73 (2019).
Cai, S., Liang, J., Gao, Q., Xu, C. & Wei, R. Particle image velocimetry based on a deep learning motion estimator. IEEE Trans. Instrum. Meas. 69, 3538–3554 (2019).
Liang, J., Cai, S., Xu, C. & Chu, J. Filtering enhanced tomographic PIV reconstruction based on deep neural networks. IET Cyber-Syst. Robot. 2, 43–52 (2020).
Lagemann, C., Lagemann, K., Mukherjee, S. & Schröder, W. Deep recurrent optical flow learning for particle image velocimetry data. Nat. Mach. Intell. 3, 641–651 (2021).
Gim, Y., Jang, D. K., Sohn, D. K., Kim, H. & Ko, H. S. Three-dimensional particle tracking velocimetry using shallow neural network for real-time analysis. Exp. Fluids 61, 26 (2020).
Mallery, K., Shao, S. & Hong, J. Dense particle tracking using a learned predictive model. Exp. Fluids 61, 223 (2020).
Qi, C. R., Su, H., Mo, K. & Guibas, L. J. Pointnet: deep learning on point sets for 3D classification and segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 652–660 (IEEE, 2017).
Liu, X., Qi, C. R. & Guibas, L. J. FlowNet3D: learning scene flow in 3D point clouds. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 529–537 (IEEE, CVF, 2019).
Puy, G., Boulch, A. & Marlet, R. FLOT: scene flow on point clouds guided by optimal transport. In European Conference on Computer Vision 527–544 (Springer, 2020).
Wu, W., Wang, Z. Y., Li, Z., Liu, W. & Fuxin, L. PointPWC-Net: cost volume on point clouds for (self-) supervised scene flow estimation. In European Conference on Computer Vision 88–107 (Springer, 2020).
Wei, Y., Wang, Z., Rao, Y., Lu, J. & Zhou, J. PV-RAFT: point-voxel correlation fields for scene flow estimation of point clouds. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 6954–6963 (IEEE, CVF, 2021).
Liang, J., Cai, S., Xu, C., Chen, T. & Chu, J. DeepPTV: particle tracking velocimetry for complex flow motion via deep neural networks.IEEE Trans. Instrum. Meas. 71, 9594090 (2021).
Wu, Z. et al. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32, 4–24 (2020).
Khojasteh, A. R., Laizet, S., Heitz, D. & Yang, Y. Lagrangian and Eulerian dataset of the wake downstream of a smooth cylinder at a Reynolds number equal to 3900. Data Brief 40, 107725 (2022).
Patel, M., Leggett, S. E., Landauer, A. K., Wong, I. Y. & Franck, C. Rapid, topology-based particle tracking for high-resolution measurements of large complex 3D motion fields. Sci. Rep. 8, 5581 (2018).
Yang, J. et al. SerialTrack: ScalE and rotation invariant augmented Lagrangian particle tracking. SoftwareX 19, 101204 (2022).
Teed, Z. & Deng, J. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision 402–419 (Springer, 2020).
Sanchez-Gonzalez, A., et al. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning 8459–8468 (PMLR, 2020).
Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A. & Battaglia, P. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations 2837 (ICLR, 2021).
Schanz, D., Gesemann, S. & Schröder, A. Shake-The-Box: Lagrangian particle tracking at high particle image densities. Exp. Fluids 57, 70 (2016).
Tan, S., Salibindla, A., Masuk, A. U. M. & Ni, R. Introducing OpenLPT: new method of removing ghost particles and high-concentration particle shadow tracking. Exp. Fluids 61, 47 (2020).
Cornic, P. et al. Double-frame tomographic PTV at high seeding densities. Exp. Fluids 61, 23 (2020).
Qi, C. R., Yi, L., Su, H. & Guibas, L. J. PointNet++: deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems Vol. 30 (NeurIPS, 2017).
Wang, Y. et al. Dynamic graph CNN for learning on point clouds. ACM Trans. Graphics 38, 1–12 (2019).
Peyré, G. et al. Computational optimal transport: with applications to data science. Found. Trends Mach. Learn. 11, 355–607 (2019).
Cuturi, M. Sinkhorn distances: lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems Vol. 26 (NeurIPS, 2013).
Li, Y. et al. A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. J. Turbulence 9, N31 (2008).
Ethier, C. R. & Steinman, D. Exact fully 3D Navier–Stokes solutions for benchmarking. Int. J. Numer. Methods Fluids 19, 369–375 (1994).
Maas, H., Gruen, A. & Papantoniou, D. Particle tracking velocimetry in three-dimensional flows. Exp. Fluids 15, 133–146 (1993).
Bhattacharya, S. & Vlachos, P. P. Volumetric particle tracking velocimetry (PTV) uncertainty quantification. Exp. Fluids 61, 197 (2020).
Atkinson, C. & Soria, J. An efficient simultaneous reconstruction technique for tomographic particle image velocimetry. Exp. Fluids 47, 553–568 (2009).
Scimone, M. T. et al. Modular approach for resolving and mapping complex neural and other cellular structures and their associated deformation fields in three dimensions. Nat. Protocols 13, 3042–3064 (2018).
This work was supported in part by: the National Key R&D Program of China (grant no. 2019YFB1705800); the National Natural Science Foundation of China (grant no. 61973270); and the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (grant no. 61621002). S.C. gratefully acknowledges the support of Zhejiang University Education Foundation Qizhen Scholar Foundation.
The authors declare no competing interests.
Peer review information
Nature Machine Intelligence thanks Steven Wereley and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended Data Fig. 1 Robustness assessments for different flow learning methods with respect to (a) displacement ratio ρ, (b) noise from particles correspondence, (c) noise from particles localization.
The Acc Strict, Acc Relax, and Outliers are demonstrated as the evaluation metrics here. The proposed GotFlow3D outperforms the others in all the assessments.
About this article
Cite this article
Liang, J., Xu, C. & Cai, S. Recurrent graph optimal transport for learning 3D flow motion in particle tracking. Nat Mach Intell 5, 505–517 (2023). https://doi.org/10.1038/s42256-023-00648-y
This article is cited by
Experiments in Fluids (2023)