Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Autonomous environment-adaptive microrobot swarm navigation enabled by deep learning-based real-time distribution planning

Nature Machine Intelligence volume 4, pages 480–493 (2022)Cite this article

Subjects

Abstract

Navigating a large swarm of micro-/nanorobots is critical for potential targeted delivery/therapy applications owing to the limited volume/function of a single microrobot, and microrobot swarms with distribution reconfigurability can adapt to environments during navigation. However, current microrobot swarms lack the intelligent behaviour to autonomously adjust their distribution and motion according to environmental change. Such autonomous navigation is challenging, and requires real-time appropriate decision-making capability of the swarm for unknown and unstructured environments. Here, to tackle this issue, we propose a framework that defines different autonomy levels for environment-adaptive microrobot swarm navigation and designs corresponding system components for each level. To realize high autonomy levels, real-time autonomous distribution planning is a key capability for the swarm, regarding which we show that deep learning is an enabling approach that allows the microrobot swarm to learn optimal distributions in extensive unstructured environmental morphologies. For real-world demonstration, we study the reconfigurable magnetic nanoparticle swarm and experimentally demonstrate autonomous swarm navigation for targeted delivery and cargo transport in environments with channels or obstacles. This work could introduce computational intelligence to micro-/nanorobot swarms, enabling them to autonomously make appropriate decisions during navigation in unstructured environments.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Autonomy levels of environment-adaptive microrobot swarm navigation.
Fig. 2: The real-time autonomous swarm distribution planning method based on DL.
Fig. 3: The reconfigurable magnetic nanoparticle swarm and its adaptive navigation for task execution.
Fig. 4: Experimental demonstrations of navigation autonomy levels from 0 to 2.
Fig. 5: Task execution using autonomy level 3.
Fig. 6: Experimental demonstrations of autonomy level 4.

Similar content being viewed by others

Data availability

The dataset49 (~20 GB) used for training the DNNs is available in figshare (https://doi.org/10.6084/m9.figshare.19149779.v1).

Code availability

All the control and planning algorithms used in this study are available within the Article. Original codes for training the DNNs, sample codes for executing the swarm distribution planning in different environmental morphologies and sample codes50 for executing the trajectory planning in a channel environment are available on GitHub (https://github.com/lidongYang22/Autonomous-microrobot-swarm-navigation) and Zenodo (https://doi.org/10.5281/zenodo.6032452). The integrated software used for experimental validation is available from the corresponding author on reasonable request.

References

  1. Floreano, D. & Lipson, H. From individual robots to robot societies. Sci. Robot. 6, eabk2787 (2021).

  2. Rubenstein, M., Cornejo, A. & Nagpal, R. Programmable self-assembly in a thousand-robot swarm. Science 345, 795–799 (2014).

    Article  Google Scholar 

  3. Soria, E., Schiano, F. & Floreano, D. Predictive control of aerial swarms in cluttered environments. Nat. Mach. Intell. 3, 545–554 (2021).

    Article  Google Scholar 

  4. Li, S. et al. Particle robotics based on statistical mechanics of loosely coupled components. Nature 567, 361–365 (2019).

    Article  Google Scholar 

  5. Shahrokhi, S., Lin, L., Ertel, C., Wan, M. & Becker, A. T. Steering a swarm of particles using global inputs and swarm statistics. IEEE Trans. Robot. 34, 207–219 (2017).

    Article  Google Scholar 

  6. Ozkan-Aydin, Y. & Goldman, D. I. Self-reconfigurable multilegged robot swarms collectively accomplish challenging terradynamic tasks. Sci. Robot. 6, eabf1628 (2021).

  7. Dorigo, M., Theraulaz, G. & Trianni, V. Reflections on the future of swarm robotics. Sci. Robot. 5, eabe4385 (2020).

  8. Ahmed, D. et al. Bioinspired acousto-magnetic microswarm robots with upstream motility. Nat. Mach. Intell. 3, 116–124 (2021).

    Article  Google Scholar 

  9. Wu, C. et al. Ion-exchange enabled synthetic swarm. Nat. Nanotechnol. 16, 288–295 (2021).

    Article  Google Scholar 

  10. Dong, X. & Sitti, M. Controlling two-dimensional collective formation and cooperative behavior of magnetic microrobot swarms. Int. J. Robot. Res. 39, 617–638 (2020).

    Article  Google Scholar 

  11. Yang, L. et al. A survey on swarm microrobotics. IEEE Trans. Robot. https://doi.org/10.1109/TRO.2021.3111788 (2021).

  12. Felfoul, O. et al. Magneto-aerotactic bacteria deliver drug-containing nanoliposomes to tumour hypoxic regions. Nat. Nanotechnol. 11, 941–947 (2016).

    Article  Google Scholar 

  13. Wu, Z. et al. A swarm of slippery micropropellers penetrates the vitreous body of the eye. Sci. Adv. 4, eaat4388 (2018).

    Article  Google Scholar 

  14. Martel, S. & Mohammadi, M. Using a swarm of self-propelled natural microrobots in the form of flagellated bacteria to perform complex micro-assembly tasks. In 2010 IEEE International Conference on Robotics and Automation 500–505 (IEEE, 2010).

  15. Dekanovsky, L. et al. Chemically programmable microrobots weaving a web from hormones. Nat. Mach. Intell. 2, 711–718 (2020).

    Article  Google Scholar 

  16. Li, J., de Ávila, B. E.-F., Gao, W., Zhang, L. & Wang, J. Micro/nanorobots for biomedicine: delivery, surgery, sensing, and detoxification. Sci. Robot. 2, eaam6431 (2017).

  17. Servant, A., Qiu, F., Mazza, M., Kostarelos, K. & Nelson, B. J. Controlled in vivo swimming of a swarm of bacteria-like microrobotic flagella. Adv. Mater. 27, 2981–2988 (2015).

    Article  Google Scholar 

  18. Hortelao, A. C. et al. Swarming behavior and in vivo monitoring of enzymatic nanomotors within the bladder. Sci. Robot. 6, eabd2823 (2021).

  19. Wang, Q. et al. Ultrasound Doppler-guided real-time navigation of a magnetic microswarm for active endovascular delivery. Sci. Adv. 7, eabe5914 (2021).

    Article  Google Scholar 

  20. Miskin, M. Z. et al. Electronically integrated, mass-manufactured, microscopic robots. Nature 584, 557–561 (2020).

    Article  Google Scholar 

  21. Cheng, R. et al. Acceleration of tissue plasminogen activator-mediated thrombolysis by magnetically powered nanomotors. ACS Nano 8, 7746–7754 (2014).

    Article  Google Scholar 

  22. Hwang, G. et al. Catalytic antimicrobial robots for biofilm eradication. Sci. Robot. 4, eaaw2388 (2019).

  23. Melde, K., Mark, A. G., Qiu, T. & Fischer, P. Holograms for acoustics. Nature 537, 518–522 (2016).

    Article  Google Scholar 

  24. Zhou, Z., Hou, Z. & Pei, Y. Reconfigurable particle swarm robotics powered by acoustic vibration tweezer. Soft Robot. 8, 735–743 (2020).

  25. Yu, J., Yang, L. & Zhang, L. Pattern generation and motion control of a vortex-like paramagnetic nanoparticle swarm. Int. J. Robot. Res. 37, 912–930 (2018).

    Article  Google Scholar 

  26. Xie, H. et al. Reconfigurable magnetic microrobot swarm: multimode transformation, locomotion, and manipulation. Sci. Robot. 4, eaav8006 (2019).

  27. Kaiser, A., Snezhko, A. & Aranson, I. S. Flocking ferromagnetic colloids. Sci. Adv. 3, e1601469 (2017).

    Article  Google Scholar 

  28. Loghin, D., Tremblay, C., Mohammadi, M. & Martel, S. Exploiting the responses of magnetotactic bacteria robotic agents to enhance displacement control and swarm formation for drug delivery platforms. Int. J. Robot. Res. 36, 1195–1210 (2017).

    Article  Google Scholar 

  29. Yu, J., Wang, B., Du, X., Wang, Q. & Zhang, L. Ultra-extensible ribbon-like magnetic microswarm. Nat. Commun. 9, 3260 (2018).

    Article  Google Scholar 

  30. Kokot, G. & Snezhko, A. Manipulation of emergent vortices in swarms of magnetic rollers. Nat. Commun. 9, 2344 (2018).

    Article  Google Scholar 

  31. Yigit, B., Alapan, Y. & Sitti, M. Programmable collective behavior in dynamically self-assembled mobile microrobotic swarms. Adv. Sci. 6, 1801837 (2019).

    Article  Google Scholar 

  32. Liang, X. et al. Hierarchical microswarms with leader–follower-like structures: electrohydrodynamic self-organization and multimode collective photoresponses. Adv. Funct. Mater. 30, 1908602 (2020).

    Article  Google Scholar 

  33. Hong, Y., Diaz, M., Córdova-Figueroa, U. M. & Sen, A. Light-driven titanium-dioxide-based reversible microfireworks and micromotor/micropump systems. Adv. Funct. Mater. 20, 1568–1576 (2010).

    Article  Google Scholar 

  34. Yang, L., Yu, J. & Zhang, L. A mobile paramagnetic nanoparticle swarm with automatic shape deformation control. In 2020 IEEE International Conference on Robotics and Automation (ICRA) 9230–9236 (IEEE, 2020).

  35. Sitti, M. Physical intelligence as a new paradigm. Extreme Mech. Lett. 46, 101340 (2021).

    Article  Google Scholar 

  36. Cichos, F., Gustavsson, K., Mehlig, B. & Volpe, G. Machine learning for active matter. Nat. Mach. Intell. 2, 94–103 (2020).

    Article  Google Scholar 

  37. Kaspar, C., Ravoo, B., van der Wiel, W., Wegner, S. & Pernice, W. The rise of intelligent matter. Nature 594, 345–355 (2021).

    Article  Google Scholar 

  38. Mamdani, E. H. & Assilian, S. An experiment in linguistic synthesis with a fuzzy logic controller. Int. J. Man–Mach. Stud. 7, 1–13 (1975).

    Article  Google Scholar 

  39. Yang, L., Yu, J. & Zhang, L. Statistics-based automated control for a swarm of paramagnetic nanoparticles in 2-D space. IEEE Trans. Robot. 36, 254–270 (2019).

    Article  Google Scholar 

  40. Muiños-Landin, S., Fischer, A., Holubec, V. & Cichos, F. Reinforcement learning with artificial microswimmers. Sci. Robot. 6, eabd9285 (2021).

  41. Karaman, S. & Frazzoli, E. Sampling-based algorithms for optimal motion planning. Int. J. Robot. Res. 30, 846–894 (2011).

    Article  Google Scholar 

  42. Yu, J., Xu, T., Lu, Z., Vong, C. I. & Zhang, L. On-demand disassembly of paramagnetic nanoparticle chains for microrobotic cargo delivery. IEEE Trans. Robot. 33, 1213–1225 (2017).

    Article  Google Scholar 

  43. Pérez-Higueras, N., Jardón, A., Rodríguez, Á. & Balaguer, C. 3D exploration and navigation with optimal-RRT planners for ground robots in indoor incidents. Sensors 20, 220 (2020).

    Article  Google Scholar 

  44. Singh, A. Deep learning 3D structures. Nat. Methods 17, 249–249 (2020).

    Article  Google Scholar 

  45. Karimi, D. et al. Accurate and robust deep learning-based segmentation of the prostate clinical target volume in ultrasound images. Med. Image Anal. 57, 186–196 (2019).

    Article  Google Scholar 

  46. Chen, C. et al. Deep learning for cardiac image segmentation: a review. Front. Cardiovasc. Med. 7, 25 (2020).

    Article  Google Scholar 

  47. Dou, Q., Coelho de Castro, D., Kamnitsas, K. & Glocker, B. Domain generalization via model-agnostic learning of semantic features. Adv. Neural Inf. Process. Syst. 32, 6450–6461 (2019).

    Google Scholar 

  48. Julien, D. 2D minimal bounding box. MATLAB Central File Exchange https://www.mathworks.com/matlabcentral/fileexchange/31126-2d-minimal-bounding-box (2011).

  49. Yang, L. et al. Autonomous environment-adaptive microrobot swarm navigation enabled by deep learning-based real-time distribution planning (dataset). Figshare https://doi.org/10.6084/m9.figshare.19149779.v1 (2022).

  50. Yang, L. et al. Autonomous environment-adaptive microrobot swarm navigation enabled by deep learning-based real-time distribution planning (sample codes). Zenodo https://doi.org/10.5281/zenodo.6032452 (2022).

Download references

Acknowledgements

We would like to thank S. Yang and D. Jin for help with the ultrasound experiments, T. Lam and K. Lai for help with the X-ray fluoroscopy experiments and K.-F. Chan for applying for permission for the navigation experiments in human placenta. We also would like to thank Prof. Ben M. Chen for the fruitful discussion.This project has received funding support from the Hong Kong Research Grants Council (E-CUHK401/20)(to L.Z.), the ITF project MRP/036/18X (to L.Z.), the Croucher Foundation grant CAS20403 (to L.Z.), CUHK internal grants (to L.Z.), the Multi-Scale Medical Robotics Center (MRC), InnoHK, at the Hong Kong Science Park (to L.Z.), the SIAT-CUHK Joint Laboratory of Robotics and Intelligent Systems (to L.Z.) and the CUHK Shun Hing Institute of Advanced Engineering (MMT-p5-20) (to Q.D.).

Author information

Author notes

  1. These authors contributed equally: Lidong Yang, Jialin Jiang.

Authors and Affiliations

  1. Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China

    Lidong Yang, Jialin Jiang, Qinglong Wang & Li Zhang

  2. Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China

    Xiaojie Gao & Qi Dou

  3. T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong, China

    Qi Dou & Li Zhang

  4. Multi-Scale Medical Robotics Center, Hong Kong Science Park, Hong Kong, China

    Qi Dou & Li Zhang

  5. Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China

    Li Zhang

  6. Chow Yuk Ho Technology Centre for Innovative Medicine, The Chinese University of Hong Kong, Hong Kong, China

    Li Zhang

Authors
  1. Lidong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jialin Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaojie Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qinglong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qi Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Li Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

L.Y. conceived the study. L.Y. and J.J. designed and implemented the system hardware and software, and performed the experiments. J.J., X.G. and Q.D. designed and coded the DL algorithms. Q.W. conducted the X-ray fluoroscopy experiment. L.Y. and J.J. wrote the manuscript with contributions from all authors. All authors contributed to the scientific discussion. L.Z. supervised the project.

Corresponding author

Correspondence to Li Zhang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Eric Diller and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes 1–10, Figs. 1–18, Table 1, Video legends 1–12 and refs. 1–3.

Reporting Summary

Supplementary Video 1

Robustness validation of the DNN-based swarm distribution planning method for different environmental morphologies, and comparison of computation time between this method and the traversal-based optimization method.

Supplementary Video 2

Experimental demonstration of the three swarm configurations and the transformations among them.

Supplementary Video 3

Experimental demonstration of the translational motion and rotational motion of the RS.

Supplementary Video 4

Experimental comparisons between manual navigation (autonomy level 0) and automated control (autonomy level 1).

Supplementary Video 5

Experimental demonstration of autonomy level 2, where two sets of experiments with different swarm sizes were conducted to assess the intelligence of the DNN-based swarm distribution planning method.

Supplementary Video 6

Experimental demonstration of autonomy level 3, where a swarm of magnetic nanorobots accomplished the delivery task to a targeted region in a channel environment.

Supplementary Video 7

Experimental demonstration of autonomy level 3, where the reconfigurable magnetic nanorobot swarm navigated in highly curved space with a 150° sharp turn and in a curved narrow channel environment.

Supplementary Video 8

Experimental demonstration of autonomy level 3, where the reconfigurable magnetic nanorobot swarm executed the cooperative micromanipulation task in confined space.

Supplementary Video 9

Experimental demonstration of the fully autonomous environment exploration using the magnetic nanorobot swarm in an unknown channel environment (autonomy level 4).

Supplementary Video 10

Experimental demonstration of the fully autonomous targeted delivery to a region with dynamic obstacles (autonomy level 4).

Supplementary Video 11

Transfer of the autonomy framework to the elliptical vortex-like magnetic nanoparticle swarm.

Supplementary Video 12

Method validation under ultrasound imaging and X-ray fluoroscopy.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, L., Jiang, J., Gao, X. et al. Autonomous environment-adaptive microrobot swarm navigation enabled by deep learning-based real-time distribution planning. Nat Mach Intell 4, 480–493 (2022). https://doi.org/10.1038/s42256-022-00482-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00482-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics