Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Autonomous 3D positional control of a magnetic microrobot using reinforcement learning

Nature Machine Intelligence volume 6, pages 92–105 (2024)Cite this article

Subjects

Abstract

Magnetic microrobots have shown promise in the field of biomedical engineering, facilitating precise drug delivery, non-invasive diagnosis and cell-based therapy. Current techniques for controlling the motion of such microrobots rely on the assumption of homogenous magnetic fields and are significantly influenced by a microrobot’s properties and surrounding environment. These strategies lack a sense of generality and adaptability when changing the environment or microrobot and exhibit a moderate delay due to independent control of the electromagnetic actuation system and microrobot’s position. To address these issues, we propose a machine learning-based positional control of magnetic microrobots via gradient fields generated by electromagnetic coils. We use reinforcement learning and a gradual training approach to control the three-dimensional position of a microrobot within a defined working area by directly managing the coil currents. We develop a simulation environment for initial exploration to reduce the overall training time. After simulation training, the learning process is transferred to a physical electromagnetic actuation system that reflects real-world intricacies. We compare our method to conventional proportional-integral-derivative control; our system is more accurate and efficient. The proposed method was combined with path planning algorithms to allow fully autonomous control. The presented approach is an alternative to complex mathematical models, which are sensitive to variations in microrobot design, the environment and the nonlinearity of magnetic systems.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: RL-based magnetic MR navigation.
Fig. 2: Evaluation and training results in the simulation environment.
Fig. 3: RL agent retraining using the EAS.
Fig. 4: Comparison between our method and use of a PID controller for closed-loop control.
Fig. 5: Navigation of the MR in the brain vessel phantom.
Fig. 6: Fully autonomous control of an MR in various environments.

Similar content being viewed by others

Data availability

No public or custom dataset was used for this study. All the data required to replicate the results of this study are given in the main article, Supplementary Information and the GitHub repository.

Code availability

All the data were processed using custom codes. Custom codes along with source code for the simulation environment are available at the GitHub repository: https://github.com/sarmadnabbasi/Autonomous-3D-positional-control-of-a-magnetic-microrobot-using-reinforcement-learning (ref. 71).

References

  1. Wang, B., Kostarelos, K., Nelson, B. J. & Zhang, L. Trends in micro‐/nanorobotics: materials development, actuation, localization, and system integration for biomedical applications. Adv. Mater. 33, 2002047 (2021).

    Article  Google Scholar 

  2. Wang, B. et al. Endoscopy-assisted magnetic navigation of biohybrid soft microrobots with rapid endoluminal delivery and imaging. Sci. Robot 6, eabd2813 (2021).

    Article  Google Scholar 

  3. Erkoc, P. et al. Mobile microrobots for active therapeutic delivery. Adv. Ther. 2, 1800064 (2019).

    Article  Google Scholar 

  4. Nauber, R. et al. Medical microrobots in reproductive medicine from the bench to the clinic. Nat. Commun. 14, 728 (2023).

    Article  Google Scholar 

  5. Wang, Y. et al. Microrobots for targeted delivery and therapy in digestive system. ACS Nano 17, 27–50 (2023).

    Article  Google Scholar 

  6. Vikram Singh, A. & Sitti, M. Targeted drug delivery and imaging using mobile milli/microrobots: a promising future towards theranostic pharmaceutical design. Curr. Pharm. Des. 22, 1418–1428 (2016).

    Article  Google Scholar 

  7. Gao, W. et al. Artificial micromotors in the mouse’s stomach: a step toward in vivo use of synthetic motors. ACS Nano 9, 117–123 (2015).

    Article  Google Scholar 

  8. Li, J. et al. Development of a magnetic microrobot for carrying and delivering targeted cells. Sci. Robot 3, eaat8829 (2018).

    Article  Google Scholar 

  9. Zhang, H. et al. Dual-responsive biohybrid neutrobots for active target delivery. Sci. Robot 6, eaaz9519 (2021).

    Article  Google Scholar 

  10. Sitti, M. et al. Biomedical applications of untethered mobile milli/microrobots. Proc. IEEE 103, 205–224 (2015).

    Article  Google Scholar 

  11. Li, J., Mayorga‐Martinez, C. C., Ohl, C. & Pumera, M. Ultrasonically propelled micro‐ and nanorobots. Adv. Funct. Mater. 32, 2102265 (2022).

    Article  Google Scholar 

  12. Wang, J., Dong, R., Wu, H., Cai, Y. & Ren, B. A review on artificial micro/nanomotors for cancer-targeted delivery. Diagn.Ther. Nanomicro. Lett. 12, 11 (2020).

    Google Scholar 

  13. Hou, Y. et al. A review on microrobots driven by optical and magnetic fields. Lab. Chip https://doi.org/10.1039/D2LC00573E (2023).

  14. Sitti, M. & Wiersma, D. S. Pros and cons: magnetic versus optical microrobots. Adv. Mater. 32, 1906766 (2020).

    Article  Google Scholar 

  15. Zhou, H., Mayorga-Martinez, C. C., Pané, S., Zhang, L. & Pumera, M. Magnetically driven micro and nanorobots. Chem. Rev. 121, 4999–5041 (2021).

    Article  Google Scholar 

  16. Choi, J., Hwang, J., Kim, J. & Choi, H. Recent progress in magnetically actuated microrobots for targeted delivery of therapeutic agents. Adv. Healthc. Mater 10, 2001596 (2021).

    Article  Google Scholar 

  17. Ebrahimi, N. et al. Magnetic actuation methods in bio/soft robotics. Adv. Funct. Mater. 31, 2005137 (2021).

    Article  Google Scholar 

  18. Yan, X. et al. Multifunctional biohybrid magnetite microrobots for imaging-guided therapy. Sci. Robot 2, eaaq1155 (2017).

    Article  Google Scholar 

  19. Kim, S. et al. Fabrication and characterization of magnetic microrobots for three‐dimensional cell culture and targeted transportation. Adv. Mater. 25, 5863–5868 (2013).

    Article  Google Scholar 

  20. Zhang, L. et al. Artificial bacterial flagella: fabrication and magnetic control. Appl. Phys. Lett. 94, 064107 (2009).

    Article  Google Scholar 

  21. Wang, X. et al. 3D printed enzymatically biodegradable soft helical microswimmers. Adv. Funct. Mater. 28, 1804107 (2018).

    Article  Google Scholar 

  22. Xin, C. et al. Conical hollow microhelices with superior swimming capabilities for targeted cargo delivery. Adv. Mater. 31, 1808226 (2019).

    Article  Google Scholar 

  23. Ceylan, H. et al. 3D-printed biodegradable microswimmer for theranostic cargo delivery and release. ACS Nano 13, 3353–3362 (2019).

    Article  Google Scholar 

  24. Jeon, S. et al. Magnetically actuated microrobots as a platform for stem cell transplantation. Sci. Robot 4, eaav4317 (2019).

    Article  Google Scholar 

  25. Alapan, Y., Bozuyuk, U., Erkoc, P., Karacakol, A. C. & Sitti, M. Multifunctional surface microrollers for targeted cargo delivery in physiological blood flow. Sci. Robot 5, eaba5726c (2020).

    Article  Google Scholar 

  26. Khalil, I. S. M. et al. Swimming back and forth using planar flagellar propulsion at low Reynolds numbers. Adv. Sci. 5, 1700461 (2018).

    Article  Google Scholar 

  27. Khalil, I. S. M. et al. Independent actuation of two-tailed microrobots. IEEE Robot. Autom. Lett. 3, 1703–1710 (2018).

    Article  Google Scholar 

  28. Yang, Z. & Zhang, L. Magnetic actuation systems for miniature robots: a review. Adv. Intell. Syst. 2, 2000082 (2020).

    Article  Google Scholar 

  29. Chen, R., Folio, D. & Ferreira, A. Analysis and comparison of electromagnetic microrobotic platforms for biomedical applications. Appl. Sci. 12, 456 (2022).

    Article  Google Scholar 

  30. Hwang, J., Kim, J. & Choi, H. A review of magnetic actuation systems and magnetically actuated guidewire- and catheter-based microrobots for vascular interventions. Intell. Serv. Robot. 13, 1–14 (2020).

    Article  Google Scholar 

  31. Kummer, M. P. et al. OctoMag: an electromagnetic system for 5-DOF wireless micromanipulation. In Proc. 2010 IEEE International Conference on Robotics and Automation 1610–1616 (IEEE, 2010).

  32. Yang, L., Du, X., Yu, E., Jin, D. & Zhang, L. DeltaMag: an electromagnetic manipulation system with parallel mobile coils. In Proc. 2019 International Conference on Robotics and Automation (ICRA) 9814–9820 (IEEE, 2019).

  33. Ongaro, F., Pane, S., Scheggi, S. & Misra, S. Design of an electromagnetic setup for independent three-dimensional control of pairs of identical and nonidentical microrobots. IEEE Trans. Rob. 35, 174–183 (2019).

    Article  Google Scholar 

  34. Yang, L. & Zhang, L. Motion control in magnetic microrobotics: from individual and multiple robots to swarms. Annu. Rev. Control Robot. Auton. Syst. 4, 509–534 (2021).

    Article  Google Scholar 

  35. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  36. Saxe, A., Nelli, S. & Summerfield, C. If deep learning is the answer, what is the question? Nat. Rev. Neurosci. 22, 55–67 (2021).

    Article  Google Scholar 

  37. Yuan, X., Shi, J. & Gu, L. A review of deep learning methods for semantic segmentation of remote sensing imagery. Expert Syst. Appl. 169, 114417 (2021).

    Article  Google Scholar 

  38. Shamshirband, S., Fathi, M., Dehzangi, A., Chronopoulos, A. T. & Alinejad-Rokny, H. A review on deep learning approaches in healthcare systems: taxonomies, challenges, and open issues. J. Biomed. Inform. 113, 103627 (2021).

    Article  Google Scholar 

  39. Kiran, B. R. et al. Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transp. Syst. 23, 4909–4926 (2022).

    Article  Google Scholar 

  40. Justesen, N., Bontrager, P., Togelius, J. & Risi, S. Deep learning for video game playing. IEEE Trans Games 12, 1–20 (2020).

    Article  Google Scholar 

  41. Liu, R., Nageotte, F., Zanne, P., de Mathelin, M. & Dresp-Langley, B. Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review. Robotics 10, 22 (2021).

    Article  Google Scholar 

  42. Soria, E., Schiano, F. & Floreano, D. Predictive control of aerial swarms in cluttered environments. Nat. Mach. Intell. 3, 545–554 (2021).

    Article  Google Scholar 

  43. Cichos, F., Gustavsson, K., Mehlig, B. & Volpe, G. Machine learning for active matter. Nat. Mach. Intell. 2, 94–103 (2020).

    Article  Google Scholar 

  44. Sutton, R. S. & Barto, A. G. Reinforcement learning: an introduction. IEEE Trans. Neural Netw. 9, 1054–1054 (1998).

    Article  Google Scholar 

  45. Tsang, A. C. H., Demir, E., Ding, Y. & Pak, O. S. Roads to smart artificial microswimmers. Adv. Intell. Syst. 2, 1900137 (2020).

    Article  Google Scholar 

  46. Yu, S., Cai, Y., Wu, Z. & He, Q. Recent progress on motion control of swimming micro/nanorobots. VIEW 2, 20200113 (2021).

    Article  Google Scholar 

  47. Jiang, J., Yang, Z., Ferreira, A. & Zhang, L. Control and autonomy of microrobots: recent progress and perspective. Adv. Intell. Syst. 4, 2100279 (2022).

    Article  Google Scholar 

  48. Yang, Y., Bevan, M. A. & Li, B. Efficient navigation of colloidal robots in an unknown environment via deep reinforcement learning. Adv. Intell. Syst. 2, 1900106 (2020).

    Article  Google Scholar 

  49. Muiños-Landin, S., Fischer, A., Holubec, V. & Cichos, F. Reinforcement learning with artificial microswimmers.Sci. Robot. 6, eabd9285 (2021).

    Article  Google Scholar 

  50. Liu, Y., Zou, Z., Tsang, A. C. H., Pak, O. S. & Young, Y.-N. Mechanical rotation at low Reynolds number via reinforcement learning. Phys. Fluids 33, 062007 (2021).

    Article  Google Scholar 

  51. Yang, Y., Bevan, M. A. & Li, B. Hierarchical planning with deep reinforcement learning for 3D navigation of microrobots in blood vessels. Adv. Intell. Syst. 4, 2200168 (2022).

    Article  Google Scholar 

  52. Yang, Y., Bevan, M. A. & Li, B. Micro/nano motor navigation and localization via deep reinforcement learning. Adv. Theory Simul. 3, 2000034 (2020).

    Article  Google Scholar 

  53. Yang, L. et al. Autonomous environment-adaptive microrobot swarm navigation enabled by deep learning-based real-time distribution planning. Nat. Mach. Intell. 4, 480–493 (2022).

    Article  MathSciNet  Google Scholar 

  54. Yang, L., Yu, J. & Zhang, L. A mobile paramagnetic nanoparticle swarm with automatic shape deformation control. In Proc. 2020 IEEE International Conference on Robotics and Automation (ICRA) 9230–9236 (IEEE, 2020).

  55. Yang, L., Yu, J. & Zhang, L. Statistics-based automated control for a swarm of paramagnetic nanoparticles in 2-D space. IEEE Trans. Rob. 36, 254–270 (2020).

    Article  Google Scholar 

  56. Behrens, M. R. & Ruder, W. C. Smart magnetic microrobots learn to swim with deep reinforcement learning. Adv. Intell. Syst. 4, 2200023 (2022).

    Article  Google Scholar 

  57. Cai, M. et al. Deep reinforcement learning framework-based flow rate rejection control of soft magnetic miniature robots. IEEE Trans. Cybern. 53, 7699–7711 (2022).

  58. Ghanbari, A., Chang, P. H., Nelson, B. J. & Choi, H. Electromagnetic steering of a magnetic cylindrical microrobot using optical feedback closed-loop control. Int. J. Optomechatronics 8, 129–145 (2014).

    Article  Google Scholar 

  59. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal policy optimization algorithms. Preprint at https://arxiv.org/abs/1707.06347 (2017).

  60. Raffin, A. RL Baselines3 Zoo. GitHub https://github.com/DLR-RM/rl-baselines3-zoo (2020).

  61. Kim, J., Choi, H. & Kim, J. A robust motion control with antiwindup scheme for electromagnetic actuated microrobot using time-delay estimation. IEEE/ASME Trans. Mechatron. 24, 1096–1105 (2019).

    Article  Google Scholar 

  62. Hart, P., Nilsson, N. & Raphael, B. A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern. 4, 100–107 (1968).

    Article  Google Scholar 

  63. Stentz, A. Optimal and efficient path planning for partially-known environments. In Proc. IEEE International Conference on Robotics and Automation 3310–3317 (IEEE Comput. Soc., 1994).

  64. Fan, X. et al. Automated noncontact micromanipulation using magnetic swimming microrobots. IEEE Trans. Nanotechnol. 17, 666–669 (2018).

    Article  Google Scholar 

  65. Liu, J. et al. 3-D autonomous manipulation system of helical microswimmers with online compensation update. IEEE Trans. Autom. Sci. Eng. 18, 1380–1391 (2021).

    Article  MathSciNet  Google Scholar 

  66. Juliani, A. et al. Unity: a general platform for intelligent agents. Preprint at https://arxiv.org/abs/1809.02627 (2018).

  67. Ahmed, A., Abbasi, S. A., Gharamaleki, N. L., Kim, J.-Y. & Choi, H. MagMaps: an economical, Lego-like approach for real-time magnetic field mapping using commercial magnetic sensors. IEEE Trans. Instrum. Meas. 72, 9500109 (2023).

    Article  Google Scholar 

  68. Bradski, G. The openCV library. Dr. Dobb’s J. Softw. Tools Prof. Program. 25, 120–123 (2000).

    Google Scholar 

  69. Raffin, A. et al. Stable-Baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res. 22, 1–8 (2021).

    Google Scholar 

  70. Brockman, G. et al. OpenAI gym. Preprint at https://arxiv.org/abs/1606.01540 (2016).

  71. Abbasi, S. et al. Autonomous 3D positional control of a magnetic microrobot using reinforcement learning (sample codes). Zenodo https://doi.org/10.5281/zenodo.10200117 (2023).

Download references

Acknowledgements

Funding: this work was financially supported by the National Convergence Research of Scientific Challenges through the National Research Foundation of Korea and the DGIST R&D Program (grant nos. 2021M3F7A1082275 and 23-CoE-BT-02) funded by the Ministry of Science and ICT. S.P. and B.J.N. thank the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 952152, project ANGIE (magnetically steerable wireless nanodevices for the targeted delivery of therapeutic agents in any vascular region of the body) for kindly supporting our research.

Author information

Authors and Affiliations

  1. Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu, South Korea

    Sarmad Ahmad Abbasi, Awais Ahmed, Seungmin Noh, Nader Latifi Gharamaleki, Seonhyoung Kim, A. M. Masum Bulbul Chowdhury, Jin-young Kim & Hongsoo Choi

  2. DGIST-ETH Microrobotics Research Center, DGIST, Daegu, South Korea

    Sarmad Ahmad Abbasi, Awais Ahmed, Seungmin Noh, Nader Latifi Gharamaleki, Seonhyoung Kim, A. M. Masum Bulbul Chowdhury, Jin-young Kim, Bradley J. Nelson & Hongsoo Choi

  3. Multi-Scale Robotics Laboratory, Institute of Robotics and Intelligent Systems, ETH Zurich, Zurich, Switzerland

    Salvador Pané & Bradley J. Nelson

Authors
  1. Sarmad Ahmad Abbasi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Awais Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seungmin Noh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nader Latifi Gharamaleki
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Seonhyoung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. A. M. Masum Bulbul Chowdhury
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jin-young Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Salvador Pané
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Bradley J. Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Hongsoo Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.A.A. and H.C. conceived the idea. S.A.A., A.A. and S.K. designed the experiments. S.A.A. wrote the code, integrated the physical and simulation setup, and performed the experiments. S.A.A. and H.C. worked on the analysis of data. S.N. characterized the MR. N.L.G. and A.A. contributed to the hardware setup. A.M.M.B.C. and S.K. helped with the revision of the manuscript. S.A.A., A.A., S.N., N.L.G., J.-Y.K., S.P., B.J.N. and H.C. contributed to the manuscript.

Corresponding author

Correspondence to Hongsoo Choi.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks the anonymous reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Camera feeds for trajectories, fluid flow, and mean episode reward.

a and b, Camera feeds for the trajectories of Fig. 3. c, Flow imparted by the peristaltic pump. d, Mean episode reward after training in fluid flow.

Extended Data Fig. 2 Camera feeds for trajectories of path planning and dynamic obstacle avoidance.

a, Camera feeds for trajectories in Fig. 6b. b, Camera feed for trajectories in Fig. 6c. c and d, camera feed at different intervals for dynamic obstacle avoidance in Fig. 6h and i, respectively.

Supplementary information

Supplementary Information

Supplementary Figs. 1–11 and Table 1.

Reporting Summary

Supplementary Video 1

Simulation environment evaluation.

Supplementary Video 2

3D position control training in simulation.

Supplementary Video 3

3D position control training in EAS.

Supplementary Video 4

Magnetic microrobot navigation in open space via RL.

Supplementary Video 5

Dynamic fluid flow environment, training and navigation.

Supplementary Video 6

Comparison between navigation with PID and RL.

Supplementary Video 7

Navigation in MCA phantom.

Supplementary Video 8

Path planning and navigation of the magnetic microrobot.

Supplementary Video 9

Navigation around dynamic obstacles.

Supplementary Video 10

Training and navigation of DMR.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abbasi, S.A., Ahmed, A., Noh, S. et al. Autonomous 3D positional control of a magnetic microrobot using reinforcement learning. Nat Mach Intell 6, 92–105 (2024). https://doi.org/10.1038/s42256-023-00779-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-023-00779-2

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing