Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Deep reinforcement learning-designed radiofrequency waveform in MRI

Nature Machine Intelligence volume 3, pages 985–994 (2021)Cite this article

Subjects

A preprint version of the article is available at arXiv.

Abstract

Carefully engineered radiofrequency (RF) pulses play a key role in a number of systems such as mobile phone, radar and magnetic resonance imaging. The design of an RF waveform, however, is often posed as an inverse problem with no general solution. As a result, various design methods, each with a specific purpose, have been developed on the basis of the intuition of human experts. In this work we propose an artificial intelligence (AI)-powered RF pulse design framework, DeepRF, which utilizes the self-learning characteristics of deep reinforcement learning to generate a novel RF pulse. The effectiveness of DeepRF is demonstrated using four types of RF pulses that are commonly used. The DeepRF-designed pulses successfully satisfy the design criteria while reporting reduced energy. Analyses demonstrate the pulses utilize new mechanisms of magnetization manipulation, suggesting the potentials of DeepRF in discovering unseen design dimensions beyond human intuition. This work may lay the foundation for an emerging field of AI-driven RF waveform design.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The design process of DeepRF.
Fig. 2: The results of the slice-selective excitation pulses and inversion pulses.
Fig. 3: The results of the B1-insensitive volume inversion pulses and selective inversion pulses.
Fig. 4: The analysis results of the DeepRF pulses.
Fig. 5: The comparison results between the SLR, OC and DeepRF pulses in the slice-selective excitation and inversion RF designs.

Similar content being viewed by others

Data availability

All processed data shown in the figures and table are available at https://github.com/SNU-LIST/DeepRF/tree/master/data.

Code availability

The source code of DeepRF is available at https://github.com/SNU-LIST/DeepRF (ref. 50).

References

  1. Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).

    Article  Google Scholar 

  2. Wu, Y. et al. Google’s neural machine translation system: bridging the gap between human and machine translation. Preprint at https://arxiv.org/abs/1609.08144 (2016).

  3. Yu, J. et al. Generative image inpainting with contextual attention. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 5505–5514 (IEEE, 2018).

  4. Segler, M. H., Preuss, M. & Waller, M. P. Planning chemical syntheses with deep neural networks and symbolic AI. Nature 555, 604–610 (2018).

    Article  Google Scholar 

  5. Runge, F., Stoll, D., Falkner, S. & Hutter, F. Learning to design RNA. In Proc. International Conference on Learning Representations (2019).

  6. Wang, H., Yang, J., Lee, H.-S. & Han, S. Learning to design circuits. In Proc. Conference on Neural Information Processing Systems (2018).

  7. Popova, M., Isayev, O. & Tropsha, A. Deep reinforcement learning for de novo drug design. Sci. Adv. 4, aap7885 (2018).

  8. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 2018).

  9. Bell, M. R. Information theory and radar waveform design. IEEE Trans. Inf. Theory 39, 1578–1597 (1993).

    Article  Google Scholar 

  10. Simpson, D. H., Chin, C. T. & Burns, P. N. Pulse inversion Doppler: a new method for detecting nonlinear echoes from microbubble contrast agents. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 46, 372–382 (1999).

    Article  Google Scholar 

  11. Tarantola, A. Inverse Problem Theory and Methods for Model Parameter Estimation (SIAM, 2005).

  12. Bloch, F. Nuclear induction. Phys. Rev. 70, 460 (1946).

    Article  Google Scholar 

  13. Pauly, J., Le Roux, P., Nishimura, D. & Macovski, A. Parameter relations for the Shinnar–Le Roux selective excitation pulse design algorithm. IEEE Trans. Med. Imaging 10, 53–65 (1991).

    Article  Google Scholar 

  14. Abragam, A. The Principles of Nuclear Magnetism (Oxford Univ. Press, 1961).

  15. Conolly, S., Nishimura, D. & Macovski, A. Optimal control solutions to the magnetic resonance selective excitation problem. IEEE Trans. Med. Imaging 5, 106–115 (1986).

    Article  Google Scholar 

  16. Khaneja, N., Reiss, T., Kehlet, C., Schulte-Herbrüggen, T. & Glaser, S. J. Optimal control of coupled spin dynamics: design of NMR pulse sequences by gradient ascent algorithms. J. Magn. Reson. 172, 296–305 (2005).

    Article  Google Scholar 

  17. Rosenfeld, D. & Zur, Y. Design of adiabatic selective pulses using optimal control theory. Magn. Reson. Med. 36, 401–409 (1996).

    Article  Google Scholar 

  18. Rund, A., Aigner, C. S., Kunisch, K. & Stollberger, R. Magnetic resonance RF pulse design by optimal control with physical constraints. IEEE Trans. Med. Imaging 37, 461–472 (2017).

    Article  Google Scholar 

  19. Rund, A., Aigner, C. S., Kunisch, K. & Stollberger, R. Simultaneous multislice refocusing via time optimal control. Magn. Reson. Med. 80, 1416–1428 (2018).

    Article  Google Scholar 

  20. Xu, D., King, K. F., Zhu, Y., McKinnon, G. C. & Liang, Z. P. Designing multichannel, multidimensional, arbitrary flip angle RF pulses using an optimal control approach. Magn. Reson. Med. 59, 547–560 (2008).

    Article  Google Scholar 

  21. Vinding, M. S., Maximov, I. I., Tošner, Z. & Nielsen, N. C. Fast numerical design of spatial-selective RF pulses in MRI using Krotov and quasi-Newton based optimal control methods. J. Chem. Phys. 137, 054203 (2012).

    Article  Google Scholar 

  22. Vinding, M. S., Guérin, B., Vosegaard, T. & Nielsen, N. C. Local SAR, global SAR, and power‐constrained large‐flip‐angle pulses with optimal control and virtual observation points. Magn. Reson. Med. 77, 374–384 (2017).

    Article  Google Scholar 

  23. Loecher, M., Magrath, P., Aliotta, E. & Ennis, D. B. Time‐optimized 4D phase contrast MRI with real‐time convex optimization of gradient waveforms and fast excitation methods. Magn. Reson. Med. 82, 213–224 (2019).

    Article  Google Scholar 

  24. Shang, H. et al. Multiband RF pulses with improved performance via convex optimization. J. Magn. Reson. 262, 81–90 (2016).

    Article  Google Scholar 

  25. Vinding, M. S., Aigner, C. S., Schmitter, S. & Lund, T. E. DeepControl: 2D RF pulses facilitating B1+ inhomogeneity and B0 off-resonance compensation in vivo at 7T. Magn. Reson. Med. 85, 3308–3317 (2021).

  26. Vinding, M. S., Skyum, B., Sangill, R. & Lund, T. E. Ultrafast (milliseconds), multidimensional RF pulse design with deep learning. Magn. Reson. Med. 82, 586–599 (2019).

    Article  Google Scholar 

  27. Mirfin, C., Glover, P. & Bowtell, R. Optimisation of parallel transmission radiofrequency pulses using neural networks. In Proc. 26th Annual Meeting of ISMRM (2018).

  28. Zhang, Y. et al. Multi‐task convolutional neural network‐based design of radio frequency pulse and the accompanying gradients for magnetic resonance imaging. NMR Biomed. 34, e4443 (2021).

  29. Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. Deep Learning (MIT Press, 2016).

  30. Silver, M. S., Joseph, R. & Hoult, D. Highly selective π2 and π pulse generation. J. Magn. Reson. 59, 347–351 (1984).

    Google Scholar 

  31. Garwood, M. & DelaBarre, L. The return of the frequency sweep: designing adiabatic pulses for contemporary NMR. J. Magn. Reson. 153, 155–177 (2001).

    Article  Google Scholar 

  32. Tannús, A. & Garwood, M. Adiabatic pulses. NMR Biomed. 10, 423–434 (1997).

    Article  Google Scholar 

  33. Henderson, P. et al. Deep reinforcement learning that matters. In Proc. AAAI Conference on Artificial Intelligence (2018).

  34. Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. International Conference on Artificial Intelligence and Statistics 249–256 (2010).

  35. Vinyals, O. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019).

    Article  Google Scholar 

  36. Zhu, B., Liu, J. Z., Koonjoo, N., Rose, B. R. & Rosen, M. S. Automated pulse sequence generation (AUTOSEQ) using Bayesian reinforcement learning in an MRI physics simulation environment. In Proc. 26th Annual Meeting of ISMRM (2018).

  37. Walker-Samuel, S. Using deep reinforcement learning to actively, adaptively and autonomously control of a simulated MRI scanner. In Proc. 27th Annual Meeting of ISMRM (2019).

  38. David, Y. et al. Reinforcement learning for online undersampling pattern optimization. In Proc. 27th Annual Meeting of ISMRM (2019).

  39. Pineda, L., Basu, S., Romero, A., Calandra, R. & Drozdzal, M. Active MR k-space sampling with reinforcement learning. Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (2020).

  40. Bahadir, C. D., Wang, A. Q., Dalca, A. V. & Sabuncu, M. R. Deep-learning-based optimization of the under-sampling pattern in MRI. IEEE Trans. Comput. Imaging 6, 1139–1152 (2020).

    Article  Google Scholar 

  41. Meyer, C. H., Pauly, J. M., Macovskiand, A. & Nishimura, D. G. Simultaneous spatial and spectral selective excitation. Magn. Reson. Med. 15, 287–304 (1990).

    Article  Google Scholar 

  42. Yip, C. Y., Fessler, J. A. & Noll, D. C. Iterative RF pulse design for multidimensional, small‐tip‐angle selective excitation. Magn. Reson. Med. 54, 908–917 (2005).

    Article  Google Scholar 

  43. Cho, K. et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proc. Conference on Empirical Methods in Natural Language Processing (2014).

  44. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal policy optimization algorithms. Preprint at https://arxiv.org/abs/1707.06347 (2017).

  45. Paszke, A. et al. Automatic differentiation in pytorch. In Proc. Conference on Neural Information Processing Systems (2017).

  46. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. International Conference for Learning Representations (2015).

  47. Matson, G. B. An integrated program for amplitude-modulated RF pulse generation and re-mapping with shaped gradients. Magn. Reson. Imaging 12, 1205–1225 (1994).

    Article  Google Scholar 

  48. Martin, J. B. et al. SigPy.RF: comprehensive open-source RF pulse design tools for reproducible research. In Proc. 28th Annual Meeting of ISMRM (2020).

  49. Stockmann, J. P. et al. A 32‐channel combined RF and B0 shim array for 3T brain imaging. Magn. Reson. Med. 75, 441–451 (2016).

    Article  Google Scholar 

  50. Shin, D. et al. DeepRF: (v1.0) (Zenodo, 2021); https://doi.org/10.5281/zenodo.5529394

Download references

Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF-2021R1A2B5B03002783), Samsung Research Funding and Incubation Center of Samsung Electronics (SRFC-IT1801-09), RadiSen, INMC and IOER at Seoul National University.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Seoul National University, Seoul, Korea

    Dongmyung Shin, Younghoon Kim, Chungseok Oh, Hongjun An, Juhyung Park, Jiye Kim & Jongho Lee

Authors
  1. Dongmyung Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Younghoon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chungseok Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongjun An
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Juhyung Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jiye Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jongho Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

D.S. conceived the study, conducted the experiments, and wrote the paper together with J.L., whereas Y.K. implemented the algorithm. C.O. and J.L. assisted with the experimental data interpretation. H.A. helped with the problem formulation. J.P. and J.K. contributed to the development of the concept of the study. All authors reviewed and commented on the manuscript.

Corresponding author

Correspondence to Jongho Lee.

Ethics declarations

Competing interests

Two patents are disclosed (in the order of patent applicant, names of inventors, application number, status of application, specific aspect of manuscript covered in patent application). Seoul National University (D.S. and J.L.), US Patent No. 17/168,274, patent pending; see the 'RF refinement module' section in the Methods. Seoul National University (D.S. and J.L.), Korea Patent No. 10-2020-0106569, patent pending, see the 'RF refinement module' section in the Methods.

Additional information

Peer review information Nature Machine Intelligence thanks Peder Larson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Discussion, Figs. 1–24, and Tables 1 and 2.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shin, D., Kim, Y., Oh, C. et al. Deep reinforcement learning-designed radiofrequency waveform in MRI. Nat Mach Intell 3, 985–994 (2021). https://doi.org/10.1038/s42256-021-00411-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00411-1

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics