Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Primer
  • Published:

Extended reality for biomedicine

Abstract

Extended reality (XR) refers to an umbrella of methods that allows users to be immersed in a 3D or a 4D (spatial + temporal) virtual environment to various extents, including virtual reality (VR), augmented reality (AR) and mixed reality (MR). Whereas VR allows a user to be fully immersed in a virtual environment, AR and MR overlay virtual objects onto the real physical world. The immersion and interaction of XR provide unparalleled opportunities to extend our world beyond conventional lifestyles. Although XR has extensive applications in fields such as entertainment and education, its numerous applications in biomedicine create transformative opportunities in both fundamental research and health care. This Primer outlines XR technology from instrumentation to software computation methods, delineating the biomedical applications that have been advanced by state-of-the-art techniques. We further describe the technical advances overcoming current limitations in XR and its applications, providing an entry point for professionals and trainees to thrive in this emerging field.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Schematic of virtual reality, augmented reality and mixed reality.
Fig. 2: Instrumentation and optical structure of virtual reality and augmented reality head-mounted displays.
Fig. 3: Tracking and haptic feedback in extended reality applications.
Fig. 4: Biomedical applications of extended reality.

Similar content being viewed by others

References

  1. Torigoe, M. et al. Zebrafish capable of generating future state prediction error show improved active avoidance behavior in virtual reality Nat. Commun. 12 5712 (2020).

    Article  ADS  Google Scholar 

  2. Blanc, T., El Beheiry, M., Caporal, C., Masson, J.-B. & Hajj, B. Genuage: visualize and analyze multidimensional single-molecule point cloud data in virtual reality. Nat. Methods 17, 1100–1102 (2020).

    Article  Google Scholar 

  3. Safaryan, K. & Mehta, M. R. Enhanced hippocampal theta rhythmicity and emergence of eta oscillation in virtual reality. Nat. Neurosci. 24, 1065–1070 (2021).

    Article  Google Scholar 

  4. Canning, C. G. et al. Virtual reality in research and rehabilitation of gait and balance in Parkinson disease. Nat. Rev. Neurol. 16, 409–425 (2020).

    Article  Google Scholar 

  5. Kim, H. R. et al. A unified framework for dopamine signals across timescales. Cell 183, 1600–1616.e25 (2020).

    Article  Google Scholar 

  6. Spark, A. et al. vLUME: 3D virtual reality for single-molecule localization microscopy. Nat. Methods 17, 1097–1099 (2020).

    Article  Google Scholar 

  7. Milgram, P. & Kishino, F. A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. 77, 1321–1329 (1994).

    Google Scholar 

  8. Mitrousia, V. & Giotakos, O. Virtual reality therapy in anxiety disorders. Psychiatriki 27, 276–286 (2016).

    Article  Google Scholar 

  9. Elor, A. et al. On shooting stars: comparing CAVE and HMD immersive virtual reality exergaming for adults with mixed ability. ACM Trans. Comput. Healthc. 1, 22 (2020).

    Article  Google Scholar 

  10. Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V. & Hart, J. C. The CAVE: audio visual experience automatic virtual environment. Commun. ACM 35, 64–73 (1992).

    Article  Google Scholar 

  11. Pettersen, E. F. et al. UCSF ChimeraX: structure visualization for researchers, educators, and developers. Protein Sci. 30, 70–82 (2021).

    Article  Google Scholar 

  12. Szugye, N. A. et al. 3D holographic virtual surgical planning for a single right ventricle Fontan patient needing heartmate III placement. ASAIO J. 67, e211–e215 (2021).

    Article  Google Scholar 

  13. Parkhomenko, E. et al. Pilot assessment of immersive virtual reality renal models as an educational and preoperative planning tool for percutaneous nephrolithotomy. J. Endourol. 33, 283–288 (2019).

    Article  Google Scholar 

  14. Tang, Y. M., Chau, K. Y., Kwok, A. P. K., Zhu, T. & Ma, X. A systematic review of immersive technology applications for medical practice and education-trends, application areas, recipients, teaching contents, evaluation methods, and performance. Educ. Res. Rev. 35, 100429 (2022).

    Article  Google Scholar 

  15. Plotzky, C. et al. Virtual reality simulations in nurse education: a systematic mapping review. Nurse Educ. Today 101, 104868 (2021).

    Article  Google Scholar 

  16. Li, L. et al. Application of virtual reality technology in clinical medicine. Am. J. Transl. Res. 9, 3867 (2017).

    ADS  Google Scholar 

  17. Guitard, T., Bouchard, S., Bélanger, C. & Berthiaume, M. Exposure to a standardized catastrophic scenario in virtual reality or a personalized scenario in imagination for generalized anxiety disorder. J. Clin. Med. 8, 309 (2019).

    Article  Google Scholar 

  18. Genova, C. et al. A simulator for both manual and powered wheelchairs in immersive virtual reality CAVE. Virtual Real. 26, 187–203 (2022).

    Article  Google Scholar 

  19. Salmas, M., Chronopoulos, E. & Chytas, D. The vague differentiation between artificial reality technologies in plastic surgery. Plast. Reconstr. Surg. Glob. Open 8, e2909 (2020).

    Article  Google Scholar 

  20. Speicher, M., Hall, B. D. & Nebeling, M. in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Paper no. 537 (CHI, 2019).

  21. Mitsuno, D., Ueda, K., Hirota, Y. & Ogino, M. Effective application of mixed reality device HoloLens: simple manual alignment of surgical field and holograms. Plast. Reconstr. Surg. 143, 647–651 (2019).

    Article  Google Scholar 

  22. Tepper, O. M. et al. Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast. Reconstr. Surg. 140, 1066–1070 (2017).

    Article  Google Scholar 

  23. Lopes, P., You, S., Ion, A. & Baudisch, P. in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Paper no. 446 (CHI, 2018).

  24. Peters, T. M., Linte, C. A., Yaniv, Z. & Williams, J. (eds) Mixed and Augmented Reality in Medicine (CRC Press, 2018).

  25. Azuma, R. T. A survey of augmented reality. Presence Teleoperators Virtual Environ. 6, 355–385 (1997).

    Article  Google Scholar 

  26. Yin, K. et al. Virtual reality and augmented reality displays: advances and future perspectives. J. Phys. Photonics 3, 022010 (2021).

    Article  ADS  Google Scholar 

  27. Parekh, P., Patel, S., Patel, N. & Shah, M. Systematic review and meta-analysis of augmented reality in medicine, retail, and games. Vis. Comput. Ind. Biomed. Art. 3, 21 (2020).

    Article  Google Scholar 

  28. Billinghurst, M., Clark, A. & Lee, G. A survey of augmented reality. Found. Trends Human Computer Interact. 8, 73–272 (2015).

    Article  Google Scholar 

  29. Lee, C. & Wong, G. K. C. Virtual reality and augmented reality in the management of intracranial tumors: a review. J. Clin. Neurosci. 62, 14–20 (2019).

    Article  Google Scholar 

  30. Birlo, M., Edwards, P. J. E., Clarkson, M. & Stoyanov, D. Utility of optical see-through head mounted displays in augmented reality-assisted surgery: a systematic review. Med. Image Anal. 77, 102361 (2022).

    Article  Google Scholar 

  31. Eckert, M., Volmerg, J. S. & Friedrich, C. M. Augmented reality in medicine: systematic and bibliographic review. JMIR Mhealth Uhealth 7, e10967 (2019).

    Article  Google Scholar 

  32. Sheridan, T. B. Interaction, imagination and immersion: some research needs. in Proceedings of the ACM Symposium on Virtual Reality Software and Technology (ACM, 2000).

  33. Cho, B. H. et al. Attention enhancement system using virtual reality and EEG biofeedback. Proc. IEEE Virtual Real. 2002, 156–163 (2002).

    Google Scholar 

  34. Hu, R., Wu, Y.-Y. & Shieh, C.-J. Effects of virtual reality integrated creative thinking instruction on students’ creative thinking abilities. Eurasia J. Math. Sci. Technol. Educ. 12, 477–486 (2016).

    Google Scholar 

  35. Barrett, A. J., Pack, A. & Quaid, E. D. Understanding learners’ acceptance of high-immersion virtual reality systems: Insights from confirmatory and exploratory PLS-SEM analyses. Comput. Educ. 169, 104214 (2021).

    Article  Google Scholar 

  36. Dangxiao, W. et al. Haptic display for virtual reality: progress and challenges. Virtual Real. Intell. Hardw. 1, 136–162 (2019).

    Article  Google Scholar 

  37. Koulieris, G. A. et al. Near-eye display and tracking technologies for virtual and augmented reality. in. Computer Graph. Forum 38, 493–519 (2019).

    Article  Google Scholar 

  38. Pelargos, P. E. et al. Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery. J. Clin. Neurosci. 35, 1–4 (2017).

    Article  Google Scholar 

  39. LaValle, S. Virtual Reality (Cambridge Univ. Press, 2016).

  40. Xiong, J., Hsiang, E.-L., He, Z., Zhan, T. & Wu, S.-T. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light. Sci. Appl. 10, 216 (2021).

    Article  ADS  Google Scholar 

  41. Jang, H. J. et al. Progress of display performances: AR, VR, QLED, OLED, and TFT. J. Inf. Disp. 20, 1–8 (2019).

    Article  Google Scholar 

  42. Bang, K., Jo, Y., Chae, M. & Lee, B. Lenslet VR: thin, flat and wide-FOV virtual reality display using fresnel lens and lenslet array. IEEE Trans. Vis. Comput. Graph. 27, 2545–2554 (2021).

    Article  Google Scholar 

  43. Narasimhan, B. A. Ultra-compact pancake optics based on ThinEyes super-resolution technology for virtual reality headsets. Proc. Digital Opt. Immersive Disp. 10676, 1G (2018).

    Google Scholar 

  44. Lee, B. & Jo, Y. in Advanced Display Technology (eds Kang, I. B., Han, C. W. & Jeong, J. K.) 307–328 (Springer, 2021).

  45. Havig, P., McIntire, J. & Geiselman, E. Virtual reality in a cave: limitations and the need for HMDs? Head-helmet-mounted Disp. XVI Des. Appl. 8041, 58–63 (2011).

    Google Scholar 

  46. Stowers, J. R. et al. Virtual reality for freely moving animals. Nat. Methods 14, 995–1002 (2017).

    Article  Google Scholar 

  47. Xiao, S. et al. Randomized controlled trial of a dichoptic digital therapeutic for amblyopia. Ophthalmology 129, 77–85 (2022).

    Article  Google Scholar 

  48. Bimber, O. & Raskar, R. Spatial Augmented Reality: Merging Real and Virtual Worlds (CRC press, 2005).

  49. Silva, R., Oliveira, J. C. & Giraldi, G. A. Introduction to augmented reality. Natl. Lab. Sci. Comput. 11, 1–11 (2003).

    Google Scholar 

  50. Colburn, M. in 2020 IEEE International Electron Devices Meeting (IEDM) 33.3.1–33.3.4 (IEEE, 2020).

  51. Luo, H. et al. Augmented reality navigation for liver resection with a stereoscopic laparoscope. Comput. Methods Prog. Biomed. 187, 105099 (2020).

    Article  Google Scholar 

  52. Kim, H. et al. Recent advances in wearable sensors and integrated functional devices for virtual and augmented reality applications. Adv. Funct. Mater. 31, 2005692 (2021).

    Article  Google Scholar 

  53. Ren, H. & Kazanzides, P. Investigation of attitude tracking using an integrated inertial and magnetic navigation system for hand-held surgical instruments. IEEE/ASME Trans. Mechatronics 17, 210–217 (2010).

    Google Scholar 

  54. Ang, W. T., Khosla, P. K. & Riviere, C. N. in 2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422) Vol. 2 1781–1786 (IEEE, 2003).

  55. Stoll, J., Ren, H. & Dupont, P. E. Passive markers for tracking surgical instruments in real-time 3-D ultrasound imaging. IEEE Trans. Med. Imaging 31, 563–575 (2011).

    Article  Google Scholar 

  56. Zhou, Z. et al. Optical surgical instrument tracking system based on the principle of stereo vision. J. Biomed. Opt. 22, 65005 (2017).

    Article  Google Scholar 

  57. Bouget, D., Allan, M., Stoyanov, D. & Jannin, P. Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med. Image Anal. 35, 633–654 (2017).

    Article  Google Scholar 

  58. Kim, Y., Kim, H. & Kim, Y. O. Virtual reality and augmented reality in plastic surgery: a review. Arch. Plast. Surg. 44, 179–187 (2017).

    Article  Google Scholar 

  59. Yamamoto, T., Abolhassani, N., Jung, S., Okamura, A. M. & Judkins, T. N. Augmented reality and haptic interfaces for robot-assisted surgery. Int. J. Med. Robot. Comput. Assist. Surg. 8, 45–56 (2012).

    Article  Google Scholar 

  60. Cho, D. et al. Detection of stress levels from biosignals measured in virtual reality environments using a kernel-based extreme learning machine. Sensors 17, 2435 (2017).

    Article  ADS  Google Scholar 

  61. Marín-Morales, J. et al. Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci. Rep. 8, 13657 (2018).

    Article  ADS  Google Scholar 

  62. Herumurti, D., Yuniarti, A., Rimawan, P. & Yunanto, A. A. in 2019 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) 139–144 (IEEE, 2019).

  63. Avari Silva, J. N., Privitera, M. B., Southworth, M. K. & Silva, J. R. in International Conference on Human-Computer Interaction 341–356 (Springer, 2020).

  64. Angelov, V., Petkov, E., Shipkovenski, G. & Kalushkov, T. Modern virtual reality headsets. in 2020 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA) 1–5 (IEEE, 2020).

  65. Jung, S. S. & Jeong, J. in Augmented Reality and Virtual Reality (eds tom Dieck, M.C. & Rauschnabel, P. A.) 323–332 (Springer, 2020).

  66. Riemann, T., Kronin, S. & Metternich, J. in Proceedings of the 12th Conference on Learning Factories (CLF, 2022) Available at SSRN 4074046 (2022).

  67. Verhey, J. T., Haglin, J. M., Verhey, E. M. & Hartigan, D. E. Virtual, augmented, and mixed reality applications in orthopedic surgery. Int. J. Med. Robot. Comput. Assist. Surg. 16, e2067 (2020).

    Article  Google Scholar 

  68. Rahman, R. et al. Head-mounted display use in surgery: a systematic review. Surg. Innov. 27, 88–100 (2020).

    Article  Google Scholar 

  69. Elbamby, M. S., Perfecto, C., Bennis, M. & Doppler, K. Toward low-latency and ultra-reliable virtual reality. IEEE Netw. 32, 78–84 (2018).

    Article  Google Scholar 

  70. Clay, V., König, P. & Koenig, S. Eye tracking in virtual reality. J. Eye Mov. Res. 12, 3 (2019).

    Google Scholar 

  71. Ae Ryu, G. & Yoo, K.-H. in The 25th International Conference on 3D Web Technology Article no. 28 (2020).

  72. Stefani, C., Lacy-Hulbert, A. & Skillman, T. ConfocalVR: immersive visualization for confocal microscopy. J. Mol. Biol. 430, 4028–4035 (2018).

    Article  Google Scholar 

  73. Buzink, S. N., Goossens, R. H. M., Ridder, H., De & Jakimowicz, J. J. Training of basic laparoscopy skills on SimSurgery SEP. Minim. Invasive Ther. Allied Technol. 19, 35–41 (2010).

    Article  Google Scholar 

  74. Tse, B. et al. in International Conference on Human Haptic Sensing and Touch Enabled Computer Applications (eds Kappers, A. M. L., van Erp, J.B.F., Bergmann Tiest, W.M. & van der Helm, F. C. T.) 101–108 (Springer, 2010).

  75. Venkatesan, M. et al. Virtual and augmented reality for biomedical applications. Cell Rep. Med. 2, 100348 (2021).

    Article  Google Scholar 

  76. Douglas, D. B., Wilke, C. A., Gibson, J. D., Boone, J. M. & Wintermark, M. Augmented reality: advances in diagnostic imaging. Multimodal Technol. Interact. 1, 29 (2017).

    Article  Google Scholar 

  77. Zhang, J. F., Paciorkowski, A. R., Craig, P. A. & Cui, F. BioVR: a platform for virtual reality assisted biological data integration and visualization. BMC Bioinformatics 20, 78 (2019).

    Article  Google Scholar 

  78. Barfield, W. (ed.) Fundamentals of Wearable Computers and Augmented Reality (CRC Press, 2015).

  79. Dumic, E., Battisti, F., Carli, M. & da Silva Cruz, L. A. in 2020 28th European Signal Processing Conference (EUSIPCO) 595–599 (IEEE, 2021).

  80. Lelek, M. et al. Single-molecule localization microscopy. Nat. Rev. Methods Primers 1, 39 (2021).

    Article  Google Scholar 

  81. Zhang, Q., Eagleson, R. & Peters, T. M. Volume visualization: a technical overview with a focus on medical applications. J. Digit. Imaging 24, 640–664 (2011).

    Article  Google Scholar 

  82. Kikinis, R., Pieper, S. D. & Vosburgh, K. G. in Intraoperative Imaging and Image-guided Therapy (ed. Jolesz, F.) 277–289 (Springer, 2014).

  83. Foley, J. D., Van, F. D., Van Dam, A., Feiner, S. K. & Hughes, J. F. Computer Graphics: Principles and Practice. vol. 12110 (Addison-Wesley Professional, 1996).

  84. Levine, J. A., Paulsen, R. R. & Zhang, Y. Mesh processing in medical-image analysis — a tutorial. IEEE Comput. Graph. Appl. 32, 22–28 (2012).

    Article  Google Scholar 

  85. Wenger, R. Isosurfaces: Geometry, Topology, and Algorithms (CRC Press, 2013).

  86. Callahan, S. P., Callahan, J. H., Scheidegger, C. E. & Silva, C. T. Direct volume rendering: a 3D plotting technique for scientific data. Comput. Sci. Eng. 10, 88–92 (2008).

    Article  Google Scholar 

  87. Engel, K. et al. in ACM Siggraph 2004 Course Notes 29-es (2004).

  88. Minaee, S. et al. Image segmentation using deep learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44, 3523–3542 (2021).

    Google Scholar 

  89. Guenter, B., Finch, M., Drucker, S., Tan, D. & Snyder, J. Foveated 3D graphics. ACM Trans. Graph. 31, 164 (2012).

    Article  Google Scholar 

  90. Nock, R. & Nielsen, F. Statistical region merging. IEEE Trans. Pattern Anal. Mach. Intell. 26, 1452–1458 (2004).

    Article  Google Scholar 

  91. Najman, L. & Schmitt, M. Watershed of a continuous function. Signal. Process. 38, 99–112 (1994).

    Article  Google Scholar 

  92. Dhanachandra, N., Manglem, K. & Chanu, Y. J. Image segmentation using K-means clustering algorithm and subtractive clustering algorithm. Procedia Comput. Sci. 54, 764–771 (2015).

    Article  Google Scholar 

  93. Boykov, Y., Veksler, O. & Zabih, R. Fast approximate energy minimization via graph cuts. IEEE Trans. Patt Anal. Mach. Intelligence 23, 1222–1239 (2001).

    Article  Google Scholar 

  94. Starck, J.-L., Elad, M. & Donoho, D. L. Image decomposition via the combination of sparse representations and a variational approach. IEEE Trans. Image Process. 14, 1570–1582 (2005).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  95. Kass, M., Witkin, A. & Terzopoulos, D. Snakes: active contour models. Int. J. Comput. Vis. 1, 321–331 (1988).

    Article  MATH  Google Scholar 

  96. Plath, N., Toussaint, M. & Nakajima, S. in Proceedings of the 26th Annual International Conference on Machine Learning 817–824 (ICML, 2009).

  97. Shi, F. et al. Deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy. Nat. Commun. 13, 6566 (2022).

    Article  ADS  Google Scholar 

  98. Antonelli, M. et al. The medical segmentation decathlon. Nat. Commun. 13, 4128 (2022).

    Article  ADS  Google Scholar 

  99. Zhou, S. K., Le, H. N., Luu, K., Nguyen, H. V. & Ayache, N. Deep reinforcement learning in medical imaging: a literature review. Med. Image Anal. 73, 102193 (2021).

    Article  Google Scholar 

  100. Alom, M. Z. et al. A state-of-the-art survey on deep learning theory and architectures. Electronics 8, 292 (2019).

    Article  Google Scholar 

  101. Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).

    Article  Google Scholar 

  102. Schmidhuber, J. Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015).

    Article  Google Scholar 

  103. Safadel, P. & White, D. Facilitating molecular biology teaching by using Augmented Reality (AR) and Protein Data Bank (PDB). TechTrends 63, 188–193 (2019).

    Article  Google Scholar 

  104. Kinjo, A. R. et al. New tools and functions in data-out activities at Protein Data Bank Japan (PDBj). Protein Sci. 27, 95–102 (2018).

    Article  Google Scholar 

  105. Bai, H., Li, S. & Shepherd, R. F. Elastomeric haptic devices for virtual and augmented reality. Adv. Funct. Mater. 31, 2009364 (2021).

    Article  Google Scholar 

  106. Anthes, C., Garcia-Hernández, R. J., Wiedemann, M. & Kranzlmüller, D. State of the art of virtual reality technology. in 2016 IEEE Aerospace Conference 1–19 (IEEE, 2016).

  107. Dho, Y.-S. et al. Development of an inside-out augmented reality technique for neurosurgical navigation. Neurosurg. Focus 51, E21 (2021).

    Article  Google Scholar 

  108. Gsaxner, C., Li, J., Pepe, A., Schmalstieg, D. & Egger, J. in Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. Article no. 4 (ACM, 2021).

  109. Bichlmeier, C., Wimmer, F., Heining, S. M. & Navab, N. in 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality 129–138 (IEEE, 2007).

  110. Yang, L. I., Huang, J., Feng, T., Hong-An, W. & Guo-Zhong, D. A. I. Gesture interaction in virtual reality. Virtual Real. Intell. Hardw. 1, 84–112 (2019).

    Article  Google Scholar 

  111. Cho, Y., Lee, A., Park, J., Ko, B. & Kim, N. Enhancement of gesture recognition for contactless interface using a personalized classifier in the operating room. Comput. Methods Prog. Biomed. 161, 39–44 (2018).

    Article  Google Scholar 

  112. Lee, A., Cho, Y., Jin, S. & Kim, N. Enhancement of surgical hand gesture recognition using a capsule network for a contactless interface in the operating room. Comput. Methods Prog. Biomed. 190, 105385 (2020).

    Article  Google Scholar 

  113. Cameirão, M. S., Faria, A. L., Paulino, T., Alves, J. & Badia, I. S. The impact of positive, negative and neutral stimuli in a virtual reality cognitive-motor rehabilitation task: a pilot study with stroke patients. J. Neuroeng. Rehabil. 13, 70 (2016).

    Article  Google Scholar 

  114. Gibbs, J. K., Gillies, M. & Pan, X. A comparison of the effects of haptic and visual feedback on presence in virtual reality. Int. J. Hum. Comput. Stud. 157, 102717 (2022).

    Article  Google Scholar 

  115. Zhang, Y., Luo, D., Li, J. & Li, J. Study on collision detection and force feedback algorithm in virtual surgery. J. Healthc. Eng. 2021, 6611196 (2021).

    Google Scholar 

  116. Burdea, G. C. Force and Touch Feedback for Virtual Reality (John Wiley & Sons, Inc., 1996).

  117. Olsson, P. et al. Haptics-assisted virtual planning of bone, soft tissue, and vessels in fibula osteocutaneous free flaps. Plast. Reconstr. Surg. Glob. Open 3, e479 (2015).

    Article  Google Scholar 

  118. Ding, Y. et al. Integrating light-sheet imaging with virtual reality to recapitulate developmental cardiac mechanics. JCI Insight 2, e97180 (2017).

    Article  Google Scholar 

  119. Abiri, A. et al. Simulating developmental cardiac morphology in virtual reality using a deformable image registration approach. Ann. Biomed. Eng. 46, 2177–2188 (2018).

    Article  Google Scholar 

  120. Casiano, R. R. Intraoperative image-guidance technology. Arch. Otolaryngol. Head. Neck Surg. 125, 1275–1278 (1999).

    Article  Google Scholar 

  121. Langhorne, P., Bernhardt, J. & Kwakkel, G. Stroke rehabilitation. Lancet 377, 1693–1702 (2011).

    Article  Google Scholar 

  122. Quero, G. et al. Virtual and augmented reality in oncologic liver surgery. Surg. Oncol. Clin. 28, 31–44 (2019).

    Article  Google Scholar 

  123. Beswick, D. M. & Ramakrishnan, V. R. The utility of image guidance in endoscopic sinus surgery: a narrative review. JAMA Otolaryngol. Head. Neck Surg. 146, 286–290 (2020).

    Article  Google Scholar 

  124. Lohre, R. et al. Effectiveness of immersive virtual reality on orthopedic surgical skills and knowledge acquisition among senior surgical residents: a randomized clinical trial. JAMA Netw. Open 3, e2031217 (2020).

    Article  Google Scholar 

  125. Felix, B. et al. Augmented reality spine surgery navigation: increasing pedicle screw insertion accuracy for both open and minimally invasive spine surgeries. Spine 47, 865–872 (2022).

    Article  Google Scholar 

  126. Iacono, V. et al. The use of augmented reality for limb and component alignment in total knee arthroplasty: systematic review of the literature and clinical pilot study. J. Exp. Orthop. 8, 52 (2021).

    Article  Google Scholar 

  127. Jung, C. et al. Virtual and augmented reality in cardiovascular care: state-of-the-art and future perspectives. JACC Cardiovasc. Imaging 15, 519–532 (2021).

    Article  Google Scholar 

  128. Kothgassner, O. D. et al. Virtual reality exposure therapy for posttraumatic stress disorder (PTSD): a meta-analysis. Eur. J. Psychotraumatol. 10, 1654782 (2019).

    Article  Google Scholar 

  129. Gorini, A. & Riva, G. Virtual reality in anxiety disorders: the past and the future. Expert. Rev. Neurother. 8, 215–233 (2008).

    Article  Google Scholar 

  130. Pourmand, A., Davis, S., Marchak, A., Whiteside, T. & Sikka, N. Virtual reality as a clinical tool for pain management. Curr. Pain Headache Rep. 22, 53 (2018).

    Article  Google Scholar 

  131. Payne, O. et al. Virtual reality and its use in post-operative pain following laparoscopy: a feasibility study. Sci. Rep. 12, 13137 (2022).

    Article  ADS  Google Scholar 

  132. Grassini, S. Virtual reality assisted non-pharmacological treatments in chronic pain management: a systematic review and quantitative meta-analysis. Int. J. Environ. Res. Public Health 19, 4071 (2022).

    Article  Google Scholar 

  133. Cassidy, K. C., Šefčík, J., Raghav, Y., Chang, A. & Durrant, J. D. ProteinVR: web-based molecular visualization in virtual reality. PLoS Comput. Biol. 16, e1007747 (2020).

    Article  ADS  Google Scholar 

  134. Kiveric, E. & Gregory, S. H. Three-dimensional assessment of the mitral valve: looking toward the future. J. Cardiothorac. Vasc. Anesth. 33, 742–743 (2019).

    Article  Google Scholar 

  135. Ballocca, F. et al. Validation of quantitative 3-dimensional transesophageal echocardiography mitral valve analysis using stereoscopic display. J. Cardiothorac. Vasc. Anesth. 33, 732–741 (2019).

    Article  Google Scholar 

  136. Geng, H. et al. Visual learning in a virtual reality environment upregulates immediate early gene expression in the mushroom bodies of honey bees. Commun. Biol. 5, 130 (2022).

    Article  Google Scholar 

  137. Huang, K.-H. et al. A virtual reality system to analyze neural activity and behavior in adult zebrafish. Nat. Methods 17, 343–351 (2020).

    Article  Google Scholar 

  138. Robinson, N. T. M. et al. Targeted activation of hippocampal place cells drives memory-guided spatial behavior. Cell 183, 2041–2042 (2020).

    Article  Google Scholar 

  139. Black, P. M. Hormones, radiosurgery and virtual reality: new aspects of meningioma management. Can. J. Neurol. Sci. 24, 302–306 (1997).

    Article  Google Scholar 

  140. Aschke, M., Wirtz, C. R., Raczkowsky, J., Worn, H. & Kunze, S. in First International IEEE EMBS Conference on Neural Engineering, 2003. Conference Proceedings 652–655 (IEEE, 2003).

  141. Torkington, J., Smith, S. G., Rees, B. I. & Darzi, A. The role of simulation in surgical training. Ann. R. Coll. Surg. Engl. 82, 88 (2000).

    Google Scholar 

  142. Southworth, M. K. et al. Performance evaluation of mixed reality display for guidance during transcatheter cardiac mapping and ablation. IEEE J. Transl. Eng. Heal. Med. 8, 1900810 (2020).

    Google Scholar 

  143. Silva, J. N. A., Southworth, M., Raptis, C. & Silva, J. Emerging applications of virtual reality in cardiovascular medicine. JACC Basic. Transl. Sci. 3, 420–430 (2018).

    Article  Google Scholar 

  144. Avari Silva, J. N. et al. First-in-human use of a mixed reality display during cardiac ablation procedures. JACC Clin. Electrophysiol. 6, 1023–1025 (2020).

    Article  Google Scholar 

  145. Bruckheimer, E. et al. Computer-generated real-time digital holography: first time use in clinical medical imaging. Eur. Heart J. Cardiovasc. Imaging 17, 845–849 (2016).

    Article  Google Scholar 

  146. Difede, J. et al. Virtual reality exposure therapy for the treatment of posttraumatic stress disorder following September 11, 2001. J. Clin. Psychiatry 68, 1639 (2007).

    Article  Google Scholar 

  147. Parsons, T. D. & Rizzo, A. A. Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis. J. Behav. Ther. Exp. Psychiatry 39, 250–261 (2008).

    Article  Google Scholar 

  148. Miloff, A. et al. Single-session gamified virtual reality exposure therapy for spider phobia vs. traditional exposure therapy: study protocol for a randomized controlled non-inferiority trial. Trials 17, 60 (2016).

    Article  Google Scholar 

  149. Islam, M. K. & Brunner, I. Cost-analysis of virtual reality training based on the virtual reality for upper extremity in subacute stroke (VIRTUES) trial. Int. J. Technol. Assess. Health Care 35, 373–378 (2019).

    Article  Google Scholar 

  150. Aminov, A., Rogers, J. M., Middleton, S., Caeyenberghs, K. & Wilson, P. H. What do randomized controlled trials say about virtual rehabilitation in stroke? A systematic literature review and meta-analysis of upper-limb and cognitive outcomes. J. Neuroeng. Rehabil. 15, 29 (2018).

    Article  Google Scholar 

  151. Vourvopoulos, A. et al. Efficacy and brain imaging correlates of an immersive motor imagery BCI-driven VR system for upper limb motor rehabilitation: a clinical case report. Front. Hum. Neurosci. 13, 244 (2019).

    Article  Google Scholar 

  152. Vourvopoulos, A. et al. Effects of a brain-computer interface with virtual reality (VR) neurofeedback: a pilot study in chronic stroke patients Front. Hum. Neurosci. 13 210 (2019).

    Article  Google Scholar 

  153. Perey, C., Engelke, T. & Reed, C. in Recent Trends of Mobile Collaborative Augmented Reality Systems (eds Alem, L., & Huang, W.) 21–38 (Springer, 2011).

  154. Borsci, S., Lawson, G. & Broome, S. Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance. Comput. Ind. 67, 17–26 (2015).

    Article  Google Scholar 

  155. Livingston, M. A. Evaluating human factors in augmented reality systems. IEEE Comput. Graph. Appl. 25, 6–9 (2005).

    Article  Google Scholar 

  156. Anik, A. A. et al. Accuracy and reproducibility of linear and angular measurements in virtual reality: a validation study. J. Digit. Imaging 33, 111–120 (2020).

    Article  Google Scholar 

  157. Hepperle, D., Dienlin, T. & Wölfel, M. in 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 100–105 (IEEE, 2021).

  158. Yuan, Y. Paving the road for virtual and augmented reality [standards]. IEEE Consum. Electron. Mag. 7, 117–128 (2017).

    Article  Google Scholar 

  159. Hardesty, J. et al. 3D Data repository features, best practices, and implications for preservation models: findings from a National forum. Coll. Res. Libr. 81, 789–801 (2020).

    Article  Google Scholar 

  160. Koller, D., Frischer, B. & Humphreys, G. Research challenges for digital archives of 3D cultural heritage models. J. Comput. Cult. Herit. 2, 7 (2010).

    Google Scholar 

  161. Hess, M. et al. Developing 3D imaging programmes – workflow and quality control. J. Comput. Cult. Herit. 9, 1 (2015).

    Article  Google Scholar 

  162. Doerr, M. et al. A repository for 3D model production and interpretation in culture and beyond. in The 11th International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST (eds Artusi, A. et al.) 97–104 (2010).

  163. Limp, W. F., Payne, A., Winters, S., Barnes, A. & Cothren, J. Approaching 3D digital heritage data from a multi-technology, lifecycle perspective. in Proceedings of the 38th Annual International Conference on Computer Applications and Quantitative Methods in Archaeology (eds Contreras F. & Melero, J.) 1–8 (CAA, 2010).

  164. Felicetti, A. & Lorenzini, M. Metadata and tools for integration and preservation of cultural heritage 3D information. Geoinformatics FCE CTU 6, 118–124 (2011).

    Article  Google Scholar 

  165. Boyer, D. M., Gunnell, G. F., Kaufman, S. & McGeary, T. M. Morphosource: archiving and sharing 3-D digital specimen data. Paleontol. Soc. Pap. 22, 157–181 (2016).

    Article  Google Scholar 

  166. Richards-Rissetto, H. & von Schwerin, J. A catch 22 of 3D data sustainability: lessons in 3D archaeological data management & accessibility. Digit. Appl. Archaeol. Cult. Herit. 6, 38–48 (2017).

    Google Scholar 

  167. Southgate, E., Smith, S. P. & Scevak, J. in 2017 IEEE Virtual Reality. 12–18 (IEEE, 2017).

  168. O’Brolcháin, F. et al. The convergence of virtual reality and social networks: threats to privacy and autonomy. Sci. Eng. Ethics 22, 1–29 (2016).

    Article  Google Scholar 

  169. Happa, J., Glencross, M. & Steed, A. Cyber security threats and challenges in collaborative mixed-reality. Front. ICT 6, 1–5 (2019).

    Article  Google Scholar 

  170. De Guzman, J. A., Thilakarathna, K. & Seneviratne, A. Security and privacy approaches in mixed reality: a literature survey. ACM Comput. Surv. 52, 110 (2019).

    Google Scholar 

  171. Rokhsaritalemi, S., Sadeghi-Niaraki, A. & Choi, S.-M. A review on mixed reality: current trends, challenges and prospects. Appl. Sci. 10, 636 (2020).

    Article  Google Scholar 

  172. Jana, S. et al. in 22nd USENIX Security Symposium. 415–430 (2013).

  173. Kenwright, B. Virtual reality: ethical challenges and dangers [opinion]. IEEE Technol. Soc. Mag. 37, 20–25 (2018).

    Article  Google Scholar 

  174. Choplin, R. H., Boehme, J. M. II & Maynard, C. D. Picture archiving and communication systems: an overview. Radiographics 12, 127–129 (1992).

    Article  Google Scholar 

  175. Chan, A., Parent, E., Narvacan, K., San, C. & Lou, E. Intraoperative image guidance compared with free-hand methods in adolescent idiopathic scoliosis posterior spinal surgery: a systematic review on screw-related complications and breach rates. Spine J. 17, 1215–1229 (2017).

    Article  Google Scholar 

  176. Su, P. et al. Use of computed tomographic reconstruction to establish the ideal entry point for pedicle screws in idiopathic scoliosis. Eur. Spine J. 21, 23–30 (2012).

    Article  Google Scholar 

  177. Elmi-Terander, A. et al. Augmented reality navigation with intraoperative 3D imaging vs fluoroscopy-assisted free-hand surgery for spine fixation surgery: a matched-control study comparing accuracy. Sci. Rep. 10, 707 (2020).

    Article  ADS  Google Scholar 

  178. Margalit, A. et al. Evaluation of a slipped capital femoral epiphysis virtual reality surgical simulation for the orthopaedic trainee. JAAOS Glob. Res. Rev. 6, e22.00028 (2022).

    Article  Google Scholar 

  179. Piromchai, P., Avery, A., Laopaiboon, M., Kennedy, G. & O’Leary, S. Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst. Rev 9, CD010198 (2015).

    Google Scholar 

  180. Brewer, D. N. et al. in Medicine Meets Virtual Reality 19 (eds Westwood, J. D. et al.) 85–91 (IOS Press, 2012).

  181. Teranishi, S. & Yamagishi, Y. Educational effects of a virtual reality simulation system for constructing self-built PCs. J. Educ. Multimed. Hypermedia 27, 411–423 (2018).

    Google Scholar 

  182. Pulijala, Y., Ma, M., Pears, M., Peebles, D. & Ayoub, A. Effectiveness of immersive virtual reality in surgical training — a randomized control trial. J. Oral. Maxillofac. Surg. 76, 1065–1072 (2018).

    Article  Google Scholar 

  183. Stepan, K. et al. Immersive virtual reality as a teaching tool for neuroanatomy. Int. Forum Allergy Rhinol. 7, 1006–1013 (2017).

    Article  Google Scholar 

  184. Makransky, G., Terkildsen, T. S. & Mayer, R. E. Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn. Instr. 60, 225–236 (2019).

    Article  Google Scholar 

  185. Lee, G.-Y. et al. Metasurface eyepiece for augmented reality. Nat. Commun. 9, 4562 (2018).

    Article  ADS  Google Scholar 

  186. Lee, Y.-H. et al. Recent progress in Pancharatnam–Berry phase optical elements and the applications for virtual/augmented realities. Opt. Data Process. Storage 3, 79–88 (2017).

    Article  Google Scholar 

  187. Zhan, T. et al. Practical chromatic aberration correction in virtual reality displays enabled by cost-effective ultra-broadband liquid crystal polymer lenses. Adv. Opt. Mater. 8, 1901360 (2020).

    Article  Google Scholar 

  188. Yin, K. et al. Advanced liquid crystal devices for augmented reality and virtual reality displays: principles and applications. Light. Sci. Appl. 11, 161 (2022).

    Article  ADS  Google Scholar 

  189. Moon, S. et al. Augmented reality near-eye display using Pancharatnam-Berry phase lenses. Sci. Rep. 9, 6616 (2019).

    Article  ADS  Google Scholar 

  190. Patney, A. et al. Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graph. 35, 179 (2016).

    Article  MathSciNet  Google Scholar 

  191. Chang, V. An overview, examples, and impacts offered by emerging services and analytics in cloud computing virtual reality. Neural Comput. Appl. 29, 1243–1256 (2018).

    Article  Google Scholar 

  192. Orlosky, J. et al. Emulation of physician tasks in eye-tracked virtual reality for remote diagnosis of neurodegenerative disease. IEEE Trans. Vis. Comput. Graph. 23, 1302–1311 (2017).

    Article  Google Scholar 

  193. Hubbard, P. M. Collision detection for interactive graphics applications. IEEE Trans. Vis. Comput. Graph. 1, 218–230 (1995).

    Article  Google Scholar 

  194. Chen, G.-D. & Wang, F.-F. Medical data point clouds reconstruction algorithm based on tensor product B-spline approximation in virtual surgery. J. Med. Biol. Eng. 37, 162–170 (2017).

    Article  Google Scholar 

  195. Anvari, M. et al. The impact of latency on surgical precision and task completion during robotic-assisted remote telepresence surgery. Comput. Aided Surg. 10, 93–99 (2005).

    Article  Google Scholar 

  196. Xu, S. et al. Determination of the latency effects on surgical performance and the acceptable latency levels in telesurgery using the dV-Trainer®simulator. Surg. Endosc. 28, 2569–2576 (2014).

    Article  Google Scholar 

  197. Liu, Y., Peng, M., Shou, G., Chen, Y. & Chen, S. Toward edge intelligence: multiaccess edge computing for 5G and Internet of Things. IEEE Internet Things J. 7, 6722–6747 (2020).

    Article  Google Scholar 

  198. Zhang, K., Mao, Y., Leng, S., He, Y. & Zhang, Y. Mobile-edge computing for vehicular networks: a promising network paradigm with predictive off-loading. IEEE Veh. Technol. Mag. 12, 36–44 (2017).

    Article  Google Scholar 

  199. Zhang, L. & Chakareski, J. UAV-assisted edge computing and streaming for wireless virtual reality: analysis, algorithm design, and performance guarantees. IEEE Trans. Veh. Technol. 71, 3267–3275 (2022).

    Article  Google Scholar 

  200. Chen, Z., Zhu, H., Song, L., He, D. & Xia, B. Wireless multiplayer interactive virtual reality game systems with edge computing: modeling and optimization. IEEE Trans. Wirel. Commun. 21, 9684–9699 (2022).

    Article  Google Scholar 

  201. Gao, G. & Li, W. Architecture of visual design creation system based on 5G virtual reality. Int. J. Commun. Syst. 35, e4750 (2022).

    Article  Google Scholar 

  202. Sugimoto, M. in Multidisciplinary Computational Anatomy (ed. Hashizume, M.) 381–387 (Springer, 2022).

  203. Brengman, M., Willems, K. & De Gauquier, L. Customer engagement in multi-sensory virtual reality advertising: the effect of sound and scent congruence. Front. Psychol. 13, 747456 (2022).

    Article  Google Scholar 

  204. Jung, S., Karki, N., Slutter, M. & Lindeman, R. W. On the use of multi-sensory cues in symmetric and asymmetric shared collaborative virtual spaces. Proc. ACM Human Computer Interact. 5, 72 (2021).

    Article  Google Scholar 

  205. Petit, O., Velasco, C. & Spence, C. Digital sensory marketing: Integrating new technologies into multisensory online experience. J. Interact. Mark. 45, 42–61 (2019).

    Article  Google Scholar 

  206. Jung, Y. H. et al. A wireless haptic interface for programmable patterns of touch across large areas of the skin. Nat. Electron. 5, 374–385 (2022).

    Article  Google Scholar 

  207. Yu, X. et al. Skin-integrated wireless haptic interfaces for virtual and augmented reality. Nature 575, 473–479 (2019).

    Article  ADS  Google Scholar 

  208. Nakamoto, T. & Yoshikawa, K. Movie with scents generated by olfactory display using solenoid valves. IEICE Trans. Fundam. Electron. Commun. Comput. Sci. 89, 3327–3332 (2006).

    Article  ADS  Google Scholar 

  209. Jung, S., Wood, A. L., Hoermann, S., Abhayawardhana, P. L. & Lindeman, R. W. in 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 463–472 (IEEE, 2020).

  210. Karunanayaka, K. et al. New thermal taste actuation technology for future multisensory virtual reality and internet. IEEE Trans. Vis. Comput. Graph. 24, 1496–1505 (2018).

    Article  Google Scholar 

  211. Laukkanen, T., Xi, N., Hallikainen, H., Ruusunen, N. & Hamari, J. Virtual technologies in supporting sustainable consumption: from a single-sensory stimulus to a multi-sensory experience. Int. J. Inf. Manage. 63, 102455 (2022).

    Article  Google Scholar 

  212. Wang, Y. et al. TeraVR empowers precise reconstruction of complete 3-D neuronal morphology in the whole brain. Nat. Commun. 10, 3474 (2019).

    Article  ADS  Google Scholar 

  213. Shi, H., Ames, J. & Randles, A. Harvis: an interactive virtual reality tool for hemodynamic modification and simulation. J. Comput. Sci. 43, 101091 (2020).

    Article  Google Scholar 

  214. Günther, U. et al. Scenery: flexible virtual reality visualization on the Java VM. Preprint at https://doi.org/10.48550/arXiv.1906.06726 (2019).

  215. Pirch, S. et al. The VRNetzer platform enables interactive network analysis in virtual reality. Nat. Commun. 12, 2432 (2021).

    Article  ADS  Google Scholar 

  216. Stein, D. F. et al. singlecellVR: interactive visualization of single-cell data in virtual reality. Front. Genet. 12, 764170 (2021).

    Article  Google Scholar 

  217. Nowinski, W. L., Yang, G. L. & Yeo, T. T. Computer-aided stereotactic functional neurosurgery enhanced by the use of the multiple brain atlas database. IEEE Trans. Med. Imaging 19, 62–69 (2000).

    Article  Google Scholar 

  218. Delorme, S., Laroche, D., DiRaddo, R. & Del Maestro, R. F. NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training. Oper. Neurosurg. 71, ons32–ons42 (2012).

    Article  Google Scholar 

Download references

Acknowledgements

The authors appreciate all lab members for constructive discussions. This work was supported by NIH R00HL148493 (Y.D.), R56HL158569 (R.R.S.P.), R01CA156775 (B.F.), R01CA204254 (B.F.), R01HL140325 (B.F.), R21CA231911 (B.F.), VA Merit BX004558 (R.R.S.P.), UCLA Cardiovascular Discovery Fund/Lauren B. Leichtman and Arthur E. Levine Investigator Award (R.R.S.P.), NIH NCATS UCLA CTSI UL1TR001881 (R.R.S.P.), the Cancer Prevention and Research Institute of Texas (CPRIT) grant RP190588 (B.F.) and UT Dallas STARs program (Y.D.).

Author information

Authors and Affiliations

Authors

Contributions

Introduction (J.Y., S.S.H. and Y.D.); Experimentation (J.Y. and S.S.H.); Results (J.Y., S.S.H., J.W., F.S. and Y.D.); Applications (J.Y., S.S.H., R.R.S.P. and Y.D.); Reproducibility and data deposition (S.S.H. and Y.D.); Limitations and optimizations (J.Y., C.R.K. and Y.D.); Outlook (J.Y., B.F. and Y.D.); Overview of the Primer (J.Y., S.S.H., J.W., C.R.K., R.R.S.P., F.S., B.F. and Y.D).

Corresponding author

Correspondence to Yichen Ding.

Ethics declarations

Competing interests

J.W. and F.S. are employees of Shanghai United Imaging Intelligence Co., Ltd. The company has no role in designing and performing the surveillance and analysing and interpreting the results. The other authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Methods Primers thanks Roger Dias, Mahdi Ebnali and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

3D Slicer: https://www.slicer.org/

Amelia: https://ameliavirtualcare.com/

AnatomyX: https://www.medivis.com/anatomyx

Balloon Blast: https://xrhealth.com.au/products/

Best Practices Guidelines: https://developer.oculus.com/resources/learn/#guides-featured-throughout-this-section

ClarifEye: https://www.philips.com/a-w/about/innovation/research/our-programs/clarifeye.html

DICOM: https://www.dicomstandard.org/

EchoPixel True3D: https://echopixeltech.com/true3d

eF-site: https://pdbj.org/eF-site/

Google Daydream View: https://developers.google.com/vr/discover/daydream-view

Happinss: http://www.happinss.com/en/

HP Reverb G2: https://www.hp.com/us-en/vr/reverb-g2-vr-headset.html

HTC Vive Pro 2: https://www.vive.com/us/product/vive-pro2-full-kit/overview/

HumanSim: http://www.virtualheroes.biz/breathe/

Immersive Rehab: https://immersiverehab.com/

IntraOpVSP: https://www.xironetic.com/technology

Khronos Group: https://www.khronos.org/openxr/

Knee+: https://www.pixee-medical.com/en/knee/

Luminopia One: https://luminopia.com/

Magic Leap 1: https://www.magicleap.care/hc/en-us/categories/9527890621325-Magic-Leap-1

Magic Leap 2: https://www.magicleap.com/magic-leap-2

Meta Quest 2: https://www.meta.com/quest/products/quest-2/tech-specs/#tech-specs

Microsoft HoloLens 2: https://www.microsoft.com/en-us/hololens/hardware

NextAR: https://nextar.medacta.com/

Nreal Light: https://www.nreal.ai/light

Oculus Rift S: https://www.oculus.com/rift-s/features/

OpenSight: https://www.novarad.net/augmentedreality

PrecisionOS: https://www.precisionostech.com/

Project LG-73-17-0141-17: https://www.imls.gov/grants/awarded/lg-73-17-0141-17

RASimAs: https://rasimas.plri.de/

REAL y-Series: https://www.realsystem.com/yseries/

RelieVRx: https://relievrx.com/

Samsung Odyssey: https://www.samsung.com/us/support/computing/hmd/hmd-odyssey/hmd-odyssey-mixed-reality/

Snap Spectacles: https://www.spectacles.com/

Social Engagement: https://oxfordvr.co/how-we-can-help/

Sony PlayStation VR: https://www.playstation.com/en-us/ps-vr/

SyncAR & StealthStation S8: https://surgicaltheater.com/syncar/

Valve Index: https://www.valvesoftware.com/en/index

Virtual Reality Industry Forum: https://www.vr-if.org/

VisAR: https://www.novarad.net/visar

XRHealth: https://www.xr.health/

xvision Spine: http://augmedics.com

Supplementary information

Glossary

Angular resolution

The ratio between the number of horizontal pixels and horizontal field of view.

Binocular disparity

The slight difference between left and right retinal images of the same object due to the location difference of the left and right eyes.

Field of view

(FOV). The visual field as one eye is stationary. In general, the monocular FOV of a human eye is about 160° × 130° (horizontal × vertical), and the combined binocular FOV is about 200° × 130°, with an overlapped region of 120° horizontally.

Foveated rendering

A rendering method designed to improve graphics performance by maintaining high visual detail near the fovea, while decreasing quality towards the eye’s periphery.

Frame rate

The number of consecutive images that are displayed and delivered to the user every second.

Gestures

The posture or movement of the user’s upper limbs, including fingers, hands and arms, containing significant interactive intentions as the input for extended reality.

Haptic gloves

A type of wearable device that functions to provide realistic sensation and manipulation of virtual objects through hand motion tracking, force feedback and tactile feedback.

Inertial measurement unit

An electronic device that contains a gyroscope, an accelerometer and a magnetometer used to measure the specific force, angular rate and orientation of the body.

Optical combiner

The component of the augmented reality display that delivers images produced by the display engine to the user’s eye while also transmitting environmental light.

Vergence–accommodation conflict

A visual phenomenon that occurs when the brain receives mismatching cues between vergence and accommodation of the eye.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, J., Hassan, S.S., Wu, J. et al. Extended reality for biomedicine. Nat Rev Methods Primers 3, 14 (2023). https://doi.org/10.1038/s43586-023-00198-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s43586-023-00198-y

This article is cited by

Search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research