Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Studying osteoarthritis with artificial intelligence applied to magnetic resonance imaging

Nature Reviews Rheumatology volume 18, pages 112–121 (2022)Cite this article

Subjects

Abstract

The 3D nature and soft-tissue contrast of MRI makes it an invaluable tool for osteoarthritis research, by facilitating the elucidation of disease pathogenesis and progression. The recent increasing employment of MRI has certainly been stimulated by major advances that are due to considerable investment in research, particularly related to artificial intelligence (AI). These AI-related advances are revolutionizing the use of MRI in clinical research by augmenting activities ranging from image acquisition to post-processing. Automation is key to reducing the long acquisition times of MRI, conducting large-scale longitudinal studies and quantitatively defining morphometric and other important clinical features of both soft and hard tissues in various anatomical joints. Deep learning methods have been used recently for multiple applications in the musculoskeletal field to improve understanding of osteoarthritis. Compared with labour-intensive human efforts, AI-based methods have advantages and potential in all stages of imaging, as well as post-processing steps, including aiding diagnosis and prognosis. However, AI-based methods also have limitations, including the arguably limited interpretability of AI models. Given that the AI community is highly invested in uncovering uncertainties associated with model predictions and improving their interpretability, we envision future clinical translation and progressive increase in the use of AI algorithms to support clinicians in optimizing patient care.

Key points

  • Applications of deep learning to accelerate MRI acquisition and reconstruction show exciting results; nevertheless, fundamental questions remain regarding the most appropriate metrics for evaluating the quality of reconstructed images.

  • Image segmentation errors from artificial intelligence (AI) models lie within the intra-reader variability range. Nonetheless, deployment in clinical practice still requires some form of quality assurance, which might include visual inspection of segmentation outputs.

  • The role of AI in osteoarthritis lesion detection is not to provide a final diagnosis but rather to serve as an additional input for decision-making.

  • AI has been tasked with searching for novel image features indicative of short-term and long-term progression of osteoarthritis to predict disease course on a patient-specific basis.

  • Future research will likely be devoted to the interpretability and estimation of the uncertainty of deep learning models, with the aim of improving clinician trust in AI for supporting patient care.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The role of artificial intelligence in stages of imaging.
Fig. 2: Anatomical structures on MRI scans influencing artificial intelligence model prediction of total knee replacement within 5 years.

Similar content being viewed by others

References

  1. Link, T. M., Neumann, J. & Li, X. Prestructural cartilage assessment using MRI. J. Magn. Reson. Imaging 45, 949–965 (2017).

    Article  PubMed  Google Scholar 

  2. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  CAS  PubMed  Google Scholar 

  3. Erickson, B. J., Korfiatis, P., Akkus, Z. & Kline, T. L. Machine learning for medical imaging. Radiographics 37, 505–515 (2017).

    Article  PubMed  Google Scholar 

  4. Liang, S. et al. Deep-learning-based detection and segmentation of organs at risk in nasopharyngeal carcinoma computed tomographic images for radiotherapy planning. Eur. Radiol. 29, 1961–1967 (2019).

    Article  PubMed  Google Scholar 

  5. Assefa, D. et al. Robust texture features for response monitoring of glioblastoma multiforme on‐weighted and‐FLAIR MR images: a preliminary investigation in terms of identification and segmentation. Med. Phys. 37, 1722–1736 (2010).

    Article  PubMed  Google Scholar 

  6. Razmjoo, A. et al. T2 analysis of the entire osteoarthritis initiative dataset. J. Orthop. Res. 39, 74–85 (2021).

    Article  PubMed  Google Scholar 

  7. Iriondo, C. et al. Towards understanding mechanistic subgroups of osteoarthritis: 8 year cartilage thickness trajectory analysis. J. Orthop. Res. 36, 1305–1317 (2020).

    Google Scholar 

  8. Desai, A. D. et al. The international workshop on osteoarthritis imaging knee MRI segmentation challenge: a multi-institute evaluation and analysis framework on a standardized dataset. Radiol. Artif. Intell. 3, e200078 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  9. Goodfellow, I., Bengio, Y. & Courville, A. Deep learning (MIT Press, 2016).

  10. Caliva, F., Iriondo, C., Martinez, A. M., Majumdar, S. & Pedoia, V. Distance map loss penalty term for semantic segmentation. Preprint at arXiv https://arxiv.org/abs/1908.03679 (2019).

  11. Hansen, M. S. & Kellman, P. Image reconstruction: an overview for clinicians. J. Magn. Reson. Imaging 41, 573–585 (2015).

    Article  PubMed  Google Scholar 

  12. Chaudhari, A. S. et al. Rapid knee MRI acquisition and analysis techniques for imaging osteoarthritis. J. Magn. Reson. Imaging 52, 1321–1339 (2020).

    Article  PubMed  Google Scholar 

  13. Deshmane, A., Gulani, V., Griswold, M. A. & Seiberlich, N. Parallel MR imaging. J. Magn. Reson. Imaging 36, 55–72 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Jakob, P. M., Grisowld, M. A., Edelman, R. R. & Sodickson, D. K. AUTO-SMASH: a self-calibrating technique for SMASH imaging. Magn. Reson. Mater. Phys. Biol. Med. 7, 42–54 (1998).

    Article  CAS  Google Scholar 

  15. Lustig, M., Donoho, D. & Pauly, J. M. Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58, 1182–1195 (2007).

    Article  PubMed  Google Scholar 

  16. Otazo, R., Candes, E. & Sodickson, D. K. Low‐rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components. Magn. Reson. Med. 73, 1125–1136 (2015).

    Article  PubMed  Google Scholar 

  17. Hammernik, K. & Knoll, F. Machine learning for image reconstruction. in Handbook of Medical Image Computing and Computer Assisted Intervention 25–64 (Elsevier, 2020).

  18. Hammernik, K. et al. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79, 3055–3071 (2018).

    Article  PubMed  Google Scholar 

  19. Souza, R., Lebel, R. M. & Frayne, R. A hybrid, dual domain, cascade of convolutional neural networks for magnetic resonance image reconstruction. PMLR 102, 437–446 (2019).

    Google Scholar 

  20. Liang, D., Cheng, J., Ke, Z. & Ying, L. Deep MRI reconstruction: unrolled optimization algorithms meet neural networks. Preprint at arXiv https://arxiv.org/abs/1907.11711 (2019).

  21. Recht, M. P. et al. Using deep learning to accelerate knee MRI at 3 T: results of an interchangeability study. Am. J. Roentgenol. 215, 1421–1429 (2020).

    Article  Google Scholar 

  22. Subhas, N. et al. Diagnostic interchangeability of deep convolutional neural networks reconstructed knee MR images: preliminary experience. Quant. Imaging Med. Surg. 10, 1748 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  23. Wang, S. et al. Accelerating magnetic resonance imaging via deep learning. in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) 514–517 (IEEE, 2016).

  24. Yang, Y., Sun, J., Li, H. & Xu, Z. Deep ADMM-Net for compressive sensing MRI. in Advances in Neural Information Processing Systems. 10–18 (MIT Press, 2016).

  25. Han, Y. et al. Deep learning with domain adaptation for accelerated projection‐reconstruction MR. Magn. Reson. Med. 80, 1189–1205 (2018).

    Article  PubMed  Google Scholar 

  26. Lee, D., Yoo, J. & Ye, J. C. Deep residual learning for compressed sensing MRI. in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017) 15–18 (IEEE, 2017).

  27. Yu, S. et al. Deep de-aliasing for fast compressive sensing MRI. Preprint at arXiv https://arxiv.org/abs/1705.07137 (2017).

  28. Yang, G. et al. DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Trans. Med. Imaging 37, 1310–1321 (2017).

    Article  Google Scholar 

  29. Jin, K. H., McCann, M. T., Froustey, E. & Unser, M. Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26, 4509–4522 (2017).

    Article  Google Scholar 

  30. Quan, T. M., Nguyen-Duc, T. & Jeong, W.-K. Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss. IEEE Trans. Med. Imaging 37, 1488–1497 (2018).

    Article  PubMed  Google Scholar 

  31. Liu, F., Samsonov, A., Chen, L., Kijowski, R. & Feng, L. SANTIS: sampling‐augmented neural network with incoherent structure for MR image reconstruction. Magn. Reson. Med. 82, 1890–1904 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  32. Han, Y., Sunwoo, L. & Ye, J. C. k-Space deep learning for accelerated MRI. IEEE Trans. Med. Imaging 39, 377–386 (2019).

    Article  PubMed  Google Scholar 

  33. Akçakaya, M., Moeller, S., Weingärtner, S. & Uğurbil, K. Scan‐specific robust artificial‐neural‐networks for k‐space interpolation (RAKI) reconstruction: database‐free deep learning for fast imaging. Magn. Reson. Med. 81, 439–453 (2019).

    Article  PubMed  Google Scholar 

  34. Schlemper, J., Caballero, J., Hajnal, J. V., Price, A. N. & Rueckert, D. A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imaging 37, 491–503 (2017).

    Article  PubMed  Google Scholar 

  35. Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R. & Rosen, M. S. Image reconstruction by domain-transform manifold learning. Nature 555, 487–492 (2018).

    Article  CAS  PubMed  Google Scholar 

  36. Chandler, D. M. Seven challenges in image quality assessment: past, present, and future research. Int. Sch. Res. Not. 2013, 1–54 (2013).

    Google Scholar 

  37. Wang, Z. & Bovik, A. C. Mean squared error: love it or leave it? A new look at signal fidelity measures. IEEE Signal. Process. Mag. 26, 98–117 (2009).

    Article  Google Scholar 

  38. Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).

    Article  PubMed  Google Scholar 

  39. Zbontar, J. et al. fastMRI: an open dataset and benchmarks for accelerated MRI. Preprint at arXiv https://arxiv.org/abs/1811.08839 (2018).

  40. Zhao, H., Gallo, O., Frosio, I. & Kautz, J. Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3, 47–57 (2016).

    Article  Google Scholar 

  41. Genzel, M., Macdonald, J. & März, M. Solving inverse problems with deep neural networks — robustness included? Preprint at arXiv https://arxiv.org/abs/2011.04268 (2020).

  42. Wang, Z., Simoncelli, E. P. & Bovik, A. C. Multiscale structural similarity for image quality assessment. in The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 2, 1398–1402 (IEEE, 2003).

  43. Sheikh, H. R. & Bovik, A. C. A visual information fidelity approach to video quality assessment. in The First International Workshop on Video Processing and Quality Metrics for Consumer Electronics 7, 2 (Academia, 2005).

  44. Cheng, K. et al. Addressing the false negative problem of deep learning MRI reconstruction models by adversarial attacks and robust training. in Medical Imaging with Deep Learning 121–135 (PMLR, 2020).

  45. Antun, V., Renna, F., Poon, C., Adcock, B. & Hansen, A. C. On instabilities of deep learning in image reconstruction and the potential costs of AI. Proc. Natl. Acad. Sci. USA 117, 30088–30095 (2020).

    Article  CAS  PubMed  Google Scholar 

  46. McAlindon, T. E. et al. OARSI guidelines for the non-surgical management of knee osteoarthritis. Osteoarthritis Cartilage 22, 363–388 (2014).

    Article  CAS  PubMed  Google Scholar 

  47. Wang, Z., Chen, J. & Hoi, S. C. H. Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3365–3387 (2020).

    Article  Google Scholar 

  48. Chaudhari, A., Fang, Z., Lee, J. H., Gold, G. & Hargreaves, B. Deep learning super-resolution enables rapid simultaneous morphological and quantitative magnetic resonance imaging. in International Workshop on Machine Learning for Medical Image Reconstruction 3–11 (Springer, 2018).

  49. Liu, F., Feng, L. & Kijowski, R. MANTIS: model‐augmented neural network with incoherent k‐space Sampling for efficient MR parameter mapping. Magn. Reson. Med. 82, 174–188 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  50. Eckstein, F. et al. Quantitative MRI measures of cartilage predict knee replacement: a case–control study from the Osteoarthritis Initiative. Ann. Rheum. Dis. 72, 707–714 (2013).

    Article  PubMed  Google Scholar 

  51. Bowes, M. A. et al. Precision, reliability, and responsiveness of a novel automated quantification tool for cartilage thickness: data from the Osteoarthritis Initiative. J. Rheumatol. 47, 282–289 (2020).

    Article  PubMed  Google Scholar 

  52. Bredbenner, T. L. et al. Statistical shape modeling describes variation in tibia and femur surface geometry between control and incidence groups from the Osteoarthritis Initiative database. J. Biomech. 43, 1780–1786 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  53. Morales Martinez, A. et al. Learning osteoarthritis imaging biomarkers from bone surface spherical encoding. Magn. Reson. Med. 84, 2190–2203 (2020).

    Article  PubMed  Google Scholar 

  54. Pedoia, V., Lee, J., Norman, B., Link, T. M. & Majumdar, S. Diagnosing osteoarthritis from T2 maps using deep learning: an analysis of the entire Osteoarthritis Initiative baseline cohort. Osteoarthritis Cartilage 27, 1002–1010 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Carballido‐Gamio, J. et al. Spatial analysis of magnetic resonance and relaxation times improves classification between subjects with and without osteoarthritis. Med. Phys. 36, 4059–4067 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  56. Carballido-Gamio, J., Joseph, G. B., Lynch, J. A., Link, T. M. & Majumdar, S. Longitudinal analysis of MRI T2 knee cartilage laminar organization in a subset of patients from the osteoarthritis initiative: a texture approach. Magn. Reson. Med. 65, 1184–1194 (2011).

    Article  PubMed  Google Scholar 

  57. Fink, A., Kosecoff, J., Chassin, M. & Brook, R. H. Consensus methods: characteristics and guidelines for use. Am. J. Public Health 74, 979–983 (1984).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  58. Tibrewala, R. et al. Computer‐aided detection AI reduces interreader variability in grading hip abnormalities with MRI. J. Magn. Reson. Imaging 52, 1163–1172 (2020).

    Article  PubMed  Google Scholar 

  59. Smith, C. P. et al. Intra‐and interreader reproducibility of PI‐RADSv2: a multireader study. J. Magn. Reson. Imaging 49, 1694–1703 (2019).

    Article  PubMed  Google Scholar 

  60. Dunn, W. R. et al. Multirater agreement of arthroscopic meniscal lesions. Am. J. Sports Med. 32, 1937–1940 (2004).

    Article  PubMed  Google Scholar 

  61. Harolds, J. A., Parikh, J. R., Bluth, E. I., Dutton, S. C. & Recht, M. P. Burnout of radiologists: frequency, risk factors, and remedies: a report of the ACR commission on human resources. J. Am. Coll. Radiol. 13, 411–416 (2016).

    Article  PubMed  Google Scholar 

  62. Prasoon, A. et al. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. in International conference on medical image computing and computer-assisted intervention 246–253 (Springer, 2013).

  63. Norman, B. et al. Use of 2D U-Net convolutional neural networks for automated cartilage and meniscus segmentation of knee MR imaging data to determine relaxometry and morphometry. Arthritis Rheumatol. 79, 3184–3189 (2018).

    Google Scholar 

  64. Liu, F. et al. Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging. Magn. Reson. Med. 79, 2379–2391 (2018).

    Article  PubMed  Google Scholar 

  65. Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. in International conference on medical image computing and computer-assisted intervention 424–432 (Springer, 2016).

  66. Milletari, F., Navab, N. & Ahmadi, S.-A. V-net: fully convolutional neural networks for volumetric medical image segmentation. in 2016 Fourth International Conference on 3D Vision (3DV) 565–571 (IEEE, 2016).

  67. Jørgensen, D. R., Lillholm, M., Genant, H. K. & Dam, E. B. On subregional analysis of cartilage loss from knee MRI. Cartilage 4, 121–130 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  68. Zeng, G. et al. Latent3DU-net: Multi-level latent shape space constrained 3D U-net for automatic segmentation of the proximal femur from radial MRI of the hip. in International Workshop on Machine Learning in Medical Imaging 188–196 (Springer, 2018).

  69. Gaj, S., Yang, M., Nakamura, K. & Li, X. Automated cartilage and meniscus segmentation of knee MRI with conditional generative adversarial networks. Magn. Reson. Med. 84, 437–449 (2020).

    Article  PubMed  Google Scholar 

  70. Wirth, W. et al. Accuracy and longitudinal reproducibility of quantitative femorotibial cartilage measures derived from automated U-Net-based segmentation of two different MRI contrasts: data from the Osteoarthritis Initiative healthy reference cohort. Magn. Reson. Mater. Phys. Biol. Med. 34, 337–354 (2021).

    Article  Google Scholar 

  71. Cheng, R. et al. Fully automated patellofemoral MRI segmentation using holistically nested networks: implications for evaluating patellofemoral osteoarthritis, pain, injury, pathology, and adolescent development. Magn. Reson. Med. 83, 139–153 (2020).

    Article  PubMed  Google Scholar 

  72. Schock, J. et al. A method for semantic knee bone and cartilage segmentation with deep 3D shape fitting using data from the Osteoarthritis Initiative. in International Workshop on Shape in Medical Imaging 85–94 (Springer, 2020).

  73. Ambellan, F., Tack, A., Ehlke, M. & Zachow, S. Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: data from the osteoarthritis initiative. Med. Image Anal. 52, 109–118 (2019).

    Article  PubMed  Google Scholar 

  74. Liu, F. SUSAN: segment unannotated image structure using adversarial network. Magn. Reson. Med. 81, 3330–3345 (2019).

    PubMed  Google Scholar 

  75. Perslev, M., Dam, E. B., Pai, A. & Igel, C. One network to segment them all: a general, lightweight system for accurate 3D medical image segmentation. in International Conference on Medical Image Computing and Computer-Assisted Intervention 30–38 (Springer, 2019).

  76. Mortazi, A., Karim, R., Rhode, K., Burt, J. & Bagci, U. CardiacNET: segmentation of left atrium and proximal pulmonary veins from MRI using multi-view CNN. in International Conference on Medical Image Computing and Computer-Assisted Intervention 377–385 (Springer, 2017).

  77. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. in Proceedings of the European conference on computer vision (ECCV). Springer Science+Business Media. 801–818 (Springer, 2018).

  78. Calivá, F. et al. Breaking speed limits with simultaneous ultra-fast MRI reconstruction and tissue segmentation. Proc. Mach. Learn. Res. 1, 17 (2020).

    Google Scholar 

  79. Brui, E. et al. Deep learning-based fully automatic segmentation of wrist cartilage in MR images. NMR Biomed. 33, e4320 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Deniz, C. M. et al. Segmentation of the proximal femur from MR images using deep convolutional neural networks. Sci. Rep. 8, 1–14 (2018).

    Article  CAS  Google Scholar 

  81. Zeng, G. et al. Entropy guided unsupervised domain adaptation for cross-center hip cartilage segmentation from MRI. in International Conference on Medical Image Computing and Computer-Assisted Intervention 447–456 (Springer, 2020).

  82. Medina, G., Buckless, C. G., Thomasson, E., Oh, L. S. & Torriani, M. Deep learning method for segmentation of rotator cuff muscles on MR images. Skeletal Radiol. 50, 683–692 (2021).

    Article  PubMed  Google Scholar 

  83. Boutillon, A., Borotikar, B., Burdin, V. & Conze, P.-H. Combining shape priors with conditional adversarial networks for improved scapula segmentation in MR images. in 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) 1164–1167 (IEEE, 2020).

  84. Conze, P. H., Brochard, S., Burdin, V., Sheehan, F. T. & Pons, C. Healthy versus pathological learning transferability in shoulder muscle MRI segmentation using deep convolutional encoder-decoders. Comput. Med. Imaging Graph. 83, 101733 (2020).

    Article  PubMed  Google Scholar 

  85. Cantarelli Rodrigues, T. et al. Three-dimensional MRI bone models of the glenohumeral joint using deep learning: evaluation of normal anatomy and glenoid bone loss. Radiol. Artif. Intell. 2, e190116 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  86. Di Giacomo, G., de Gasperis, N. & Scarso, P. Bipolar bone defect in the shoulder anterior dislocation. Knee Surg. Sports Traumatol. Arthrosc. 24, 479–488 (2016).

    Article  PubMed  Google Scholar 

  87. Eckstein, F. et al. Cartilage thickness change as an imaging biomarker of knee osteoarthritis progression data from the FNIH OA biomarkers consortium. Arthritis Rheumatol. 67, 3184 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  88. Samek, W., Montavon, G., Vedaldi, A., Hansen, L. K. & Müller, K.-R. Explainable AI: interpreting, explaining and visualizing deep learning. Vol. 11700 (Springer Nature, 2019).

  89. Peterfy, C. G. et al. Whole-organ magnetic resonance imaging score (WORMS) of the knee in osteoarthritis. Osteoarthritis Cartilage 12, 177–190 (2004).

    Article  CAS  PubMed  Google Scholar 

  90. Hunter, D. J. et al. The reliability of a new scoring system for knee osteoarthritis MRI and the validity of bone marrow lesion assessment: BLOKS (Boston–Leeds Osteoarthritis Knee Score). Ann. Rheum. Dis. 67, 206–211 (2008).

    Article  CAS  PubMed  Google Scholar 

  91. Hunter, D. J. et al. Evolution of semi-quantitative whole joint assessment of knee OA: MOAKS (MRI osteoarthritis knee score). Osteoarthritis Cartilage 19, 990–1002 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  92. Lee, S. et al. Scoring hip osteoarthritis with MRI (SHOMRI): a whole joint osteoarthritis evaluation system. J. Magn. Reson. Imaging 41, 1549–1557 (2015).

    Article  PubMed  Google Scholar 

  93. Liu, F. et al. Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. Radiology 289, 160–169 (2018).

    Article  PubMed  Google Scholar 

  94. Pedoia, V. et al. 3D convolutional neural networks for detection and severity staging of meniscus and PFJ cartilage morphological degenerative changes in osteoarthritis and anterior cruciate ligament subjects. J. Magn. Reson. Imaging 49, 400–410 (2019).

    Article  PubMed  Google Scholar 

  95. Roblot, V. et al. Artificial intelligence to diagnose meniscus tears on MRI. Diagn. Interv. Imaging 100, 243–249 (2019).

    Article  CAS  PubMed  Google Scholar 

  96. Fritz, B., Marbach, G., Civardi, F., Fucentese, S. F. & Pfirrmann, C. W. A. Deep convolutional neural network-based detection of meniscus tears: comparison with radiologists and surgery as standard of reference. Skeletal Radiol. 49, 1207–1217 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  97. Namiri, N. K. et al. Deep learning for hierarchical severity staging of anterior cruciate ligament injuries from MRI. Radiol. Artif. Intell. 2, e190207 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  98. Liu, F. et al. Fully automated diagnosis of anterior cruciate ligament tears on knee MR images by using deep learning. Radiol. Artif. Intell. 1, 180091 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  99. Bien, N. et al. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med. 15, e1002699 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  100. Astuto, B. et al. Automatic deep learning assisted detection and grading of abnormalities in knee MRI studies. Radiol. Artif. Intell. 3, e219001 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  101. Jordan, J. M. et al. Prevalence of hip symptoms and radiographic and symptomatic hip osteoarthritis in African Americans and Caucasians: the Johnston County Osteoarthritis Project. J. Rheumatol. 36, 809–815 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  102. Panfilov, E., Tiulpin, A., Klein, S., Nieminen, M. T. & Saarakkala, S. Improving robustness of deep learning based knee MRI segmentation: mixup and adversarial domain adaptation. in Proceedings of the IEEE International Conference on Computer Vision Workshops. Computer Vision Foundation (CVF). (IEEE, 2019).

  103. Shah, R. F., Bini, S. A., Martinez, A. M., Pedoia, V. & Vail, T. P. Incremental inputs improve the automated detection of implant loosening using machine-learning algorithms. Bone Jt. J. 102, 101–106 (2020).

    Article  Google Scholar 

  104. Deng, J. et al. Imagenet: a large-scale hierarchical image database. in 2009 IEEE conference on computer vision and pattern recognition 248–255 (IEEE, 2009).

  105. Liu, B., Luo, J. & Huang, H. Toward automatic quantification of knee osteoarthritis severity using improved Faster R-CNN. Int. J. Comput. Assist. Radiol. Surg. 15, 457–466 (2020).

    Article  PubMed  Google Scholar 

  106. Tiulpin, A., Thevenot, J., Rahtu, E., Lehenkari, P. & Saarakkala, S. Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Sci. Rep. 8, 1–10 (2018).

    Article  CAS  Google Scholar 

  107. Norman, B., Pedoia, V., Noworolski, A., Link, T. M. & Majumdar, S. Applying densely connected convolutional neural networks for staging osteoarthritis severity from plain radiographs. J. Digit. Imaging 32, 471–477 (2019).

    Article  PubMed  Google Scholar 

  108. Xue, Y., Zhang, R., Deng, Y., Chen, K. & Jiang, T. A preliminary examination of the diagnostic value of deep learning in hip osteoarthritis. PLoS One 12, e0178992 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  109. von Schacky, C. E. et al. Development and validation of a multitask deep learning model for severity grading of hip osteoarthritis features on radiographs. Radiology 295, 136–145 (2020).

    Article  Google Scholar 

  110. Tiulpin, A. et al. Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data. Sci. Rep. 9, 1–11 (2019).

    Article  Google Scholar 

  111. Huang, Z., Ding, C., Li, T. & Yu, S. P.-C. Current status and future prospects for disease modification in osteoarthritis. Rheumatology 57, iv108–iv123 (2018).

    Article  CAS  PubMed  Google Scholar 

  112. Loeser, R. F., Goldring, S. R., Scanzello, C. R. & Goldring, M. B. Osteoarthritis: a disease of the joint as an organ. Arthritis Rheum. 64, 1697 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  113. Neogi, T. et al. Magnetic resonance imaging–based three‐dimensional bone shape of the knee predicts onset of knee osteoarthritis: data from the Osteoarthritis Initiative. Arthritis Rheum. 65, 2048–2058 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  114. Kohn, M. D., Sassoon, A. A. & Fernando, N. D. Classifications in brief: Kellgren-Lawrence classification of osteoarthritis. Clin. Orthop. Relat. Res. 474, 1886–1893 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  115. Bowes, M. A. et al. Machine-learning, MRI bone shape and important clinical outcomes in osteoarthritis: data from the Osteoarthritis Initiative. Ann. Rheum. Dis. 80, 502–508 (2020).

    Article  Google Scholar 

  116. Tolpadi, A. A., Lee, J. J., Pedoia, V. & Majumdar, S. Deep learning predicts total knee replacement from magnetic resonance images. Sci. Rep. 10, 1–12 (2020).

    Article  Google Scholar 

  117. Roemer, F. W. et al. MRI-based screening for structural definition of eligibility in clinical DMOAD trials: rapid osteoarthritis MRI eligibility score (ROAMES). Osteoarthritis Cartilage 28, 71–81 (2020).

    Article  CAS  PubMed  Google Scholar 

  118. Namiri, N. K. et al. Deep learning for large scale MRI-based morphological phenotyping of osteoarthritis. Sci. Rep. 11, 1–10 (2021).

    Article  Google Scholar 

  119. Leung, K. et al. Prediction of total knee replacement and diagnosis of osteoarthritis by using deep learning on knee radiographs: data from the osteoarthritis initiative. Radiology 296, 584–593 (2020).

    Article  PubMed  Google Scholar 

  120. Friedman, B. R., Jones, J. P., Chavez-Munoz, G., Salmon, A. P. & Merritt, C. R. B. Principles of MRI. (McGraw-Hill Health Professions Division, 1989).

  121. Uecker, M. et al. ESPIRiT — an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Magn. Reson. Med. 71, 990–1001 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  122. Lustig, M., Donoho, D. L., Santos, J. M. & Pauly, J. M. Compressed sensing MRI. IEEE Signal. Process. Mag. 25, 72 (2008).

    Article  Google Scholar 

  123. Sodickson, D. K. & Manning, W. J. Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays. Magn. Reson. Med. 38, 591–603 (1997).

    Article  CAS  PubMed  Google Scholar 

  124. Griswold, M. A. et al. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn. Reson. Med 47, 1202–1210 (2002).

    Article  PubMed  Google Scholar 

  125. Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. in International Conference on Medical image computing and computer-assisted intervention 234–241 (Springer, 2015).

  126. Goodfellow, I. J. et al. Generative adversarial networks. Adv. Neural Inf. Process. Syst. 3, 1–9 (2014).

    Google Scholar 

Download references

Author information

Author notes

  1. These authors contributed equally: Francesco Calivà, Nikan K. Namiri.

Authors and Affiliations

  1. Department of Radiology and Biomedical Imaging and Center for Intelligent Imaging, University of California, San Francisco, San Francisco, CA, USA

    Francesco Calivà, Nikan K. Namiri, Valentina Pedoia, Eugene Ozhinsky & Sharmila Majumdar

  2. Section of Rheumatology, Department of Medicine, Boston University School of Medicine, Boston, MA, USA

    Maureen Dubreuil

Authors
  1. Francesco Calivà
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikan K. Namiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maureen Dubreuil
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Valentina Pedoia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eugene Ozhinsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sharmila Majumdar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to all aspects of the article.

Corresponding author

Correspondence to Sharmila Majumdar.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Rheumatology thanks E. B. Dam, F. Berenbaum, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

FASTMRI challenge: http://fastmri.med.nyu.edu/

MC-MRRec challenge: https://sites.google.com/view/calgary-campinas-dataset/mr-reconstruction-challenge

Glossary

Machine learning

Computer programmes that learn how to perform tasks with some degree of automation and are based in statistics and algorithms.

Deep learning

A type of machine learning that performs multiple layers of mathematical algorithms (analogous to artificial neurons) on input data to execute a pre-defined task as output.

Classification

Predicting an observation based on a training dataset that includes observations about the predicted variable. The prediction assumes categorical values. In osteoarthritis, an example is predicting the presence or absence of a knee meniscus tear.

Regression

Predicting an observation based on a training dataset that includes observations about the predicted variable that assumes a continuous value (that is, not categorical); for example, the task of prescribing the imaging acquisition plane in MRI.

Segmentation

A form of classification at the pixel level of an image to identify a region of interest (ROI). The goal of segmentation is to assign each pixel to a label that describes the category to which the pixels belongs.

Raw data

Refers to the data stored during the acquisition of an MRI sequence as a function of time. Also known as k-space in MRI.

Supervised learning

The process of learning the relationship between input data and their labels, where all the samples in the training dataset have been labelled.

Ground truth

Information provided by empirical evidence rather than by the inference of a machine learning model; for example, the pixel-by-pixel annotation of a particular region of interest in a segmentation task, or the fully sampled image in accelerated MRI reconstruction.

Unsupervised learning

The process of learning the relationship between input data and their labels, where the data have no labels and the role of the algorithm is to find the underlying structure of the data to use it on the task at hand.

Model training

The stage in machine learning in which an algorithm (regression or classification) is instructed to perform a task. Training can be supervised, semi-supervised or unsupervised. Also known as learning.

Loss function

A mathematical entity for quantifying how well a machine learning algorithm models the data. Higher values indicate poorer modelling ability. During training, the loss function is coupled with an optimizer, which is used to tune the parameters of the machine learning or deep learning algorithm to minimize the loss function and ultimately maximize algorithm performance.

k-space

The k-space represents the spatial frequency information in two or three dimensions of an object, and each point in the k-space data matrix contains a portion of the information for the complete image. Also known as raw data in MRI.

Fourier transform

A mathematical transform utilized in image processing to decompose an image into its sine and cosine components, which allows mapping from the spatial domain of an image to its frequency domain. In MRI, the Fourier transform and its inverse maps raw data to image space and vice versa, respectively.

Image-domain learning

The procedure in which an algorithm learns to reconstruct a fully sampled MR image starting from an undersampled MR image.

k-space-domain learning

The procedure in MRI reconstruction in which an algorithm learns to fill in the missing information in an undersampled k-space to ultimately obtain a fully sampled k-space.

Domain mapping

In image reconstruction, this term refers to the procedure where an algorithm learns the mapping from undersampled k-space to fully sampled image.

Aliased image

Aliasing is a signal processing term referring to effects that cause different signals to be indistinguishable. Aliasing can also describe artefacts or distortions in a signal reconstructed from a limited number of samples, which cause the output signal to differ from the original signal.

Relaxometry maps

Quantitative maps reporting measurements of relaxation times from MR images. T1, T2 and T2* are common relaxometry maps that all require appropriate pulse sequence and parameters.

T2 relaxation time

A metric to quantify the T2 relaxation rate in a region of interest. The relaxation time is described by a combination of exponential decay curves, in part owing to different compartments of water in tissues.

Encoder–decoder

A machine-learning algorithm comprising an encoder, which is used to condense the input into a smaller (encoded), meaningful and descriptive representation, and a decoder, which re-expands the encoded representation until the original input is reconstructed.

Domain adaptation

A set of techniques that are used to apply an algorithm trained on a source domain to a different related domain termed the target domain, with the aim of minimizing performance drop.

Neural network

A mathematical system comprising artificial neurons that mimic human neural networks and that are used to identify relationships in datasets. In AI, neural networks are considered building blocks of the most powerful machine learning algorithms. Multiple layers of neurons are concatenated to produce the popular deep neural networks.

Dice similarity coefficient

A statistic that is used to quantify the similarity between two samples; in image segmentation, it measures the overlap between the ground truth and the model-produced segmentation.

Volumetric overlap error

A metric for quantifying the dissimilarity between two datasets. In segmentation, it is computed as the overlap between ground truth and a model-produced segmentation, divided by the union of the two segmentations, subtracted from 1.

Root-mean-squared

A statistic for quantifying the difference between values predicted by a model and the observed values (ground truth). It is computed as the square root of the average of squared errors between predicted and expected values.

Coefficient of variation

A statistical measure of the dispersion of a probability distribution, which is calculated as the ratio between the standard deviation and the average.

Average symmetric surface distance

(ASSD). A metric for evaluating segmentation performance, which is computed by averaging all the distances from points on the boundary of the region segmented by a model to the boundary of the ground truth, and vice versa.

Figure of merit

A quantity utilized to characterize the performance of an algorithm. Sometimes, it is used interchangeably with the term loss function.

Interpretability

A model is interpretable when the relationship between input and predicted output is clear to the user.

Annotation

Labelling the data so that it can be utilized for training and testing machine learning algorithms. In MRI, annotation can be done at different levels of granularity depending on the type of task, such as classification tasks, segmentation or regression.

Gradient boosting

A machine learning technique used for regression and classification. Starting from multiple prediction models, gradient boosting is used to create a prediction model in the form of an ensemble of those models.

Occlusion maps

An image overlay representing the change in probability for a model’s prediction as a function of position in the image.

Multitask learning

Training a machine learning algorithm to simultaneously learn to perform more than one task at a time, which forces the algorithm to identify a more general model for the data.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Calivà, F., Namiri, N.K., Dubreuil, M. et al. Studying osteoarthritis with artificial intelligence applied to magnetic resonance imaging. Nat Rev Rheumatol 18, 112–121 (2022). https://doi.org/10.1038/s41584-021-00719-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41584-021-00719-7

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research