Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Deep learning in rheumatological image interpretation

Nature Reviews Rheumatology volume 20, pages 182–195 (2024)Cite this article

Subjects

Abstract

Artificial intelligence techniques, specifically deep learning, have already affected daily life in a wide range of areas. Likewise, initial applications have been explored in rheumatology. Deep learning might not easily surpass the accuracy of classic techniques when performing classification or regression on low-dimensional numerical data. With images as input, however, deep learning has become so successful that it has already outperformed the majority of conventional image-processing techniques developed during the past 50 years. As with any new imaging technology, rheumatologists and radiologists need to consider adapting their arsenal of diagnostic, prognostic and monitoring tools, and even their clinical role and collaborations. This adaptation requires a basic understanding of the technical background of deep learning, to efficiently utilize its benefits but also to recognize its drawbacks and pitfalls, as blindly relying on deep learning might be at odds with its capabilities. To facilitate such an understanding, it is necessary to provide an overview of deep-learning techniques for automatic image analysis in detecting, quantifying, predicting and monitoring rheumatic diseases, and of currently published deep-learning applications in radiological imaging for rheumatology, with critical assessment of possible limitations, errors and confounders, and conceivable consequences for rheumatologists and radiologists in clinical practice.

Key points

  • The number of research studies on deep learning in rheumatological imaging has grown rapidly during the past 5 years, but they mainly consist of pilot studies that require external validation.

  • Confounding factors and errors in deep-learning methods need to be ruled out before deep learning can be applied in clinical practice, for which the intended use should be strictly defined.

  • Deep-learning techniques, together with mapping to explain their reasoning, will enable hypothesis-free image analysis and could identify new imaging biomarkers.

  • Deep learning might assist rheumatologists and radiologists in interpreting rheumatological images, increasing their diagnostic, prognostic and monitoring accuracy, and decreasing workloads and costs.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Optical illusion illustrating the strength and weakness of both computer and human visual systems.
Fig. 2: Different routes to calculating an outcome measure or category from an image.
Fig. 3: Example of convolution, as a core building block in deep learning in image analysis.
Fig. 4: Categorization of published studies on deep learning in rheumatological imaging.
Fig. 5: AI-based comparison between baseline and follow-up MRI of the wrist, without prior knowledge.

Similar content being viewed by others

References

  1. Kingsmore, K. M., Puglisi, C. E., Grammer, A. C. & Lipsky, P. E. An introduction to machine learning and analysis of its use in rheumatic diseases. Nat. Rev. Rheumatol. 17, 710–730 (2021).

    Article  PubMed  Google Scholar 

  2. Christodoulou, E. et al. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J. Clin. Epidemiol. 110, 12–22 (2019).

    Article  PubMed  Google Scholar 

  3. Calivà, F. et al. Studying osteoarthritis with artificial intelligence applied to magnetic resonance imaging. Nat. Rev. Rheumatol. 18, 112–121 (2022).

    Article  PubMed  Google Scholar 

  4. Cipolletta, E. et al. Artificial intelligence for ultrasound informative image selection of metacarpal head cartilage. a pilot study. Front. Med. 8, 589197 (2021).

    Article  Google Scholar 

  5. Prasoon, A. et al. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med. Image Comput. Comput. Assist. Interv. 16, 246–253 (2013).

    PubMed  Google Scholar 

  6. Banerjee, S., Bhunia, S. & Schaefer, G. Osteophyte detection for hand osteoarthritis identification in X-ray images using CNNs. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 6196–6199 (2011).

    Google Scholar 

  7. Folle, L. et al. Advanced neural networks for classification of MRI in psoriatic arthritis, seronegative, and seropositive rheumatoid arthritis. Rheumatology 61, 4945–4951 (2022).

    Article  PubMed  Google Scholar 

  8. Hassanzadeh, T. et al. AB0205 RA treatment effects in wrist MRIs, determined by deep learning. Ann. Rheum. Dis. 82, 1286 (2023).

    Google Scholar 

  9. Abedin, J. et al. Predicting knee osteoarthritis severity: comparative modeling based on patient’s data and plain X-ray images. Sci. Rep. 9, 5761 (2019).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  10. Leung, K. et al. Prediction of total knee replacement and diagnosis of osteoarthritis by using deep learning on knee radiographs: data from the osteoarthritis initiative. Radiology 296, 584–593 (2020).

    Article  PubMed  Google Scholar 

  11. Jia, J. et al. Automatic pulmonary function estimation from chest CT scans using deep regression neural networks: the relation between structure and function in systemic sclerosis. IEEE Access 11, 135272–135282 (2023).

    Article  Google Scholar 

  12. Chang, G. H. et al. Assessment of knee pain from MR imaging using a convolutional Siamese network. Eur. Radiol. 30, 3538–3548 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  13. Ras, G., Xie, N., Gerven, M. V. & Doran, D. Explainable deep learning: a field guide for the uninitiated. J. Artif. Int. Res. 73, 68 (2022).

    MathSciNet  Google Scholar 

  14. National Institutes of Health. The Osteoarthritis Initiative. NIMH Data Archive https://nda.nih.gov/oai (2023).

  15. Chen, N. et al. A fully automatic target detection and quantification strategy based on object detection convolutional neural network YOLOv3 for one-step X-ray image grading. Anal. Methods 15, 164–170 (2023).

    Article  PubMed  Google Scholar 

  16. Chen, P., Gao, L., Shi, X., Allen, K. & Yang, L. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. Comput. Med. Imaging Graph. 75, 84–92 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Liu, B., Luo, J. & Huang, H. Toward automatic quantification of knee osteoarthritis severity using improved Faster R-CNN. Int. J. Comput. Assist. Radiol. Surg. 15, 457–466 (2020).

    Article  PubMed  Google Scholar 

  18. Norman, B., Pedoia, V., Noworolski, A., Link, T. M. & Majumdar, S. Applying densely connected convolutional neural networks for staging osteoarthritis severity from plain radiographs. J. Digit. Imaging 32, 471–477 (2019).

    Article  PubMed  Google Scholar 

  19. Tiulpin, A., Thevenot, J., Rahtu, E., Lehenkari, P. & Saarakkala, S. Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Sci. Rep. 8, 1727 (2018).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  20. Tolpadi, A. A., Lee, J. J., Pedoia, V. & Majumdar, S. Deep learning predicts total knee replacement from magnetic resonance images. Sci. Rep. 10, 6371 (2020).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  21. Tiulpin, A. et al. Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data. Sci. Rep. 9, 20038 (2019).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  22. Hirvasniemi, J. et al. The KNee OsteoArthritis Prediction (KNOAP2020) challenge: an image analysis challenge to predict incident symptomatic radiographic knee osteoarthritis from MRI and X-ray images. Osteoarthritis Cartilage 31, 115–125 (2023).

    Article  CAS  PubMed  Google Scholar 

  23. Jansen, M. P. et al. Artificial intelligence in osteoarthritis: repair by knee joint distraction shows association of pain, radiographic and immunological outcomes. Rheumatology 62, 2789–2796 (2022).

    Article  PubMed Central  Google Scholar 

  24. Ambellan, F., Tack, A., Ehlke, M. & Zachow, S. Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: data from the Osteoarthritis Initiative. Med. Image Anal. 52, 109–118 (2019).

    Article  PubMed  Google Scholar 

  25. Cheng, R. et al. Fully automated patellofemoral MRI segmentation using holistically nested networks: implications for evaluating patellofemoral osteoarthritis, pain, injury, pathology, and adolescent development. Magn. Reson. Med. 83, 139–153 (2020).

    Article  PubMed  Google Scholar 

  26. Gaj, S., Yang, M., Nakamura, K. & Li, X. Automated cartilage and meniscus segmentation of knee MRI with conditional generative adversarial networks. Magn. Reson. Med. 84, 437–449 (2020).

    Article  PubMed  Google Scholar 

  27. Liu, F. et al. Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging. Magn. Reson. Med. 79, 2379–2391 (2018).

    Article  PubMed  Google Scholar 

  28. Norman, B., Pedoia, V. & Majumdar, S. Use of 2D U-net convolutional neural networks for automated cartilage and meniscus segmentation of knee MR imaging data to determine relaxometry and morphometry. Radiology 288, 177–185 (2018).

    Article  PubMed  Google Scholar 

  29. Panfilov, E., Tiulpin, A., Nieminen, M. T., Saarakkala, S. & Casula, V. Deep learning-based segmentation of knee MRI for fully automatic subregional morphological assessment of cartilage tissues: data from the Osteoarthritis Initiative. J. Orthop. Res. 40, 1113–1124 (2022).

    Article  PubMed  Google Scholar 

  30. Razmjoo, A. et al. T2 analysis of the entire osteoarthritis initiative dataset. J. Orthop. Res. 39, 74–85 (2021).

    Article  PubMed  Google Scholar 

  31. Chang, G. H. et al. Subchondral bone length in knee osteoarthritis: a deep learning-derived imaging measure and its association with radiographic and clinical outcomes. Arthritis Rheumatol. 73, 2240–2248 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  32. Pedoia, V. et al. 3D convolutional neural networks for detection and severity staging of meniscus and PFJ cartilage morphological degenerative changes in osteoarthritis and anterior cruciate ligament subjects. J. Magn. Reson. Imaging 49, 400–410 (2019).

    Article  PubMed  Google Scholar 

  33. Liu, F. et al. Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. Radiology 289, 160–169 (2018).

    Article  PubMed  Google Scholar 

  34. Astuto, B. et al. Automatic deep learning-assisted detection and grading of abnormalities in knee MRI studies. Radiol. Artif. Intell. 3, e200165 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  35. Namiri, N. K. et al. Deep learning for large scale MRI-based morphological phenotyping of osteoarthritis. Sci. Rep. 11, 10915 (2021).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  36. Brui, E. et al. Deep learning-based fully automatic segmentation of wrist cartilage in MR images. NMR Biomed. 33, e4320 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Üreten, K. et al. Detection of hip osteoarthritis by using plain pelvic radiographs with deep learning methods. Skeletal Radiol. 49, 1369–1374 (2020).

    Article  PubMed  Google Scholar 

  38. Xue, Y., Zhang, R., Deng, Y., Chen, K. & Jiang, T. A preliminary examination of the diagnostic value of deep learning in hip osteoarthritis. PLoS ONE 12, e0178992 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  39. von Schacky, C. E. et al. Development and validation of a multitask deep learning model for severity grading of hip osteoarthritis features on radiographs. Radiology 295, 136–145 (2020).

    Article  Google Scholar 

  40. Radke, K. L. et al. Adaptive IoU thresholding for improving small object detection: a proof-of-concept study of hand erosions classification of patients with rheumatic arthritis on X-ray images. Diagnostics 13, 104 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  41. Murakami, S., Hatano, K., Tan, J., Kim, H. & Aoki, T. Automatic identification of bone erosions in rheumatoid arthritis from hand radiographs based on deep convolutional neural network. Multimed. Tools Appl. 77, 10921–10937 (2018).

    Article  Google Scholar 

  42. Hirano, T. et al. Development and validation of a deep-learning model for scoring of radiographic finger joint destruction in rheumatoid arthritis. Rheumatol. Adv. Prac. 3, rkz047 (2019).

    Article  Google Scholar 

  43. Chaturvedi, N. DeepRA: predicting joint damage from radiographs using CNN with attention. Preprint at https://doi.org/10.48550/arXiv.2102.06982 (2021).

  44. Rohrbach, J., Reinhard, T., Sick, B. & Dürr, O. Bone erosion scoring for rheumatoid arthritis with deep convolutional neural networks. Comput, Electr. Eng. 78, 472–481 (2019).

    Article  Google Scholar 

  45. Üreten, K., Erbay, H. & Maraş, H. H. Detection of rheumatoid arthritis from hand radiographs using a convolutional neural network. Clin. Rheumatol. 39, 969–974 (2020).

    Article  PubMed  Google Scholar 

  46. Izumi, K. et al. Detecting hand joint ankylosis and subluxation in radiographic images using deep learning: a step in the development of an automatic radiographic scoring system for joint destruction. PLoS ONE 18, e0281088 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Fiorentino, M. C., Moccia, S., Cipolletta, E., Filippucci, E. & Frontoni, E. A learning approach for informative-frame selection in US rheumatology images. in: Cristani, M., Prati, A., Lanz, O., Messelodi, S., Sebe, N. (eds) New Trends in Image Analysis and Processing — ICIAP 2019. ICIAP 2019. Lecture Notes in Computer Science, vol 11808. https://doi.org/10.1007/978-3-030-30754-7_23 (Springer, Cham, 2019).

  48. Tang, J. et al. Enhancing convolutional neural network scheme for rheumatoid arthritis grading with limited clinical data. Chin. Phys. B 28, 038701 (2019).

    Article  ADS  Google Scholar 

  49. Hemalatha, R. J., Vijaybaskar, V. & Thamizhvani, T. R. Automatic localization of anatomical regions in medical ultrasound images of rheumatoid arthritis using deep learning. Proc. Inst. Mech. Eng. H. 233, 657–667 (2019).

    Article  CAS  PubMed  Google Scholar 

  50. Christensen, A. B. H., Just, S. A., Andersen, J. K. H. & Savarimuthu, T. R. Applying cascaded convolutional neural network design further enhances automatic scoring of arthritis disease activity on ultrasound images from rheumatoid arthritis patients. Ann. Rheum. Dis. 79, 1189–1193 (2020).

    Article  PubMed  Google Scholar 

  51. Fiorentino, M. C. et al. A deep-learning framework for metacarpal-head cartilage-thickness estimation in ultrasound rheumatological images. Comput. Biol. Med. 141, 105117 (2022).

    Article  PubMed  Google Scholar 

  52. Folle, L. et al. Deep learning methods allow fully automated segmentation of metacarpal bones to quantify volumetric bone mineral density. Sci. Rep. 11, 9697 (2021).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  53. Folle, L. et al. Deep learning-based classification of inflammatory arthritis by identification of joint shape patterns — how neural networks can tell us where to “Deep Dive” clinically. Front. Med. 9, 850552 (2022).

    Article  Google Scholar 

  54. Wong, L. M., Shi, L., Xiao, F. & Griffith, J. F. Fully automated segmentation of wrist bones on T2-weighted fat-suppressed MR images in early rheumatoid arthritis. Quant. Imaging Med. Surg. 9, 579–589 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  55. Shamonin, D. P. et al. POS0920 quantification of tenosynovitis from wrist MRIs, based on deep learning. Ann. Rheum. Dis. 82, 770–771 (2023).

    Google Scholar 

  56. Li, Y. et al. OP0002 exploring the use of artificial intelligence in predicting rheumatoid arthritis, based on extremity MR scans in early arthritis and clinically suspect arthralgia patients. Ann. Rheum. Dis. 82, 1–2 (2023).

    Google Scholar 

  57. Hassanzadeh, T. et al. A deep learning-based comparative MRI model to detect inflammatory changes in rheumatoid arthritis. Biomed. Signal. Process. Control. 88, 105612 (2024).

    Article  Google Scholar 

  58. Hepburn, C. E. et al. Towards deep learning-assisted quantification of inflammation in spondyloarthritis: intensity-based lesion segmentation. Preprint at https://doi.org/10.48550/arXiv.2106.11343 (2021).

  59. Lin, K. Y. Y., Peng, C., Lee, K. H., Chan, S. C. W. & Chung, H. Y. Deep learning algorithms for magnetic resonance imaging of inflammatory sacroiliitis in axial spondyloarthritis. Rheumatology 61, 4198–4206 (2022).

    Article  PubMed  Google Scholar 

  60. Han, Q. et al. Automatic quantification and grading of hip bone marrow oedema in ankylosing spondylitis based on deep learning. Mod. Rheumatol. 32, 968–973 (2022).

    Article  PubMed  Google Scholar 

  61. Bressem, K. K. et al. Deep learning detects changes indicative of axial spondyloarthritis at MRI of sacroiliac joints. Radiology 305, 655–665 (2022).

    Article  PubMed  Google Scholar 

  62. Lee, K. H., Choi, S. T., Lee, G. Y., Ha, Y. J. & Choi, S. I. Method for diagnosing the bone marrow edema of sacroiliac joint in patients with axial spondyloarthritis using magnetic resonance image analysis based on deep learning. Diagnostics 11, 1156 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  63. Koo, B. S. et al. A pilot study on deep learning-based grading of corners of vertebral bodies for assessment of radiographic progression in patients with ankylosing spondylitis. Ther. Adv. Musculoskelet. Dis. 14, 1759720x221114097 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Bressem, K. K. et al. Deep learning for detection of radiographic sacroiliitis: achieving expert-level performance. Arthritis Res. Ther. 23, 106 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Üreten, K., Maraş, Y., Duran, S. & Gök, K. Deep learning methods in the diagnosis of sacroiliitis from plain pelvic radiographs. Mod. Rheumatol. 33, 202–206 (2023).

    Article  PubMed  Google Scholar 

  66. Grob, A. et al. External validation of the deep learning system “SpineNet” for grading radiological features of degeneration on MRIs of the lumbar spine. Eur. Spine J. 31, 2137–2148 (2022).

    Article  PubMed  Google Scholar 

  67. Smerilli, G. et al. Development of a convolutional neural network for the identification and the measurement of the median nerve on ultrasound images acquired at carpal tunnel level. Arthritis Res. Ther. 24, 38 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  68. Fabry, V. et al. A deep learning tool without muscle-by-muscle grading to differentiate myositis from facio-scapulo-humeral dystrophy using MRI. Diagn. Interv. Imaging 103, 353–359 (2022).

    Article  PubMed  Google Scholar 

  69. Wang, F. et al. Assessment of idiopathic inflammatory myopathy using a deep learning method for muscle T2 mapping segmentation. Eur. Radiol. 33, 2350–2357 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  70. Burlina, P., Billings, S., Joshi, N. & Albayda, J. Automated diagnosis of myositis from muscle ultrasound: exploring the use of machine learning and deep learning methods. PLoS ONE 12, e0184059 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  71. Roncato, C. et al. Colour Doppler ultrasound of temporal arteries for the diagnosis of giant cell arteritis: a multicentre deep learning study. Clin. Exp. Rheumatol. 38, 120–125 (2020).

    PubMed  Google Scholar 

  72. Garaiman, A. et al. Vision transformer assisting rheumatologists in screening for capillaroscopy changes in systemic sclerosis: an artificial intelligence model. Rheumatology 62, 2492–2500 (2022).

    Article  PubMed Central  Google Scholar 

  73. Gurunath Bharathi, P. et al. A deep learning system for quantitative assessment of microvascular abnormalities in nailfold capillary images. Rheumatology 62, 2325–2329 (2023).

    Article  Google Scholar 

  74. Mohajer, B. et al. Role of thigh muscle changes in knee osteoarthritis outcomes: osteoarthritis initiative data. Radiology 305, 169–178 (2022).

    Article  PubMed  Google Scholar 

  75. Adebayo, J. et al. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 31 (NeurIPS, 2018).

  76. Dakkak, Y. J., Jansen, F. P., DeRuiter, M. C., Reijnierse, M. & van der Helm-van Mil, A. H. M. Rheumatoid arthritis and tenosynovitis at the metatarsophalangeal joints: an anatomic and MRI study of the forefoot tendon sheaths. Radiology 295, 146–154 (2020).

    Article  PubMed  Google Scholar 

  77. Maleki, F. et al. Generalizability of machine learning models: quantitative evaluation of three methodological pitfalls. Radiol. Artif. Intell. 5, e220028 (2023).

    Article  PubMed  Google Scholar 

  78. Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 15, e1002683 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  79. Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020).

    Article  Google Scholar 

  80. Reinke, A. et al. Understanding metric-related pitfalls in image analysis validation. Preprint at https://doi.org/10.48550/arXiv.2302.01790 (2023).

  81. Abdalla, M. & Fine, B. Hurdles to artificial intelligence deployment: noise in schemas and “Gold” labels. Radiol. Artif. Intell. 5, e220056 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  82. Meszaros, J., Minari, J. & Huys, I. The future regulation of artificial intelligence systems in healthcare services and medical research in the European Union. Front. Genet. 13, 927721 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  83. Pesapane, F., Volonté, C., Codari, M. & Sardanelli, F. Artificial intelligence as a medical device in radiology: ethical and regulatory issues in Europe and the United States. Insights Imaging 9, 745–753 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  84. Mangnus, L., van Steenbergen, H. W., Reijnierse, M. & van der Helm-van Mil, A. H. Magnetic resonance imaging-detected features of inflammation and erosions in symptom-free persons from the general population. Arthritis Rheumatol. 68, 2593–2602 (2016).

    Article  CAS  PubMed  Google Scholar 

  85. Boer, A. C. et al. Using a reference when defining an abnormal MRI reduces false-positive MRI results-a longitudinal study in two cohorts at risk for rheumatoid arthritis. Rheumatology 56, 1700–1706 (2017).

    Article  PubMed  Google Scholar 

  86. Boeren, A. M. P. et al. Towards a simplified fluid-sensitive MRI protocol in small joints of the hand in early arthritis patients: reliability between modified Dixon and regular Gadolinium enhanced TSE fat saturated MRI-sequences. Skeletal Radiol. 52, 1193–1202 (2023).

    Article  PubMed  Google Scholar 

  87. Hassanzadeh, T. et al. A deep learning model to locate inflammatory changes in rheumatoid arthritis. Ann. Rheum. Dis. 82, 298–299 (2023).

    Google Scholar 

  88. Mongan, J., Moy, L. & Kahn, C. E. Jr Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol. Artif. Intell. 2, e200029 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The work presented in Fig. 5 has been funded by the Dutch Research Council (NWO) Applied and Engineering Sciences (project number 17970).

Author information

Authors and Affiliations

  1. Division of Image Processing, Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

    Berend C. Stoel & Marius Staring

  2. Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

    Monique Reijnierse

  3. Department of Rheumatology, Leiden University Medical Center, Leiden, the Netherlands

    Annette H. M. van der Helm-van Mil

Authors
  1. Berend C. Stoel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marius Staring
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Monique Reijnierse
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Annette H. M. van der Helm-van Mil
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

B.C.S. researched data for the article. All authors contributed substantially to discussion of the content. B.C.S., M.S. and A.H.M.v.d.H.-v.M. wrote the article. All authors reviewed and/or edited the manuscript before submission.

Corresponding author

Correspondence to Berend C. Stoel.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Rheumatology thanks Reza Forghani and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

CS230 Deep Learning tutorial: https://cs230.stanford.edu/

List of commercial AI products for radiology: https://grand-challenge.org/aiforradiology/

Machine Learning Glossary: https://developers.google.com/machine-learning/glossary

Glossary

Cross-entropy

A measure of the difference between two probability distributions, where the one distribution is from the ground truth and the other from the model’s output. Used as a loss function, it penalizes errors especially when the model is confident, but wrong.

Data augmentation

Artificially expanding the number and diversity of training examples by performing random transformations, or adding noise or simulated objects (such as lesions) to existing image data.

Dice coefficient

A statistic that is used to quantify the similarity between two samples; in image segmentation, it measures the overlap between the ground truth and the model-produced segmentation3.

Loss function

A mathematical entity for quantifying how well a machine-learning algorithm models the data. Higher values indicate poorer modelling ability. During training, the loss function is coupled with an optimizer, which is used to tune the parameters of the machine-learning or deep-learning algorithm to minimize the loss function and ultimately maximize algorithm performance3.

Saliency maps

Derived images usually displayed as heat maps that show the locations in the input image that contributed most to the model’s output.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stoel, B.C., Staring, M., Reijnierse, M. et al. Deep learning in rheumatological image interpretation. Nat Rev Rheumatol 20, 182–195 (2024). https://doi.org/10.1038/s41584-023-01074-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41584-023-01074-5

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing