Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Clinical Studies

Attention-based deep learning for breast lesions classification on contrast enhanced spectral mammography: a multicentre study

British Journal of Cancer volume 128, pages 793–804 (2023)Cite this article

Subjects

Abstract

Background

This study aims to develop an attention-based deep learning model for distinguishing benign from malignant breast lesions on CESM.

Methods

Preoperative CESM images of 1239 patients, which were definitely diagnosed on pathology in a multicentre cohort, were divided into training and validation sets, internal and external test sets. The regions of interest of the breast lesions were outlined manually by a senior radiologist. We adopted three conventional convolutional neural networks (CNNs), namely, DenseNet 121, Xception, and ResNet 50, as the backbone architectures and incorporated the convolutional block attention module (CBAM) into them for classification. The performance of the models was analysed in terms of the receiver operating characteristic (ROC) curve, accuracy, the positive predictive value (PPV), the negative predictive value (NPV), the F1 score, the precision recall curve (PRC), and heat maps. The final models were compared with the diagnostic performance of conventional CNNs, radiomics models, and two radiologists with specialised breast imaging experience.

Results

The best-performing deep learning model, that is, the CBAM-based Xception, achieved an area under the ROC curve (AUC) of 0.970, a sensitivity of 0.848, a specificity of 1.000, and an accuracy of 0.891 on the external test set, which was higher than those of other CNNs, radiomics models, and radiologists. The PRC and the heat maps also indicated the favourable predictive performance of the attention-based CNN model. The diagnostic performance of two radiologists improved with deep learning assistance.

Conclusions

Using an attention-based deep learning model based on CESM images can help to distinguishing benign from malignant breast lesions, and the diagnostic performance of radiologists improved with deep learning assistance.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The study flowchart.
Fig. 2: Receiver operating characteristic curves of different models.
Fig. 3: The confusion matrices for CBAM-based CNN models across the three sets.
Fig. 4: CESM images and heat maps of three breast lesions.
Fig. 5: Receiver operating characteristic curves of CBAM-based Xception model and radiologists’ performance.
Fig. 6: BI-RADS 4 and different lesion diameter subgroups analysis.

Similar content being viewed by others

Data availability

Some or all data, models, or code generated or used during the study are available from the corresponding author by request.

References

  1. DeSantis CE, Ma J, Gaudet MM, Newman LA, Miller KD, Goding Sauer A, et al. Breast cancer statistics, 2019. CA Cancer J Clin. 2019;69:438–51.

    Article  PubMed  Google Scholar 

  2. Pace LE, Keating NL. A systematic assessment of benefits and risks to guide breast cancer screening decisions. JAMA. 2014;311:1327–35.

    Article  CAS  PubMed  Google Scholar 

  3. Puliti D, Duffy SW, Miccinesi G, de Koning H, Lynge E, Zappa M, et al. Overdiagnosis in mammographic screening for breast cancer in Europe: a literature review. J Med Screen. 2012;19:42–56.

    Article  PubMed  Google Scholar 

  4. Lee-Felker SA, Tekchandani L, Thomas M, Gupta E, Andrews-Tang D, Roth A, et al. Newly diagnosed breast cancer: comparison of contrast-enhanced spectral mammography and breast MR imaging in the evaluation of extent of disease. Radiology. 2017;285:389–400.

    Article  PubMed  Google Scholar 

  5. Mao N, Yin P, Li Q, Wang Q, Liu M, Ma H, et al. Radiomics nomogram of contrast-enhanced spectral mammography for prediction of axillary lymph node metastasis in breast cancer: a multicenter study. Eur Radiol. 2020;30:6732–9.

    Article  PubMed  Google Scholar 

  6. Mao N, Yin P, Wang Q, Liu M, Dong J, Zhang X, et al. Added value of radiomics on mammography for breast cancer diagnosis: a feasibility study. J Am Coll Radiol. 2019;16:485–91.

    Article  PubMed  Google Scholar 

  7. Mao N, Wang Q, Liu M, Dong J, Xiao C, Sun N, et al. Computerized image analysis to differentiate benign and malignant breast tumors on magnetic resonance diffusion weighted image: a preliminary study. J Comput Assist Tomogr. 2019;43:93–97.

    Article  PubMed  Google Scholar 

  8. Liu Z, Li Z, Qu J, Zhang R, Zhou X, Li L, et al. Radiomics of multiparametric MRI for pretreatment prediction of pathologic complete response to neoadjuvant chemotherapy in breast cancer: a multicenter study. Clin Cancer Res. 2019;25:3538–47.

    Article  CAS  PubMed  Google Scholar 

  9. Li H, Zhu Y, Burnside ES, Drukker K, Hoadley KA, Fan C, et al. MR imaging radiomics signatures for predicting the risk of breast cancer recurrence as given by research versions of MammaPrint, Oncotype DX, and PAM50 gene assays. Radiology. 2016;281:382–91.

    Article  PubMed  Google Scholar 

  10. Patel BK, Ranjbar S, Wu T, Pockaj BA, Li J, Zhang N, et al. Computer-aided diagnosis of contrast-enhanced spectral mammography: a feasibility study. Eur J Radiol. 2018;98:207–13.

    Article  PubMed  Google Scholar 

  11. Lin F, Wang Z, Zhang K, Yang P, Ma H, Shi Y, et al. Contrast-enhanced spectral mammography-based radiomics nomogram for identifying benign and malignant breast lesions of sub-1 cm. Front Oncol. 2020;10:573630.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Truhn D, Schrading S, Haarburger C, Schneider H, Merhof D, Kuhl C. Radiomic versus convolutional neural networks analysis for classification of contrast-enhancing lesions at multiparametric breast MRI. Radiology. 2019;290:290–7.

    Article  PubMed  Google Scholar 

  13. Zheng X, Yao Z, Huang Y, Yu Y, Wang Y, Liu Y, et al. Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer. Nat Commun. 2020;11:1236.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Jiang M, Li CL, Luo XM, Chuan ZR, Lv WZ, Li X, et al. Ultrasound-based deep learning radiomics in the assessment of pathological complete response to neoadjuvant chemotherapy in locally advanced breast cancer. Eur J Cancer. 2021;147:95–105.

    Article  CAS  PubMed  Google Scholar 

  15. Perek S, Kiryati N, Zimmerman-Moreno G, Sklair-Levy M, Konen E, Mayer A. Classification of contrast-enhanced spectral mammography (CESM) images. Int J Comput Assist Radiol Surg. 2019;14:249–57.

    Article  PubMed  Google Scholar 

  16. Song J, Zheng Y, Zakir Ullah M, Wang J, Jiang Y, Xu C, et al. Multiview multimodal network for breast cancer diagnosis in contrast-enhanced spectral mammography images. Int J Comput Assist Radiol Surg. 2021;16:979–88.

    Article  PubMed  Google Scholar 

  17. Wang H, Wang S, Qin Z, Zhang Y, Li R, Xia Y. Triple attention learning for classification of 14 thoracic diseases using chest radiography. Med Image Anal. 2021;67:101846.

    Article  PubMed  Google Scholar 

  18. Zhang R, Duan H, Cheng J, Zheng Y. A study on tuberculosis classification in chest X-ray using deep residual attention networks. Annu Int Conf IEEE Eng Med Biol Soc. 2020;2020:1552–5.

    PubMed  Google Scholar 

  19. Woo S, Park J, Lee J-Y, Kweon IS. CBAM: convolutional block attention module. arXiv:1807.06521v2 [Preprint]. 2018 [cited 2018 Jul 18]: [17 p]. Available from: https://arxiv.org/abs/1807.06521

  20. Chen H, Yang BW, Qian L, Meng YS, Bai XH, Hong XW, et al. Deep learning prediction of ovarian malignancy at US compared with O-RADS and expert assessment. Radiology 2022;304:106–13.

    Article  PubMed  Google Scholar 

  21. Huang G, Liu Z, Maaten Lvd, Weinberger KQ. Densely connected convolutional networks. arXiv:1608.06993v5 [Preprint]. 2018 [cited 2018 Jan 28]: [9 p]. Available from: https://arxiv.org/abs/1608.06993

  22. Chollet F. Xception: deep learning with depthwise separable convolutions. arXiv:1610.02357v3 [Preprint]. 2018 [cited 2018 Apr 4]: [8 p]. Available from: https://arxiv.org/abs/1610.02357

  23. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. arXiv:1512.03385v1 [Preprint]. 2015 [cited 2015 Dec 10]: [12 p]. Available from: https://arxiv.org/abs/1512.03385

  24. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al. ImageNet large scale visual recognition challenge. Int J Comput Vis. 2015;115:211–52.

  25. Jiang Y, Zhang Z, Yuan Q, Wang W, Wang H, Li T, et al. Predicting peritoneal recurrence and disease-free survival from CT images in gastric cancer with multitask deep learning: a retrospective study. Lancet Digital Health. 2022;4:e340–50.

    Article  CAS  PubMed  Google Scholar 

  26. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int J Comput Vis. 2020;128:336–59.

    Article  Google Scholar 

  27. Lee CH, Phillips J, Sung JS, Lewin JM, Newell MS. Contrast enhanced mammography (CEM) (A supplement to ACR BI-RADS® Mammography 2013). American College of Radiology. 2022. https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS_CEM_2022.pdf

  28. Whitney HM, Taylor NS, Drukker K, Edwards AV, Papaioannou J, Schacht D, et al. Additive benefit of radiomics over size alone in the distinction between benign lesions and luminal a cancers on a large clinical breast MRI dataset. Acad Radiol. 2019;26:202–9.

    Article  PubMed  Google Scholar 

  29. Parekh VS, Jacobs MA. Integrated radiomic framework for breast cancer and tumor biology using advanced machine learning and multiparametric MRI. NPJ Breast Cancer. 2017;3:43.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Bickelhaupt S, Jaeger PF, Laun FB, Lederer W, Daniel H, Kuder TA, et al. Radiomics based on adapted diffusion kurtosis imaging helps to clarify most mammographic findings suspicious for cancer. Radiology. 2018;287:761–70.

    Article  PubMed  Google Scholar 

  31. Zhang Q, Xiao Y, Suo J, Shi J, Yu J, Guo Y, et al. Sonoelastomics for breast tumor classification: a radiomics approach with clustering-based feature selection on sonoelastography. Ultrasound Med Biol. 2017;43:1058–69.

    Article  PubMed  Google Scholar 

  32. Qiao M, Li C, Suo S, Cheng F, Hua J, Xue D, et al. Breast DCE-MRI radiomics: a robust computer-aided system based on reproducible BI-RADS features across the influence of datasets bias and segmentation methods. Int J Comput Assist Radiol Surg. 2020;15:921–30.

    Article  PubMed  Google Scholar 

  33. Zhao S, Zhang X, Zhong H, Qin Y, Li Y, Song B, et al. Background parenchymal enhancement on contrast-enhanced spectral mammography: influence of age, breast density, menstruation status, and menstrual cycle timing. Sci Rep. 2020;10:8608.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Lei C, Wei W, Liu Z, Xiong Q, Yang C, Yang M, et al. Mammography-based radiomic analysis for predicting benign BI-RADS category 4 calcifications. Eur J Radiol. 2019;121:108711.

    Article  PubMed  Google Scholar 

  35. Wang S, Sun Y, Li R, Mao N, Li Q, Jiang T, et al. Diagnostic performance of perilesional radiomics analysis of contrast-enhanced mammography for the differentiation of benign and malignant breast lesions. Eur Radiol. 2021;32:639–49.

  36. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts H. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18:500–10.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Becker AS, Marcon M, Ghafoor S, Wurnig MC, Frauenfelder T, Boss A. Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer. Invest Radiol. 2017;52:434–40.

    Article  PubMed  Google Scholar 

  38. Yala A, Lehman C, Schuster T, Portnoi T, Barzilay R. A deep learning mammography-based model for improved breast cancer risk prediction. Radiology. 2019;292:60–6.

    Article  PubMed  Google Scholar 

  39. Zhang Q, Peng Y, Liu W, Bai J, Zheng J, Yang X, et al. Radiomics based on multimodal MRI for the differential diagnosis of benign and malignant breast lesions. J Magn Reson Imaging. 2020;52:596–607.

    Article  PubMed  Google Scholar 

  40. Guo X, Liu Z, Sun C, Zhang L, Wang Y, Li Z, et al. Deep learning radiomics of ultrasonography: Identifying the risk of axillary non-sentinel lymph node involvement in primary breast cancer. EBioMedicine. 2020;60:103018.

    Article  PubMed  PubMed Central  Google Scholar 

  41. McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, et al. International evaluation of an AI system for breast cancer screening. Nature. 2020;577:89–94.

    Article  CAS  PubMed  Google Scholar 

  42. Dalmis MU, Gubern-Merida A, Vreemann S, Bult P, Karssemeijer N, Mann R, et al. Artificial intelligence-based classification of breast lesions imaged with a multiparametric breast MRI protocol with Ultrafast DCE-MRI, T2, and DWI. Invest Radiol. 2019;54:325–32.

    Article  PubMed  Google Scholar 

  43. Qian X, Pei J, Zheng H, Xie X, Yan L, Zhang H, et al. Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning. Nat Biomed Eng. 2021;5:522–32.

    Article  PubMed  Google Scholar 

  44. Xi IL, Zhao Y, Wang R, Chang M, Purkayastha S, Chang K, et al. Deep learning to distinguish benign from malignant renal lesions based on routine MR imaging. Clin Cancer Res. 2020;26:1944–52.

    Article  PubMed  Google Scholar 

Download references

Funding

The study was supported by the National Natural Science Foundation of China (82001775 and 62176140), the Natural Science Foundation of Shandong Province of China (ZR2021MH120 and ZR2022MH274), the Special Fund for Breast Disease Research of Shandong Medical Association (YXH2021ZX055), and Taishan Scholars (tsqn202103197).

Author information

Author notes

  1. These authors jointly supervised this work: Ning Mao, Haicheng Zhang.

Authors and Affiliations

  1. Department of Radiology, Yantai Yuhuangding Hospital, Qingdao University, 264000, Yantai, Shandong, P. R. China

    Ning Mao, Fan Lin, Jing Gao, Tiantian Zheng, Haizhu Xie & Heng Ma

  2. Big Data and Artificial Intelligence Laboratory, Yantai Yuhuangding Hospital, Qingdao University, 264000, Yantai, Shandong, P. R. China

    Ning Mao & Haicheng Zhang

  3. Department of Radiology, Peking University Shenzhen Hospital, 518000, Shenzhen, P. R. China

    Yi Dai

  4. Department of Radiology, Fudan University Cancer Center, 200433, Shanghai, P. R. China

    Qin Li

  5. School of Computer Science and Technology, Shandong Technology and Business University, 264005, Yantai, Shandong, P. R. China

    Feng Zhao

  6. Physical Examination Center, Yantai Yuhuangding Hospital, Qingdao University, 264000, Yantai, Shandong, P. R. China

    Cong Xu

Authors
  1. Ning Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haicheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fan Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jing Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tiantian Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Feng Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Haizhu Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Cong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Heng Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

(1) Guarantor of integrity of the entire study: HM and CX. (2) Study concepts and design: all authors. (3) Literature research: all authors. (4) Clinical studies: all authors. (5) Experimental studies/data analysis: all authors. (6) Statistical analysis: HZ. (7) Manuscript preparation: all authors. (8) Manuscript editing: all authors.

Corresponding authors

Correspondence to Cong Xu or Heng Ma.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

This retrospective multicentre study was approved by the institutional review board of Yantai Yuhuangding Hospital, and the patient informed consent was waived.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mao, N., Zhang, H., Dai, Y. et al. Attention-based deep learning for breast lesions classification on contrast enhanced spectral mammography: a multicentre study. Br J Cancer 128, 793–804 (2023). https://doi.org/10.1038/s41416-022-02092-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41416-022-02092-y

Search

Advanced search

Quick links