Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Self-supervised learning in medicine and healthcare

Nature Biomedical Engineering volume 6, pages 1346–1352 (2022)Cite this article

Subjects

Abstract

The development of medical applications of machine learning has required manual annotation of data, often by medical experts. Yet, the availability of large-scale unannotated data provides opportunities for the development of better machine-learning models. In this Review, we highlight self-supervised methods and models for use in medicine and healthcare, and discuss the advantages and limitations of their application to tasks involving electronic health records and datasets of medical images, bioelectrical signals, and sequences and structures of genes and proteins. We also discuss promising applications of self-supervised learning for the development of models leveraging multimodal datasets, and the challenges in collecting unbiased data for their training. Self-supervised learning may accelerate the development of medical artificial intelligence.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Contrastive learning.
Fig. 2: Generative pre-training.

Similar content being viewed by others

References

  1. Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. AI in health and medicine. Nat. Med. 28, 31–38 (2022).

    Article  CAS  Google Scholar 

  2. Sambasivan, N. et al. “Everyone wants to do the model work, not the data work”: data cascades in high-stakes AI. In Proc. 2021 CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery, 2021); https://doi.org/10.1145/3411764.3445518

  3. Russakovsky, O. et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).

    Article  Google Scholar 

  4. Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019 590–597 (AAAI Press, 2019).

  5. Huh, M., Agrawal, P. & Efros, A. What makes ImageNet good for transfer learning? Preprint at https://doi.org/10.48550/arXiv.1608.08614 (2016).

  6. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. 37th International Conference on Machine Learning (eds. Daumé, H. III & Singh, A.) 1597–1607 (PMLR, 2020).

  7. Chen, X., Fan, H., Girshick, R. & He, K. Improved baselines with momentum contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2003.04297 (2020).

  8. Zbontar, J., Jing, L., Misra, I., LeCun, Y. & Deny, S. Barlow Twins: self-supervised learning via redundancy reduction. In Proc. 38th International Conference on Machine Learning (eds. Meila, M. & Zhang, T.) 12310–12320 (PMLR, 2021).

  9. Sowrirajan, H., Yang, J., Ng, A. Y. & Rajpurkar, P. MoCo-CXR: MoCo pretraining improves representation and transferability of chest X-ray models. In Medical Imaging with Deep Learning 2021 727–743 (PMLR, 2021).

  10. Soni, P. N., Shi, S., Sriram, P. R., Ng, A. Y. & Rajpurkar, P. Contrastive learning of heart and lung sounds for label-efficient diagnosis. Patterns 3, 100400 (2022).

    Article  Google Scholar 

  11. Zhang, Y., Jiang, H., Miura, Y., Manning, C. D. & Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. Preprint at https://doi.org/10.48550/arXiv.2010.00747 (2020).

  12. Sriram, A. et al. COVID-19 prognosis via self-supervised representation learning and multi-image prediction. Preprint at https://doi.org/10.48550/arXiv.2101.04909 (2021).

  13. Han, Y., Chen, C., Tewfik, A. H., Ding, Y. & Peng, Y. Pneumonia detection on chest X-ray using radiomic features and contrastive learning. In 2021 IEEE 18th International Symposium on Biomedical Imaging ISBI 247–251 (IEEE Computer Society, 2021).

  14. Azizi, S. et al. Big self-supervised models advance medical image classification. In 2021 IEEECVF International Conference on Computer Vision ICCV 3458–3468 (IEEE Computer Society, 2021).

  15. Vu, Y. N. T. et al. MedAug: contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation. In Proc. 6th Machine Learning for Healthcare Conference (eds Jung, K. et al.) 755–769 (PMLR, 2021).

  16. Lu, M. Y., Chen, R. J. & Mahmood, F. Semi-supervised breast cancer histology classification using deep multiple instance learning and contrast predictive coding. In Medical Imaging 2020: Digital Pathology (eds. Tomaszewski, J. E. & Ward, A. D.) 11320J (SPIE, 2020).

  17. Yang, P., Hong, Z., Yin, X., Zhu, C. & Jiang, R. Self-supervised visual representation learning for histopathological images. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (eds. de Bruijne, M. et al.) 47–57 (Springer International Publishing, 2021).

  18. Srinidhi, C. L., Kim, S. W., Chen, F.-D. & Martel, A. L. Self-supervised driven consistency training for annotation efficient histopathology image analysis. Med. Image Anal. 75, 102256 (2022).

    Article  Google Scholar 

  19. DiPalma, J., Suriawinata, A. A., Tafe, L. J., Torresani, L. & Hassanpour, S. Resolution-based distillation for efficient histology image classification. Artif. Intell. Med. 119, 102136 (2021).

    Article  Google Scholar 

  20. Kiyasseh, D., Zhu, T. & Clifton, D. A. CLOCS: Contrastive Learning of Cardiac Signals across space, time, and patients. In Proc. 38th International Conference on Machine Learning (eds. Meila, M. & Zhang, T.) 5606–5615 (PMLR, 2021).

  21. Banville, H. J. et al. Self-supervised representation learning from electroencephalography signals. In 2019 IEEE 29th International Workshop on Machine Learning Signal Process MLSP (IEEE Computer Society, 2019); https://doi.org/10.1109/MLSP.2019.8918693

  22. Gopal, B. et al. 3KG: contrastive learning of 12-lead electrocardiograms using physiologically-inspired augmentations. in Proc. Machine Learning for Health (eds. Roy, S. et al.) 156–167 (PMLR, 2021).

  23. Jiao, J. et al. Self-supervised contrastive video-speech representation learning for ultrasound. Med. Image Comput. Comput. Assist. Interv. 12263, 534–543 (2020).

    Google Scholar 

  24. Wang, Y., Wang, J., Cao, Z. & Barati Farimani, A. Molecular contrastive learning of representations via graph neural networks. Nat. Mach. Intell. 4, 279–287 (2022).

    Article  Google Scholar 

  25. Xie, Y., Xu, Z., Zhang, J., Wang, Z. & Ji, S. Self-supervised learning of graph neural networks: a unified review. IEEE Trans. Pattern Anal. Mach. Intell. (2022); https://doi.org/10.1109/TPAMI.2022.3170559

  26. Meng, X., Ganoe, C. H., Sieberg, R. T., Cheung, Y. Y. & Hassanpour, S. Self-supervised contextual language representation of radiology reports to improve the identification of communication urgency. AMIA Jt. Summits Transl. Sci. Proc. 2020, 413–421 (2020).

    Google Scholar 

  27. Girgis, H. Z., James, B. T. & Luczak, B. B. Identity: rapid alignment-free prediction of sequence alignment identity scores using self-supervised general linear models. NAR Genom. Bioinform. 3, lqab001 (2021).

  28. Li, Y. et al. BEHRT: transformer for electronic health records. Sci. Rep. 10, 7155 (2020).

    Article  CAS  Google Scholar 

  29. Wang, X., Xu, Z., Tam, L., Yang, D. & Xu, D. Self-supervised image-text pre-training with mixed data in chest X-rays. Preprint at https://doi.org/10.48550/arXiv.2103.16022 (2021).

  30. Rasmy, L., Xiang, Y., Xie, Z., Tao, C. & Zhi, D. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digit. Med. 4, 86 (2021).

    Article  Google Scholar 

  31. Li, F. et al. Fine-tuning Bidirectional Encoder Representations From Transformers (BERT)-based models on large-scale electronic health record notes: an empirical study. JMIR Med. Inform. 7, e14830 (2019).

    Article  Google Scholar 

  32. Kraljevic, Z. et al. Multi-domain clinical natural language processing with MedCAT: the Medical Concept Annotation Toolkit. Artif. Intell. Med. 117, 102083 (2021).

    Article  Google Scholar 

  33. Kostas, D., Aroca-Ouellette, S. & Rudzicz, F. BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data. Front. Hum. Neurosci. 15, 653659 (2021).

    Article  Google Scholar 

  34. Baevski, A., Zhou, Y., Mohamed, A. & Auli, M. wav2vec 2.0: a framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (eds Larochelle, H. et al.) 12449–12460 (Curran Associates, 2020).

  35. Boyd, J. et al. Self-supervised representation learning using visual field expansion on digital pathology. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) 639–647 (IEEE Computer Society, 2021).

  36. Vaswani, A. et al. Attention is all you need. In Proc. 31st International Conference on Neural Information Processing Systems 6000–6010 (Curran Associates, 2017).

  37. Jaegle, A. et al. Perceiver IO: a general architecture for structured inputs and outputs. In International Conference on Learning Representations 4039 (ICLR, 2022).

  38. Akbari, H. et al. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. In Advances in Neural Information Processing Systems (eds. Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. S. & Vaughan, J. W.) 24206–24221 (Curran Associates, 2021).

  39. Nagrani, A. et al. Attention bottlenecks for multimodal fusion. In Advances in Neural Information Processing Systems (eds Ranzato, M. et al.) 14200–14213 (Curran Associates, 2021).

  40. Choromanski, K. et al. Masked language modeling for proteins via linearly scalable long-context transformers. Preprint at https://doi.org/10.48550/arXiv.2006.03555 (2020).

  41. Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021).

    Article  CAS  Google Scholar 

  42. Rao, R. M. et al. MSA Transformer. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 8844–8856 (PMLR, 2021).

  43. Lu, A. X., Zhang, H., Ghassemi, M. & Moses, A. Self-supervised contrastive learning of protein representations by mutual information maximization. Preprint at bioRxiv https://doi.org/10.1101/2020.09.04.283929 (2020).

  44. Yang, C., Wu, Z., Zhou, B. & Lin, S. Instance localization for self-supervised detection pretraining. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 3986–3995 (IEEE Computer Society, 2021).

  45. Jana, A. et al. Deep learning based NAS score and fibrosis stage prediction from CT and pathology data. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering BIBE 981–986 (IEEE Computer Society, 2020).

  46. Ohri, K. & Kumar, M. Review on self-supervised image recognition using deep neural networks. Knowl. Based Syst. 224, 107090 (2021).

    Article  Google Scholar 

  47. Holmberg, O. G. et al. Self-supervised retinal thickness prediction enables deep learning from unlabelled data to boost classification of diabetic retinopathy. Nat. Mach. Intell. 2, 719–726 (2020).

    Article  Google Scholar 

  48. Spahr, A., Bozorgtabar, B. & Thiran, J.-P. Self-taught semi-supervised anomaly detection on upper limb X-rays. In 2021 IEEE 18th International Symposium on Biomedical Imaging ISBI 1632–1636 (IEEE Computer Society, 2021).

  49. Radford, A. et al. Learning transferable visual models from natural language supervision. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 8748–8763 (PMLR, 2021).

  50. Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020).

    Article  Google Scholar 

  51. Sagawa, S., Koh, P. W., Hashimoto, T. B. & Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations 1796 (ICLR, 2020).

  52. Fedorov, A. et al. Tasting the cake: evaluating self-supervised generalization on out-of-distribution multimodal MRI data. Preprint at https://doi.org/10.48550/arXiv.2103.15914 (2021).

  53. Li, Z. et al. Domain generalization for mammography detection via multi-style and multi-view contrastive learning. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (eds de Bruijne, M. et al.) 98–108 (Springer, 2021).

  54. Endo, M., Krishnan, R., Krishna, V., Ng, A. Y. & Rajpurkar, P. Retrieval-based chest X-ray report generation using a pre-trained contrastive language-image model. in Proc. Machine Learning for Health (eds. Roy, S. et al.) 209–219 (PMLR, 2021).

  55. Sriram, A. et al. COVID-19 prognosis via self-supervised representation learning and multi-image prediction. Preprint at https://doi.org/10.48550/arXiv.2101.04909 (2021).

  56. Chen, R. J. & Krishnan, R. G. Self-supervised vision transformers learn visual concepts in histopathology. in LMLR at Neural Information Processing Systems (NeurIPS, 2021).

  57. Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (eds Larochelle, H. et al.) 1877–1901 (Curran Associates, 2020).

  58. Logé, C. et al. Q-Pain: a question answering dataset to measure social bias in pain management. In Proc. Neural Information Processing Systems Track on Datasets and Benchmarks (eds. Vanschoren, J. & Yeung, S.) 105 (NeurIPS, 2021).

  59. Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H. & Ferrante, E. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl Acad. Sci. USA 117, 12592 (2020).

    Article  CAS  Google Scholar 

  60. Gamble, P. et al. Determining breast cancer biomarker status and associated morphological features using deep learning. Commun. Med. 1, 14 (2021).

    Article  Google Scholar 

Download references

Acknowledgements

We thank A. Saporta and A. Tamkin for their helpful suggestions. We acknowledge support from the NIH grant UL1TR002550 to E.J.T.

Author information

Author notes

  1. These authors contributed equally: Pranav Rajpurkar, Eric J. Topol.

Authors and Affiliations

  1. Department of Computer Science, Stanford University, Stanford, CA, USA

    Rayan Krishnan

  2. Department of Biomedical Informatics, Harvard University, Boston, MA, USA

    Pranav Rajpurkar

  3. Scripps Research Translational Institute, Scripps Research, La Jolla, CA, USA

    Eric J. Topol

Authors
  1. Rayan Krishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pranav Rajpurkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric J. Topol
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to researching the literature and to the writing and editing of the manuscript.

Corresponding author

Correspondence to Pranav Rajpurkar.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Biomedical Engineering thanks Su-In Lee, Faisal Mahmood and Collin Stultz for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Krishnan, R., Rajpurkar, P. & Topol, E.J. Self-supervised learning in medicine and healthcare. Nat. Biomed. Eng 6, 1346–1352 (2022). https://doi.org/10.1038/s41551-022-00914-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-022-00914-1

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing