Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The future of bioimage analysis: a dialog between mind and machine

Nature Methods volume 20, pages 951–952 (2023)Cite this article

Subjects

The field of bioimage analysis is poised for a major transformation, owing to advancements in imaging technologies and artificial intelligence. The emergence of multimodal foundation models — which are akin to large language models (such as ChatGPT) but are capable of comprehending and processing biological images — holds great potential for ushering in a revolutionary era in bioimage analysis.

Indeed, recently developed large language models (LLMs) such as ChatGPT12 have shown that incredible feats can be achieved when very large models are trained on all-encompassing corpora. LLMs can solve myriad different tasks for which they have never explicitly been trained: translating text into different languages, writing code, solving puzzles and composing poems. These LLMs are part of a new trend in AI: large foundation models that cost millions of dollars to train on enormous datasets of images, text, sound or combinations of these. These models are usually trained in a self-supervised fashion that removes the need for manual data annotation13. We might ask whether we can we apply these ideas to image analysis, and whether we could construct a ‘large vision model’ (LVM)14,15 that is capable of understanding biological images by training it on a large corpus of images from the natural world and from microscopy. To be truly disruptive, such a model would have to consider more than only images16. It would also need to understand textual commands and answer questions, as ChatGPT does today.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

References

  1. Li, W., Maekiniemi, A., Sato, H., Osman, C. & Singer, R. H. Nat. Methods 19, 1558–1562 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Yeh, L.-H. et al. Preprint at bioRxiv https://doi.org/10.1101/2020.12.15.422951 (2021).

  3. Tang, P. et al. Light Sci. Appl. 10, 237 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Hobson, C. M., Aaron, J. S., Heddleston, J. M. & Chew, T.-L. Front. Cell Dev. Biol. 9, 706126 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Antonacci, G. et al. Biophys. Rev. 12, 615–624 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  6. Fujiwara, M. et al. Carbon 198, 57–62 (2022).

    Article  CAS  Google Scholar 

  7. Slenders, E. et al. Light Sci. Appl. 10, 31 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Popkin, G. Nature 529, 16–18 (2016).

    Article  CAS  PubMed  Google Scholar 

  9. Pachitariu, M. & Stringer, C. Nat. Methods 19, 1634–1641 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Schmidt, U., Weigert, M., Broaddus, C. & Myers, G. Cell detection with star-convex polygons. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (eds. Frangi, A. F. et al. 265–273 (Springer International, 2018).

  11. Ditchburn, R. W. & Drysdale, A. E. Vision Res. 13, 2423–2433 (1973).

    Article  CAS  PubMed  Google Scholar 

  12. Brown, T. B. et al. Preprint at https://doi.org/10.48550/arXiv.2005.14165 (2020).

  13. Kobayashi, H., Cheveralls, K. C., Leonetti, M. D. & Royer, L. A. Nat. Methods 19, 995–1003 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Fang, Y. et al. Preprint at https://doi.org/10.48550/arXiv.2211.07636 (2022).

  15. Yuan, L. et al. Preprint at https://doi.org/10.48550/arXiv.2111.11432 (2021).

  16. Radford, A. et al. Preprint at https://doi.org/10.48550/arXiv.2103.00020 (2021).

  17. Caron, M. et al. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 9630–9640 (IEEE, 2021).

  18. Wu, C. et al. Preprint at https://doi.org/10.48550/arXiv.2303.04671 (2023).

  19. Tay, Y. et al. Preprint at https://doi.org/10.48550/arXiv.2210.11399 (2022).

  20. Xiao, G. et al. Preprint at https://doi.org/10.48550/arXiv.2211.10438 (2022).

  21. Frantar, E. & Alistarh, D. Preprint at https://doi.org/10.48550/arXiv.2301.00774 (2023).

Download references

Acknowledgements

I thank S. Schmid, K. Balla, J. Bragantini, A. Jacobo, M. Lange and S. Mehta for proofreading and feedback.

Author information

Authors and Affiliations

  1. Chan Zuckerberg Biohub, San Francisco, CA, USA

    Loïc A. Royer

Authors
  1. Loïc A. Royer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Loïc A. Royer.

Ethics declarations

Competing interests

The author declares no competing interests.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Royer, L.A. The future of bioimage analysis: a dialog between mind and machine. Nat Methods 20, 951–952 (2023). https://doi.org/10.1038/s41592-023-01930-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-023-01930-y

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing