Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

BioImage.IO Chatbot: a community-driven AI assistant for integrative computational bioimaging

Nature Methods volume 21, pages 1368–1370 (2024)Cite this article

Subjects

Bioimaging is a rapidly evolving field with vast and complex data, presenting both opportunities and challenges. Researchers use a variety of tools, such as ImageJ1, alongside specialized AI tools2. The embrace of open science has led to the development of community-based resources such as the BioImage Informatics Index, BioImage Archive, image.sc forum and BioImage Model Zoo3, providing a wealth of tools and data but also creating an overwhelming landscape. Managing diverse data formats, complex workflows and AI models often requires coding or scripting, posing significant challenges for biologists without extensive programming experience. Meanwhile, large language model (LLM)-based conversational assistants such as ChatGPT, powered by GPT-4 with tool call and vision capabilities and advanced reasoning frameworks such as ReAct4, are ideal for helping bioimaging users navigate complex ecosystems and solve bioimage analysis tasks5.

The BioImage.IO Chatbot goes beyond information retrieval by incorporating an extension mechanism enabling enhancements through in-browser ImJoy9 extensions or remote Hypha3 services. These extensions allow the chatbot to perform actions like making function calls, generating code, interacting with online databases and services such as the BioImage Informatics Index and the image.sc forum, generating Python code for image analysis and running AI models (Fig. 1a).

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Integration of BioImage.IO Chatbot for enhanced computational bioimaging support.

Data availability

The data used to support the BioImage.IO Chatbot are either publicly accessible from their original sources or available via the chatbot GitHub repository (https://github.com/bioimage-io/bioimageio-chatbot). The prebuilt knowledge bases are created by compiling a list of community data sources, detailed in the manifest file within the same repository.

Code availability

The source code for the BioImage.IO Chatbot is available at https://github.com/bioimage-io/bioimageio-chatbot. In addition to the raw code, the repository contains detailed documentation, usage examples and other supplementary material for the chatbot.

References

  1. Schneider, C. A., Rasband, W. S. & Eliceiri, K. W. Nat. Methods 9, 671–675 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Nat. Methods 18, 100–106 (2021).

    Article  CAS  PubMed  Google Scholar 

  3. Ouyang, W. et al. Preprint at bioRxiv https://doi.org/10.1101/2022.06.07.495102 (2022).

  4. Yao, S. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2210.03629 (2022).

  5. Royer, L. A. Nat. Methods https://doi.org/10.1038/s41592-024-02310-w (2024).

    Article  PubMed  Google Scholar 

  6. Lewis, P. et al. Proc. NIPS '20 793, 16 (2020); https://doi.org/10.5555/3495724.3496517

  7. Hong, S. et al. in Twelfth International Conference on Learning Representations https://openreview.net/forum?id=VtmBAGCN7o (2024).

  8. Gómez-de-Mariscal, E., García-López-de-Haro, C., Munoz-Barrutia, A. & Sage, D. Nat. Methods 18, 1192–1195 (2021).

    Article  PubMed  Google Scholar 

  9. Ouyang, W., Mueller, F., Hjelmare, M., Lundberg, E. & Zimmer, C. Nat. Methods 16, 1199–1200 (2019).

    Article  CAS  PubMed  Google Scholar 

  10. Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. Nature 630, 625–630 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank C. Rueden for his suggestions on accessing ImageJ wiki documentation and image.sc forum, as well as for testing and reporting bugs, providing valuable feedback during the review process. We also thank M. Kalas for his advice on accessing the bio.tools metadata and M. Hartley for guidance on accessing the BioImage Archive API. We appreciate the AI4Life consortium members’ efforts in creating and improving the BioImage Model Zoo and its documentation. Additionally, we are grateful to F. Jug for testing our system and providing valuable feedback on its design. We also thank W. Xu for his support in implementing and testing chatbot extensions. This work was partially supported by the European Union’s Horizon Europe research and innovation program under grant agreement number 101057970 (AI4Life project) awarded to A.M.-B. and W.O., by the SciLifeLab & Wallenberg Data Driven Life Science Program (grant: KAW 2020.0239) and the Göran Gustafsson Prize (grant: 2317) awarded to W.O., and by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, under grant PID2019-109820RB-I00, MCIN/AEI/10.13039/501100011033/, co-financed by the European Regional Development Fund (ERDF), ‘A way of making Europe’, awarded to A.M.-B. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. The authors used the language model ChatGPT developed by OpenAI to assist in structuring and drafting this paper.

Author information

Authors and Affiliations

  1. Department of Intelligent Systems, KTH Royal Institute of Technology, Stockholm, Sweden

    Wanlu Lei

  2. Ericsson Inc., Santa Clara, CA, USA

    Wanlu Lei

  3. Bioengineering Department, Universidad Carlos III de Madrid, Leganes, Spain

    Caterina Fuster-Barceló & Arrate Muñoz-Barrutia

  4. Bioengineering Division, Instituto de Investigación Sanitaria Gregorio Marañón, Madrid, Spain

    Caterina Fuster-Barceló & Arrate Muñoz-Barrutia

  5. Department of Applied Physics, Science for Life Laboratory, KTH Royal Institute of Technology, Stockholm, Sweden

    Gabriel Reder & Wei Ouyang

Authors
  1. Wanlu Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caterina Fuster-Barceló
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gabriel Reder
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arrate Muñoz-Barrutia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Ouyang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

W.O. led conceptualization and design of the project, with supporting code and design contributions from W.L., C.F.-B. and A.M.-B. The development and implementation of the BioImage.IO Chatbot were carried out by W.L., G.R. and W.O. C.F.-B. and A.M.-B. performed testing. C.F.-B. organized the documentation and user interaction design. The manuscript was written by W.O. with input from the other authors. Funding and project administration were managed by A.M.-B. and W.O.

Corresponding author

Correspondence to Wei Ouyang.

Ethics declarations

Competing interests

W.O. is a co-founder of Amun AI AB, a commercial company that builds, delivers, supports and integrates AI-powered data management systems for academic, biotech and pharmaceutical industries. W.L. is an employee of Ericsson Inc.; however, Ericsson Inc. did not influence the study design.

Peer review

Peer review information

Nature Methods thanks Curtis Rueden and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Supplementary information

Reporting Summary

Supplementary Video 1

Supplementary Video 1: BioImage.IO Chatbot provides reliable answers by retrieving information from the community knowledge base and tools. This video highlights the BioImage.IO Chatbot’s ability to interact with the extensive bioimaging community knowledge base and connected online services, providing reliable answers supported by credible sources. It features a user querying the chatbot about AI models from the BioImage Model Zoo, technical details about DeepImageJ, protein and cell images from the Human Protein Atlas, and studies with corresponding cell images in the BioImage Archive. Additionally, it shows how the chatbot retrieves relevant discussions from the image.sc forum and napari documentation about using Cellpose. These interactions illustrate the chatbot’s proficiency in accessing and synthesizing information from specialized sources, enhancing the research and discovery process in bioimaging.

Supplementary Video 2

Supplementary Video 2: BioImage.IO Chatbot assists users with automated bioimage analysis, self-error correction and visual evaluation. This video demonstrates the BioImage.IO Chatbot performing AI-based bioimage analysis on user-provided images, using the Cellpose model via the remote BioEngine server for segmentation. The sequence begins with the user loading a folder from their local file system and querying the chatbot to “segment the image with Cellpose.” The chatbot consults the BioEngine documentation to list available models and retrieve usage information, and then generates Python code to segment cells using Cellpose. When the first attempt results in an input image array shape error, the chatbot detects and corrects it automatically. After successfully segmenting the nuclei image, the user can request a visual inspection. The chatbot calls the visual inspection extension powered by GPT-4 vision, creating a performance evaluation report that concludes the performance is adequate. This video showcases autonomous analysis with automatic visual evaluation, ensuring the quality of results.

Supplementary Video 3

Supplementary Video 3: BioImage.IO Chatbot assists users with performing automated counting, plotting segmentation results, and generating reports. This video illustrates how users can command the BioImage.IO Chatbot to count objects, plot their size distribution and create statistical reports after successful image segmentation. It demonstrates the chatbot’s capability to generate code for counting nuclei, plotting their sizes, exporting results as a CSV file and performing a visual examination of the generated plots. Finally, the chatbot creates reports for the visual evaluation, showcasing its ability to conduct comprehensive counting and plotting, and present findings effectively.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lei, W., Fuster-Barceló, C., Reder, G. et al. BioImage.IO Chatbot: a community-driven AI assistant for integrative computational bioimaging. Nat Methods 21, 1368–1370 (2024). https://doi.org/10.1038/s41592-024-02370-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-024-02370-y

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics