Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Chunkflow: hybrid cloud processing of large 3D images by convolutional nets

Nature Methods volume 18, pages 328–330 (2021)Cite this article

Subjects

To the Editor — Automated microscopes with both high resolution and large field of view are generating terascale and even petascale 3D images. A local cluster might not have enough computational resources to process them in reasonable time, but public cloud platforms can provide computational resources on demand. Convolutional networks have become the state-of-the-art approach for 3D biological image analysis1,2, and cloud processing by 3D convolutional nets has been used for processing independent small image stacks3,4,5. However, cloud computing tools to perform distributed processing of terascale or petascale 3D images by convolutional nets are lacking. Here, we report chunkflow, a framework for distributing computational tasks over both cloud and local computational resources, including both GPUs and CPUs with multiple deep-learning framework back ends, to maximize efficiency, increase flexibility and reduce cost.

Chunkflow has several features (Supplementary Note). Chunkflow is fault tolerant using the visibility timeout mechanism of Simple Queue Service; it can utilize the cheap but unstable cloud instances offered by many cloud vendors; it has a modular and extensible design; it comes with a command line interface for defining an image processing pipeline by composing operators (Supplementary Fig. 2a and Supplementary Table 1); it can be deployed to most cloud platforms using widely supported virtual cluster manager; and it allows real-time visualization while processing (Fig. 1).

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Architecture and processing results from chunkflow.

Data availability

Two datasets that used chunkflow in the image processing pipeline are publicly available at https://microns-explorer.org/ and https://flywire.ai/.

Code availability

Source code and documentation is available as Supplementary Software and online at https://github.com/seung-lab/chunkflow.

References

  1. Caicedo, J. C. et al. Cytometry A 95, 952–965 (2019).

    Article  Google Scholar 

  2. Lee, K. et al. Curr. Opin. Neurol. 55, 188–198 (2019).

    Article  CAS  Google Scholar 

  3. Haberl, M. G. et al. Nat. Methods 15, 677–680 (2018).

    Article  CAS  Google Scholar 

  4. Falk, T. et al. Nat. Methods 16, 67–70 (2019).

    Article  CAS  Google Scholar 

  5. Bannon, D. et al. Nat. Methods 18, 43–45 (2021).

    Article  CAS  Google Scholar 

  6. Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. In Adv. Neural Inf. Process. Syst. 32, 8026–8037 (2019); https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf

  7. Zlateski, A. & Seung, H.S. Compile-time optimized and statically scheduled N-D convnet primitives for multi-core and many-core (Xeon Phi) CPUs. In Proc. Int. Conf. Supercomputing 8, 1–10 (ACM, 2017); https://doi.org/10.1145/3079079.3079081

  8. Popovych, S., Buniatyan, D., Zlateski, A., Li, K. & Seung, H.S. Pznet: efficient 3D convnet inference on manycore CPUs. In Advances in Computer Vision (eds. Popovych, S. et al.) 369–383 (Springer, 2019); https://doi.org/10.1007/978-3-030-17795-9_27

  9. Kasthuri, N. et al. Cell 162, 648–661 (2015).

    Article  CAS  Google Scholar 

  10. Lee, K., Zung, J., Li, P., Jain, V. & Seung, H.S. Preprint at https://arxiv.org/abs/1706.00120 (2017).

Download references

Acknowledgements

We would like to thank T. Macrina for realigning the somatosensory cortex dataset. We would also like to thank W. Wong for discussions and N. Kemnitz for cloud deployment help. We are grateful to Google for providing the technical support and computational resources, including early access to NVIDIA T4 GPUs on the Google Cloud Platform. We are grateful for technical assistance from Google, Amazon and Intel. These companies were not involved in the design of this study. This research was supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DoI/IBC) contract number D16PC0005, NIH/NIMH (U01MH114824, U01MH117072, RF1MH117815), NIH/NINDS (U19NS104648, R01NS104926), NIH/NEI (R01EY027036), ARO (W911NF-12-1-0594) and the Mathers Foundation. The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/IBC or the US Government.

Author information

Author notes

  1. Jingpeng Wu

    Present address: Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA

Authors and Affiliations

  1. Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA

    Jingpeng Wu, William M. Silversmith, Kisuk Lee & H. Sebastian Seung

  2. Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA, USA

    Kisuk Lee

  3. Department of Computer Science, Princeton University, Princeton, NJ, USA

    H. Sebastian Seung

Authors
  1. Jingpeng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. William M. Silversmith
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kisuk Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. H. Sebastian Seung
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.W. developed the chunkflow framework and performed experiments. W.M.S. developed CloudVolume and some other dependent packages. K.L. trained the convolutional net for boundary detection. J.W. and H.S.S. wrote the report with help from W.M.S. and K.L.

Corresponding author

Correspondence to H. Sebastian Seung.

Ethics declarations

Competing interests

H.S.S. declares financial interests in Certerra and Zetta AI.

Additional information

Peer review information Nature Methods thanks Pavel Tomancak and the other, anonymous reviewer(s) for their contribution to the peer review of this work.

Supplementary information

Supplementary Information

Supplementary Note, Supplementary Figs. 1–7 and Supplementary Table 1

Supplementary Software

Code and documentation for chunkflow

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, J., Silversmith, W.M., Lee, K. et al. Chunkflow: hybrid cloud processing of large 3D images by convolutional nets. Nat Methods 18, 328–330 (2021). https://doi.org/10.1038/s41592-021-01088-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-021-01088-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics