Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Distributed-Something: scripts to leverage AWS storage and computing for distributed workflows at scale

Nature Methods volume 20, pages 1120–1121 (2023)Cite this article

Subjects

On-demand computational infrastructure, such as that provided by Amazon Web Services (AWS), is ideal for at-scale parallelizable workflows (especially workflows for which demand is not constant but comes in occasional spikes), as neither computing power nor data storage are limited by local availability and costs are limited to actual resource usage. However, cloud infrastructure configuration is time-consuming and confusing, and cloud-native services that automatically monitor and scale resources can increase the workflow price. Distributed-Something (DS) is a collection of easy-to-use Python scripts that leverage the power of the former while minimizing the problems of the latter.

DS makes it possible for a developer with moderate computational comfort to design a way to deploy a new tool or program to the cloud, and for a user with relatively low computational comfort to then deploy this tool at will. It simplifies the process of using AWS by scripting the majority of the setup, triggering and monitoring of jobs, requiring only minimal human-readable config files to be edited before the run and simple, single-line commands to trigger each step. This increases the ability of novice computational users to be able to execute workflows and lowers the barrier to creating a new workflow for developers. Unlike most existing tools for running hosted containerized analyses, it does not require learning new workflow languages (either for new tool addition or end-user deployment) and minimizes the understanding requirement of the AWS components that are used (see Supplementary Table 1).

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Distributed-Something.

Code availability

Distributed-Something is available at https://github.com/DistributedScience/Distributed-Something and https://doi.org/10.5281/zenodo.7949283. Distributed-CellProfiler is available at https://github.com/DistributedScience/Distributed-CellProfiler and https://doi.org/10.5281/zenodo.7949380. Distributed-Fiji is available at https://github.com/DistributedScience/Distributed-Fiji and https://doi.org/10.5281/zenodo.7949387. Distributed-OMEZarrCreator is available at https://github.com/DistributedScience/Distributed-OMEZarrCreator and https://doi.org/10.5281/zenodo.7949394.

References

  1. McQuin, C. et al. PLoS Biol. 16, e2005970 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Merkel, D. Linux J. 2, 2014 (2014).

    Google Scholar 

  3. Rueden, C. T. et al. BMC Bioinformatics 18, 529 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  4. Schindelin, J. et al. Nat. Methods 9, 676–682 (2012).

    Article  CAS  PubMed  Google Scholar 

  5. Swedlow, J. R., Goldberg, I., Brauner, E. & Sorger, P. K. Science 300, 100–102 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Moore, J. et al. Nat. Methods 18, 1496–1498 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Wilkinson, M. D. et al. Sci. Data 3, 160018 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Williams, E. et al. Nat. Methods 14, 775–781 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. da Veiga Leprevost, F. et al. Bioinformatics 33, 2580–2582 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank Juan Caicedo and Shantanu Singh for creating the original Distributed-CellProfiler, Callum Tromans-Coia, Josh Moore and Sébastien Besson for their assistance with DOZC, and other members of the Cimini and Carpenter-Singh labs for their feedback on this project and manuscript. This study was supported by Calico Life Sciences LLC, NIH P41 GM135019, and grant 2020-225720 from the Chan Zuckerberg Initiative DAF. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA

    Erin Weisbart & Beth A. Cimini

Authors
  1. Erin Weisbart
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beth A. Cimini
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

B.A.C. conceived the project, wrote DS and DF, assisted in writing DCP, and revised the manuscript. E.W. wrote the manuscript, wrote DOZC, and assisted with DS and DF.

Corresponding author

Correspondence to Beth A. Cimini.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Methods thanks Vladimír Ulman and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Supplementary information

Supplementary Information

Supplementary Table 1

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Weisbart, E., Cimini, B.A. Distributed-Something: scripts to leverage AWS storage and computing for distributed workflows at scale. Nat Methods 20, 1120–1121 (2023). https://doi.org/10.1038/s41592-023-01918-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-023-01918-8

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics