Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

Benchmarking single-cell RNA-sequencing protocols for cell atlas projects

Nature Biotechnology volume 38, pages 747–755 (2020)Cite this article

Subjects

Abstract

Single-cell RNA sequencing (scRNA-seq) is the leading technique for characterizing the transcriptomes of individual cells in a sample. The latest protocols are scalable to thousands of cells and are being used to compile cell atlases of tissues, organs and organisms. However, the protocols differ substantially with respect to their RNA capture efficiency, bias, scale and costs, and their relative advantages for different applications are unclear. In the present study, we generated benchmark datasets to systematically evaluate protocols in terms of their power to comprehensively describe cell types and states. We performed a multicenter study comparing 13 commonly used scRNA-seq and single-nucleus RNA-seq protocols applied to a heterogeneous reference sample resource. Comparative analysis revealed marked differences in protocol performance. The protocols differed in library complexity and their ability to detect cell-type markers, impacting their predictive value and suitability for integration into reference cell atlases. These results provide guidance both for individual researchers and for consortium projects such as the Human Cell Atlas.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of the experimental design and data processing.
Fig. 2: Comparison of 13 sc/snRNA-seq methods.
Fig. 3: Similarity measures of sc/snRNA-seq methods.
Fig. 4: Clustering analysis of 13 sc/snRNA-seq methods on down-sampled datasets (20,000).
Fig. 5: Integration of sc/snRNA-seq methods.
Fig. 6: Benchmarking summary of 13 sc/snRNA-seq methods.

Similar content being viewed by others

Data availability

All raw sequencing data and processed gene expression files are freely available through the Gene Expression Omnibus (accession no. GSE133549).

Code availability

All code for the analysis is provided as supplementary material. All code is also available under https://github.com/ati-lz/HCA_Benchmarking and https://github.com/elimereu/matchSCore2.

References

  1. Lafzi, A., Moutinho, C., Picelli, S. & Heyn, H. Tutorial: guidelines for the experimental design of single-cell RNA sequencing studies. Nat. Protoc. 13, 2742–2757 (2018).

    Article  CAS  PubMed  Google Scholar 

  2. Prakadan, S. M., Shalek, A. K. & Weitz, D. A. Scaling by shrinking: empowering single-cell ‘omics’ with microfluidic devices. Nat. Rev. Genet. 18, 345–361 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Svensson, V., Vento-Tormo, R. & Teichmann, S. A. Exponential scaling of single-cell RNA-seq in the past decade. Nat. Protoc. 13, 599–604 (2018).

    Article  CAS  PubMed  Google Scholar 

  4. Montoro, D. T. et al. A revised airway epithelial hierarchy includes CFTR-expressing ionocytes. Nature 560, 319–324 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Plasschaert, L. W. et al. A single-cell atlas of the airway epithelium reveals the CFTR-rich pulmonary ionocyte. Nature 560, 377–381 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Aizarani, N. et al. A human liver cell atlas reveals heterogeneity and epithelial progenitors. Nature 572, 199–204 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Karaiskos, N. et al. The Drosophila embryo at single-cell transcriptome resolution. Science 358, 194–199 (2017).

    Article  CAS  PubMed  Google Scholar 

  8. Wagner, D. E. et al. Single-cell mapping of gene expression landscapes and lineage in the zebrafish embryo. Science 360, 981–987 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Regev, A. et al. Science forum: the human cell atlas. eLife 6, e27041 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  10. Cao, J. et al. Comprehensive single-cell transcriptional profiling of a multicellular organism. Science 357, 661–667 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Plass, M. et al. Cell type atlas and lineage tree of a whole complex animal by single-cell transcriptomics. Science 360, eaaq1723 (2018).

  12. Moffitt, J. R. et al. High-throughput single-cell gene-expression profiling with multiplexed error-robust fluorescence in situ hybridization. Proc. Natl Acad. Sci. USA 113, 11046–11051 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Lubeck, E., Coskun, A. F., Zhiyentayev, T., Ahmad, M. & Cai, L. Single-cell in situ RNA profiling by sequential hybridization. Nat. Methods 11, 360–361 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Alioto, T. S. et al. A comprehensive assessment of somatic mutation detection in cancer using whole-genome sequencing. Nat. Commun. 6, 10001 (2015).

    Article  CAS  PubMed  Google Scholar 

  15. Ziegenhain, C. et al. Comparative analysis of single-cell RNA sequencing methods. Mol. Cell 65, 631–643.e4 (2017).

    Article  CAS  PubMed  Google Scholar 

  16. Svensson, V. et al. Power analysis of single-cell RNA-sequencing experiments. Nat. Methods 14, 381–387 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Tung, P.-Y. et al. Batch effects and the effective design of single-cell gene expression studies. Sci. Rep. 7, 39921 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Zheng, G. X. Y. et al. Massively parallel digital transcriptional profiling of single cells. Nat. Commun. 8, 14049 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Haber, A. L. et al. A single-cell survey of the small intestinal epithelium. Nature 551, 333–339 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Grün, D. et al. Single-cell messenger RNA sequencing reveals rare intestinal cell types. Nature 525, 251–255 (2015).

    Article  PubMed  CAS  Google Scholar 

  21. Guillaumet-Adkins, A. et al. Single-cell transcriptome conservation in cryopreserved cells and tissues. Genome Biol. 18, 45 (2017).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  22. Schaum, N. et al. Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris. Nature 562, 367–372 (2018).

    Article  PubMed Central  CAS  Google Scholar 

  23. Büttner, M., Miao, Z., Wolf, F. A., Teichmann, S. A. & Theis, F. J. A test metric for assessing single-cell RNA-seq batch correction. Nat. Methods 16, 43–49 (2019).

    Article  PubMed  CAS  Google Scholar 

  24. Azuaje, F. A cluster validity framework for genome expression data. Bioinforma 18, 319–320 (2002).

    Article  CAS  Google Scholar 

  25. Lin, Y. et al. scMerge leverages factor analysis, stable expression, and pseudoreplication to merge multiple single-cell RNA-seq datasets. Proc. Natl Acad. Sci. USA 116, 9775–9784 (2019).

  26. Kang, H. M. et al. Multiplexed droplet single-cell RNA-sequencing using natural genetic variation. Nat. Biotechnol. 36, 89–94 (2018).

    Article  CAS  PubMed  Google Scholar 

  27. Stoeckius, M. et al. Cell hashing with barcoded antibodies enables multiplexing and doublet detection for single cell genomics. Genome Biol. 19, 224 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. McGinnis, C. S. et al. MULTI-seq: sample multiplexing for single-cell RNA sequencing using lipid-tagged indices. Nat. Methods 16, 619–626 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Gaublomme, J. T. et al. Nuclei multiplexing with barcoded antibodies for single-nucleus genomics. Nat. Commun. 10, 1–8 (2019).

    Article  CAS  Google Scholar 

  30. Mora-Castilla, S. et al. Miniaturization technologies for efficient single-cell library preparation for next-generation sequencing. J. Lab. Autom. 21, 557–567 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Picelli, S. et al. Tn5 transposase and tagmentation procedures for massively scaled sequencing projects. Genome Res. 24, 2033–2040 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Brink, S. Cvanden et al. Single-cell sequencing reveals dissociation-induced gene expression in tissue subpopulations. Nat. Methods 14, 935–936 (2017).

    Article  CAS  PubMed  Google Scholar 

  33. Wohnhaas, C. T. et al. DMSO cryopreservation is the method of choice to preserve cells for droplet-based single-cell RNA sequencing. Sci. Rep. 9, 1–14 (2019).

    Article  CAS  Google Scholar 

  34. Tosti, L. et al. Single nucleus RNA sequencing maps acinar cell states in a human pancreas cell atlas. Preprint at bioRxiv https://doi.org/10.1101/733964 (2019).

  35. Massoni-Badosa, R. et al. Sampling artifacts in single-cell genomics cohort studies. Preprint at bioRxiv https://doi.org/10.1101/2020.01.15.897066 (2020).

  36. Picelli, S. et al. Smart-seq2 for sensitive full-length transcriptome profiling in single cells. Nat. Methods 10, 1096–1098 (2013).

    Article  CAS  PubMed  Google Scholar 

  37. Sasagawa, Y. et al. Quartz-Seq2: a high-throughput single-cell RNA-sequencing method that effectively uses limited sequence reads. Genome Biol. 19, 29 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  38. Hashimshony, T. et al. CEL-Seq2: sensitive highly-multiplexed single-cell RNA-Seq. Genome Biol. 17, 77 (2016).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  39. Bagnoli, J. W. et al. Sensitive and powerful single-cell RNA sequencing using mcSCRB-seq. Nat. Commun. 9, 2937 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  40. Sasagawa, Y. et al. Quartz-Seq: a highly reproducible and sensitive single-cell RNA sequencing method, reveals non-genetic gene-expression heterogeneity. Genome Biol. 14, 3097 (2013).

    Article  CAS  Google Scholar 

  41. Parekh, S., Ziegenhain, C., Vieth, B., Enard, W. & Hellmann, I. The impact of amplification on differential expression analyses by RNA-seq. Sci. Rep. 6, 25533 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Soneson, C. & Robinson, M. D. Bias, robustness and scalability in single-cell differential expression analysis. Nat. Methods 15, 255–261 (2018).

    Article  CAS  PubMed  Google Scholar 

  43. Saelens, W. et al. A comparison of single-cell trajectory inference methods. Nat. Biotechnol. 37, 547–554 (2019).

    Article  CAS  PubMed  Google Scholar 

  44. Holland, C. H. et al. Robustness and applicability of transcription factor and pathway analysis tools on single-cell RNA-seq data. Genome Biol. 21, 36 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Kharchenko, P. V., Silberstein, L. & Scadden, D. T. Bayesian approach to single-cell differential expression analysis. Nat. Methods 11, 740–742 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Satija, R., Farrell, J. A., Gennert, D., Schier, A. F. & Regev, A. Spatial reconstruction of single-cell gene expression data. Nat. Biotechnol. 33, 495–502 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Korsunsky, I. et al. Fast, sensitive and accurate integration of single-cell data with Harmony. Nat. Methods 16, 1289–1296 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This project has been made possible in part by grant no. 2018-182827 from the Chan Zuckerberg Initiative DAF, an advised fund of the Silicon Valley Community Foundation. H.H. is a Miguel Servet (CP14/00229) researcher funded by the Spanish Institute of Health Carlos III (ISCIII). C.M. is supported by an AECC postdoctoral fellowship. This work has received funding from the European Union’s Horizon 2020 research and innovation program under Marie Skłodowska-Curie grant agreement no. H2020-MSCA-ITN-2015-675752 (Singek), and the Ministerio de Ciencia, Innovación y Universidades (SAF2017-89109-P; AEI/FEDER, UE). S. was supported by the German Research Foundation’s (DFG’s) (GR4980) Behrens-Weise-Foundation. D.G. and S. are supported by the Max Planck Society. C.Z. was supported by the European Molecular Biology Organization through the long-term fellowship ALTF 673-2017. The snRNA-seq data were generated with support from the National Institute of Allergy and Infectious Diseases (grant no. U24AI118672), the Manton Foundation and the Klarman Cell Observatory (to A.R.). I.N. was supported by JST CREST (grant no. JPMJCR16G3), Japan, and the Projects for Technological Development, Research Center Network for Realization of Regenerative Medicine by Japan, the Japan Agency for Medical Research and Development. A.J., L.E.W., J.W.B. and W.E. were supported by funding from the DFG (EN 1093/2-1 and SFB1243 TP A14). We thank ThePaperMill for critical reading and scientific editing services and the Eukaryotic Single Cell Genomics Facility at Scilifelab (Stockholm, Sweden) for support. This publication is part of a project (BCLLATLAS) that received funding from the European Research Council under the European Union’s Horizon 2020 research and innovation program (grant agreement no. 810287). Core funding was from the ISCIII and the Generalitat de Catalunya.

Author information

Author notes

  1. These authors contributed equally: Elisabetta Mereu, Atefeh Lafzi.

Authors and Affiliations

  1. CNAG-CRG, Centre for Genomic Regulation, Barcelona Institute of Science and Technology, Barcelona, Spain

    Elisabetta Mereu, Atefeh Lafzi, Catia Moutinho, Marta Gut, Ivo Gut & Holger Heyn

  2. Department of Cell and Molecular Biology, Karolinska Institutet, Stockholm, Sweden

    Christoph Ziegenhain & Rickard Sandberg

  3. European Molecular Biology Laboratory, European Bioinformatics Institute, Hinxton, Cambridge, UK

    Davis J. McCarthy & Oliver Stegle

  4. European Molecular Biology Laboratory, Genome Biology Unit, Heidelberg, Germany

    Davis J. McCarthy & Oliver Stegle

  5. St Vincent’s Institute of Medical Research, Fitzroy, Victoria, Australia

    Davis J. McCarthy

  6. Institute for Research in Biomedicine, Barcelona Institute of Science and Technology, Barcelona, Spain

    Adrián Álvarez-Varela & Eduard Batlle

  7. Catalan Institution for Research and Advanced Studies, Barcelona, Spain

    Eduard Batlle

  8. Centro de Investigación Biomédica en Red de Cáncer, Barcelona, Spain

    Eduard Batlle

  9. Max-Planck-Institute of Immunobiology and Epigenetics, Freiburg, Germany

    Sagar & Dominic Grün

  10. 10x Genomics, Pleasanton, CA, USA

    Julia K. Lau & Stéphane C. Boutet

  11. Fluidigm Corporation, South San Francisco, CA, USA

    Chad Sanada & Aik Ooi

  12. Department of Bioengineering, Stanford University, Stanford, CA, USA

    Robert C. Jones

  13. Bio-Rad, Hercules, CA, USA

    Kelly Kaihara, Chris Brampton & Yasha Talaga

  14. Laboratory for Bioinformatics Research, RIKEN Center for Biosystems, Dynamics Research, Saitama, Japan

    Yohei Sasagawa, Kaori Tanaka, Tetsutaro Hayashi & Itoshi Nikaido

  15. Max Delbrück Center for Molecular Medicine/Berlin Institute of Health, Berlin, Germany

    Caroline Braeuning, Cornelius Fischer & Sascha Sauer

  16. Digital Health Center, Berlin Institute of Health, Charité-Universitätsmedizin Berlin, Berlin, Germany

    Timo Trefzer & Christian Conrad

  17. Klarman Cell Observatory, Broad Institute of MIT and Harvard, Cambridge, MA, USA

    Xian Adiconis, Lan T. Nguyen, Aviv Regev & Joshua Z. Levin

  18. Stanley Center for Psychiatric Research, Broad Institute of MIT and Harvard, Cambridge, MA, USA

    Xian Adiconis & Joshua Z. Levin

  19. Koch Institute of Integrative Cancer Research, MIT, Cambridge, MA, USA

    Aviv Regev

  20. Howard Hughes Medical Institute, Department of Biology, MIT, Cambridge, MA, USA

    Aviv Regev

  21. Max-Planck-Institute for Biology of Ageing, Cologne, Germany

    Swati Parekh

  22. Anthropology & Human Genomics, Department of Biology II, Ludwig-Maximilians-University, Martinsried, Germany

    Aleksandar Janjic, Lucas E. Wange, Johannes W. Bagnoli & Wolfgang Enard

  23. School of Integrative and Global Majors, University of Tsukuba, Wako, Saitama, Japan

    Itoshi Nikaido

  24. Universitat Pompeu Fabra, Barcelona, Spain

    Ivo Gut & Holger Heyn

  25. Division of Computational Genomics and Systems Genetics, German Cancer Research Center, Heidelberg, Germany

    Oliver Stegle

Authors
  1. Elisabetta Mereu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Atefeh Lafzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Catia Moutinho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Ziegenhain
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Davis J. McCarthy
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Adrián Álvarez-Varela
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Eduard Batlle
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sagar
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Dominic Grün
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Julia K. Lau
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Stéphane C. Boutet
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Chad Sanada
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Aik Ooi
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Robert C. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Kelly Kaihara
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Chris Brampton
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Yasha Talaga
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Yohei Sasagawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Kaori Tanaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Tetsutaro Hayashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Caroline Braeuning
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Cornelius Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Sascha Sauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Timo Trefzer
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Christian Conrad
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Xian Adiconis
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Lan T. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Aviv Regev
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. Joshua Z. Levin
    View author publications

    You can also search for this author in PubMed Google Scholar

  30. Swati Parekh
    View author publications

    You can also search for this author in PubMed Google Scholar

  31. Aleksandar Janjic
    View author publications

    You can also search for this author in PubMed Google Scholar

  32. Lucas E. Wange
    View author publications

    You can also search for this author in PubMed Google Scholar

  33. Johannes W. Bagnoli
    View author publications

    You can also search for this author in PubMed Google Scholar

  34. Wolfgang Enard
    View author publications

    You can also search for this author in PubMed Google Scholar

  35. Marta Gut
    View author publications

    You can also search for this author in PubMed Google Scholar

  36. Rickard Sandberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  37. Itoshi Nikaido
    View author publications

    You can also search for this author in PubMed Google Scholar

  38. Ivo Gut
    View author publications

    You can also search for this author in PubMed Google Scholar

  39. Oliver Stegle
    View author publications

    You can also search for this author in PubMed Google Scholar

  40. Holger Heyn
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H.H. designed the study. E.M. and A.L. performed all data analyses. C.M., A.A.V. and E.B. prepared the reference sample. C.Z., D.J.M., S.P. and O.S. supported the data analysis. M.G. and I.G. provided technical and sequencing support. S., D.G., J.K.L., S.C.B., C.S., A.O., R.C.J., K.K., C.B., Y.T., Y.S., K.T., T.H., C.B., C.F., S.S., T.T., C.C., X.A., L.T.N., A.R., J.Z.L., A.J., L.E.W., J.W.B., W.E., R.S. and I.N. provided sequencing-ready single-cell libraries or sequencing raw data. H.H., E.M. and A.L. wrote the manuscript with contributions from the co-authors. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Holger Heyn.

Ethics declarations

Competing interests

A.R. is a co-founder and equity holder of Celsius Therapeutics, and an SAB member of Thermo Fisher Scientific and Syros Pharmaceuticals. He is also a co-inventor on patent applications to numerous advances in single-cell genomics, including droplet-based sequencing technologies, as in PCT/US2015/0949178, and methods for expression and analysis, as in PCT/US2016/059233 and PCT/US2016/059239. K.K., C.B. and Y.T. are employed by Bio-Rad Laboratories. J.K.L. and S.C.B. are employees and shareholders at 10x Genomics, Inc. S.C.B. is a former employee and shareholder of Fluidigm Corporation. C.S. and A.O. are employed by Fluidigm. All other authors declare no conflicts of interest associated with this manuscript.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mereu, E., Lafzi, A., Moutinho, C. et al. Benchmarking single-cell RNA-sequencing protocols for cell atlas projects. Nat Biotechnol 38, 747–755 (2020). https://doi.org/10.1038/s41587-020-0469-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41587-020-0469-4

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing