Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

Challenges in benchmarking metagenomic profilers

Nature Methods volume 18, pages 618–626 (2021)Cite this article

Subjects

Abstract

Accurate microbial identification and abundance estimation are crucial for metagenomics analysis. Various methods for classification of metagenomic data and estimation of taxonomic profiles, broadly referred to as metagenomic profilers, have been developed. Nevertheless, benchmarking of metagenomic profilers remains challenging because some tools are designed to report relative sequence abundance while others report relative taxonomic abundance. Here we show how misleading conclusions can be drawn by neglecting this distinction between relative abundance types when benchmarking metagenomic profilers. Moreover, we show compelling evidence that interchanging sequence abundance and taxonomic abundance will influence both per-sample summary statistics and cross-sample comparisons. We suggest that the microbiome research community pay attention to potentially misleading biological conclusions arising from this issue when benchmarking metagenomic profilers, by carefully considering the type of abundance data that were analyzed and interpreted and clearly stating the strategy used for metagenomic profiling.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Comparison of profiling results.
Fig. 2: Correlation between sequence and taxonomic abundance in synthetic profiles based on different kingdoms.
Fig. 3: Quantitative and qualitative benchmarking results of four representative metagenomic profilers using 25 simulated communities.
Fig. 4: Alpha diversity based on sequence and taxonomic abundance.
Fig. 5: Ordination analyses of simulated profiles based on rJSD.

Similar content being viewed by others

Data availability

All simulated datasets can be downloaded from https://figshare.com/projects/Challenges_in_Benchmarking_Metagenomic_Profilers/79916.Source data are provided with this paper.

Code availability

R scripts used in this paper are available at https://github.com/shihuang047/re-benchmarking

References

  1. Knight, R. et al. Best practices for analysing microbiomes. Nat. Rev. Microbiol. 16, 410–422 (2018).

    Article  CAS  Google Scholar 

  2. Ye, S. H., Siddle, K. J., Park, D. J. & Sabeti, P. C. Benchmarking metagenomics tools for taxonomic classification. Cell 178, 779–794 (2019).

    Article  CAS  Google Scholar 

  3. Liu, B., Gibbons, T., Ghodsi, M., Treangen, T. & Pop, M. Accurate and fast estimation of taxonomic profiles from metagenomic shotgun sequences. BMC Genomics 12, S4 (2011).

    Article  CAS  Google Scholar 

  4. Arumugam, M. et al. Enterotypes of the human gut microbiome. Nature 473, 174–180 (2011).

    Article  CAS  Google Scholar 

  5. Milanese, A. et al. Microbial abundance, activity and population genomic profiling with mOTUs2. Nat. Commun. 10, 1014 (2019).

    Article  Google Scholar 

  6. Wood, D. E., Lu, J. & Langmead, B. Improved metagenomic analysis with Kraken 2. Genome Biol. 20, 257 (2019).

    Article  CAS  Google Scholar 

  7. Wood, D. E. & Salzberg, S. L. Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome Biol. 15, R46 (2014).

    Article  Google Scholar 

  8. Lu, J., Breitwieser, F. P., Thielen, P. & Salzberg, S. L. Bracken: estimating species abundance in metagenomics data. PeerJ Comput. Sci. 3, e104 (2017).

    Article  Google Scholar 

  9. Kostic, A. D. et al. PathSeq: software to identify or discover microbes by deep sequencing of human tissue. Nat. Biotechnol. 29, 393–396 (2011).

    Article  CAS  Google Scholar 

  10. Menzel, P., Ng, K. L. & Krogh, A. Fast and sensitive taxonomic classification for metagenomics with Kaiju. Nat. Commun. 7, 11257 (2016).

    Article  CAS  Google Scholar 

  11. Buchfink, B., Xie, C. & Huson, D. H. Fast and sensitive protein alignment using DIAMOND. Nat. Methods 12, 59–60 (2015).

    Article  CAS  Google Scholar 

  12. Truong, D. T. et al. MetaPhlAn2 for enhanced metagenomic taxonomic profiling. Nat. Methods 12, 902–903 (2015).

    Article  CAS  Google Scholar 

  13. Segata, N. et al. Metagenomic microbial community profiling using unique clade-specific marker genes. Nat. Methods 9, 811–814 (2012).

    Article  CAS  Google Scholar 

  14. Sunagawa, S. et al. Metagenomic species profiling using universal phylogenetic marker genes. Nat. Methods 10, 1196–1199 (2013).

    Article  CAS  Google Scholar 

  15. Nurk, S., Meleshko, D., Korobeynikov, A. & Pevzner, P. A. metaSPAdes: a new versatile metagenomic assembler. Genome Res. 27, 824–834 (2017).

    Article  CAS  Google Scholar 

  16. Li, D. et al. MEGAHIT v1.0: a fast and scalable metagenome assembler driven by advanced methodologies and community practices. Methods 102, 3–11 (2016).

    Article  CAS  Google Scholar 

  17. Mavromatis, K. et al. Use of simulated data sets to evaluate the fidelity of metagenomic processing methods. Nat. Methods 4, 495–500 (2007).

    Article  CAS  Google Scholar 

  18. Meyer, F. et al. Assessing taxonomic metagenome profilers with OPAL. Genome Biol. 20, 51 (2019).

  19. Sczyrba, A. et al. Critical Assessment of Metagenome Interpretation—a benchmark of metagenomics software. Nat. Methods 14, 1063–1071 (2017).

    Article  CAS  Google Scholar 

  20. McIntyre, A. B. R. et al. Comprehensive benchmarking and ensemble approaches for metagenomic classifiers. Genome Biol. 18, 182 (2017).

    Article  Google Scholar 

  21. Lindgreen, S., Adair, K. L. & Gardner, P. P. An evaluation of the accuracy and speed of metagenome analysis tools. Sci. Rep. 6, 19233 (2016).

    Article  CAS  Google Scholar 

  22. Chen, F., Mackey, A. J., Vermunt, J. K. & Roos, D. S. Assessing performance of orthology detection strategies applied to eukaryotic genomes. PLoS ONE 2, e383 (2007).

    Article  Google Scholar 

  23. Soppa, J. Polyploidy in archaea and bacteria: about desiccation resistance, giant cell size, long-term survival, enforcement by a eukaryotic host and additional aspects. J. Mol. Microbiol. Biotechnol. 24, 409–419 (2014).

    Article  CAS  Google Scholar 

  24. Mendell, J. E., Clements, K. D., Choat, J. H. & Angert, E. R. Extreme polyploidy in a large bacterium. Proc. Natl Acad. Sci. USA 105, 6730–6734 (2008).

    Article  CAS  Google Scholar 

  25. Gloor, G. B., Macklaim, J. M., Pawlowsky-Glahn, V. & Egozcue, J. J. Microbiome datasets are compositional: and this is not optional. Front. Microbiol. 8, 2224 (2017).

    Article  Google Scholar 

  26. Aitchison, J. On criteria for measures of compositional distance. Math. Geol. 24, 365–379 (1992).

    Article  Google Scholar 

  27. Martino, C. et al. A novel sparse compositional technique reveals microbial perturbations. mSystems 4, e00016–e00019 (2019).

    Article  Google Scholar 

  28. Aitchison, J., Barceló-Vidal, C., Martín-Fernández, J. A., Pawlowsky-Glahn, V. & Logratio Analysis and compositional distance. Math. Geol. 32, 271–275 (2000).

    Article  Google Scholar 

  29. Breitwieser, F. P., Lu, J. & Salzberg, S. L. A review of methods and databases for metagenomic classification and assembly. Brief. Bioinform. 20, 1125–1136 (2019).

    Article  CAS  Google Scholar 

  30. Legendre, P., Borcard, D. & Peres-Neto, P. R. Analyzing beta diversity: partitioning the spatial variation of community composition data. Ecol. Monogr. 75, 435–450 (2005).

    Article  Google Scholar 

  31. Mantel, N. The detection of disease clustering and a generalized regression approach. Cancer Res. 27, 209–220 (1967).

    CAS  PubMed  Google Scholar 

  32. Faith, D. P., Minchin, P. R. & Belbin, L. Compositional dissimilarity as a robust measure of ecological distance. Vegetatio 69, 57–68 (1987).

    Article  Google Scholar 

  33. Legendre, P. & Gallagher, E. D. Ecologically meaningful transformations for ordination of species data. Oecologia 129, 271–280 (2001).

    Article  Google Scholar 

  34. van der Maaten, L. J. P. & Hinton, G. E. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

    Google Scholar 

  35. McInnes, L. & Healy, J. UMAP: uniform manifold approximation and projection for dimension reduction. J. Open Source Softw. 3, 861 (2018).

    Article  Google Scholar 

  36. Dray, S., Chessel, D. & Thioulouse, J. Procrustean co-inertia analysis for the linking of multivariate datasets. Écoscience 10, 110–119 (2003).

    Article  Google Scholar 

  37. Digby, P. & Kempton, R. Multivariate Analysis of Ecological Communities (Palgrave MacMillan, 1987).

  38. Human Microbiome Project Consortium. Structure, function and diversity of the healthy human microbiome. Nature 486, 207–214 (2012).

    Article  Google Scholar 

  39. Hsu, T. et al. Urban transit system microbial communities differ by surface type and interaction with humans and the environment. mSystems 1, e00018-16 (2016).

    Article  Google Scholar 

Download references

Acknowledgements

Research reported in this publication was supported by grant nos. R01AI141529, R01HD093761, R01AG067744, UH3OD023268, U19AI095219 and U01HL089856 from the National Institutes of Health. This work was also supported by IBM Research through the AI Horizons Network, UC San Diego AI for Healthy Living program in partnership with the UC San Diego Center for Microbiome Innovation.

Author information

Author notes

  1. These authors contributed equally: Zheng Sun, Shi Huang.

Authors and Affiliations

  1. Channing Division of Network Medicine, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA

    Zheng Sun & Yang-Yu Liu

  2. Department of Pediatrics, University of California, San Diego, La Jolla, CA, USA

    Shi Huang, Qiyun Zhu, Yoshiki Vázquez-Baeza & Rob Knight

  3. Center for Microbiome Innovation, Jacobs School of Engineering, University of California, San Diego, La Jolla, CA, USA

    Shi Huang, Qiyun Zhu, Yoshiki Vázquez-Baeza & Rob Knight

  4. Key Laboratory of Dairy Biotechnology and Engineering, Ministry of Education, Inner Mongolia Agricultural University, Hohhot, China

    Meng Zhang

  5. IBM T. J. Watson Research Center, Yorktown Heights, NY, USA

    Niina Haiminen & Laxmi Parida

  6. IBM Research Europe, The Hartree Centre, Warrington, UK

    Anna Paola Carrieri

  7. AI and Cognitive Software, IBM Research-Almaden, San Jose, CA, USA

    Ho-Cheol Kim

  8. Department of Computer Science & Engineering, University of California, San Diego, La Jolla, CA, USA

    Rob Knight

  9. Department of Bioengineering, University of California, San Diego, La Jolla, CA, USA

    Rob Knight

Authors
  1. Zheng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qiyun Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Niina Haiminen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Anna Paola Carrieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yoshiki Vázquez-Baeza
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Laxmi Parida
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ho-Cheol Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Rob Knight
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Yang-Yu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.-Y.L. and R.K. conceived and designed the analysis. Z.S. and S.H. led the analysis. M.Z., Q.Z., N.H., A.P.C., Y.V.-B, L.P. and H.-C.K. contributed evaluation strategies. All authors analyzed the results. Z.S., S.H., Y.-Y.L. and R.K. wrote the paper. All authors edited the paper.

Corresponding authors

Correspondence to Rob Knight or Yang-Yu Liu.

Ethics declarations

Competing interests

This work received support from IBM Research through the AI Horizons Network. Coauthors N.H., A.P.C., L.P. and H.-C.K. are employees of IBM. The authors declare no other competing interests.

Additional information

Peer review information Nature Methods thanks the anonymous reviewers for their contribution to the peer review of this work. Lin Tang was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Note, Figs. 1–6 and references.

Reporting Summary

Supplementary Data

An Excel file that includes the source data used in each of the Supplementary Figures.

Source data

Source Data Fig. 1

Source data used in Fig. 1.

Source Data Fig. 2

Source data used in Fig. 2.

Source Data Fig. 3

Source data used in Fig. 3.

Source Data Fig. 4

Source data used in Fig. 4.

Source Data Fig. 5

Source data used in Fig. 5.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, Z., Huang, S., Zhang, M. et al. Challenges in benchmarking metagenomic profilers. Nat Methods 18, 618–626 (2021). https://doi.org/10.1038/s41592-021-01141-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-021-01141-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing