Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Challenges in benchmarking metagenomic profilers

Abstract

Accurate microbial identification and abundance estimation are crucial for metagenomics analysis. Various methods for classification of metagenomic data and estimation of taxonomic profiles, broadly referred to as metagenomic profilers, have been developed. Nevertheless, benchmarking of metagenomic profilers remains challenging because some tools are designed to report relative sequence abundance while others report relative taxonomic abundance. Here we show how misleading conclusions can be drawn by neglecting this distinction between relative abundance types when benchmarking metagenomic profilers. Moreover, we show compelling evidence that interchanging sequence abundance and taxonomic abundance will influence both per-sample summary statistics and cross-sample comparisons. We suggest that the microbiome research community pay attention to potentially misleading biological conclusions arising from this issue when benchmarking metagenomic profilers, by carefully considering the type of abundance data that were analyzed and interpreted and clearly stating the strategy used for metagenomic profiling.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Comparison of profiling results.
Fig. 2: Correlation between sequence and taxonomic abundance in synthetic profiles based on different kingdoms.
Fig. 3: Quantitative and qualitative benchmarking results of four representative metagenomic profilers using 25 simulated communities.
Fig. 4: Alpha diversity based on sequence and taxonomic abundance.
Fig. 5: Ordination analyses of simulated profiles based on rJSD.

Data availability

All simulated datasets can be downloaded from https://figshare.com/projects/Challenges_in_Benchmarking_Metagenomic_Profilers/79916.Source data are provided with this paper.

Code availability

R scripts used in this paper are available at https://github.com/shihuang047/re-benchmarking

References

  1. 1.

    Knight, R. et al. Best practices for analysing microbiomes. Nat. Rev. Microbiol. 16, 410–422 (2018).

    CAS  Article  Google Scholar 

  2. 2.

    Ye, S. H., Siddle, K. J., Park, D. J. & Sabeti, P. C. Benchmarking metagenomics tools for taxonomic classification. Cell 178, 779–794 (2019).

    CAS  Article  Google Scholar 

  3. 3.

    Liu, B., Gibbons, T., Ghodsi, M., Treangen, T. & Pop, M. Accurate and fast estimation of taxonomic profiles from metagenomic shotgun sequences. BMC Genomics 12, S4 (2011).

    CAS  Article  Google Scholar 

  4. 4.

    Arumugam, M. et al. Enterotypes of the human gut microbiome. Nature 473, 174–180 (2011).

    CAS  Article  Google Scholar 

  5. 5.

    Milanese, A. et al. Microbial abundance, activity and population genomic profiling with mOTUs2. Nat. Commun. 10, 1014 (2019).

    Article  Google Scholar 

  6. 6.

    Wood, D. E., Lu, J. & Langmead, B. Improved metagenomic analysis with Kraken 2. Genome Biol. 20, 257 (2019).

    CAS  Article  Google Scholar 

  7. 7.

    Wood, D. E. & Salzberg, S. L. Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome Biol. 15, R46 (2014).

    Article  Google Scholar 

  8. 8.

    Lu, J., Breitwieser, F. P., Thielen, P. & Salzberg, S. L. Bracken: estimating species abundance in metagenomics data. PeerJ Comput. Sci. 3, e104 (2017).

    Article  Google Scholar 

  9. 9.

    Kostic, A. D. et al. PathSeq: software to identify or discover microbes by deep sequencing of human tissue. Nat. Biotechnol. 29, 393–396 (2011).

    CAS  Article  Google Scholar 

  10. 10.

    Menzel, P., Ng, K. L. & Krogh, A. Fast and sensitive taxonomic classification for metagenomics with Kaiju. Nat. Commun. 7, 11257 (2016).

    CAS  Article  Google Scholar 

  11. 11.

    Buchfink, B., Xie, C. & Huson, D. H. Fast and sensitive protein alignment using DIAMOND. Nat. Methods 12, 59–60 (2015).

    CAS  Article  Google Scholar 

  12. 12.

    Truong, D. T. et al. MetaPhlAn2 for enhanced metagenomic taxonomic profiling. Nat. Methods 12, 902–903 (2015).

    CAS  Article  Google Scholar 

  13. 13.

    Segata, N. et al. Metagenomic microbial community profiling using unique clade-specific marker genes. Nat. Methods 9, 811–814 (2012).

    CAS  Article  Google Scholar 

  14. 14.

    Sunagawa, S. et al. Metagenomic species profiling using universal phylogenetic marker genes. Nat. Methods 10, 1196–1199 (2013).

    CAS  Article  Google Scholar 

  15. 15.

    Nurk, S., Meleshko, D., Korobeynikov, A. & Pevzner, P. A. metaSPAdes: a new versatile metagenomic assembler. Genome Res. 27, 824–834 (2017).

    CAS  Article  Google Scholar 

  16. 16.

    Li, D. et al. MEGAHIT v1.0: a fast and scalable metagenome assembler driven by advanced methodologies and community practices. Methods 102, 3–11 (2016).

    CAS  Article  Google Scholar 

  17. 17.

    Mavromatis, K. et al. Use of simulated data sets to evaluate the fidelity of metagenomic processing methods. Nat. Methods 4, 495–500 (2007).

    CAS  Article  Google Scholar 

  18. 18.

    Meyer, F. et al. Assessing taxonomic metagenome profilers with OPAL. Genome Biol. 20, 51 (2019).

  19. 19.

    Sczyrba, A. et al. Critical Assessment of Metagenome Interpretation—a benchmark of metagenomics software. Nat. Methods 14, 1063–1071 (2017).

    CAS  Article  Google Scholar 

  20. 20.

    McIntyre, A. B. R. et al. Comprehensive benchmarking and ensemble approaches for metagenomic classifiers. Genome Biol. 18, 182 (2017).

    Article  Google Scholar 

  21. 21.

    Lindgreen, S., Adair, K. L. & Gardner, P. P. An evaluation of the accuracy and speed of metagenome analysis tools. Sci. Rep. 6, 19233 (2016).

    CAS  Article  Google Scholar 

  22. 22.

    Chen, F., Mackey, A. J., Vermunt, J. K. & Roos, D. S. Assessing performance of orthology detection strategies applied to eukaryotic genomes. PLoS ONE 2, e383 (2007).

    Article  Google Scholar 

  23. 23.

    Soppa, J. Polyploidy in archaea and bacteria: about desiccation resistance, giant cell size, long-term survival, enforcement by a eukaryotic host and additional aspects. J. Mol. Microbiol. Biotechnol. 24, 409–419 (2014).

    CAS  Article  Google Scholar 

  24. 24.

    Mendell, J. E., Clements, K. D., Choat, J. H. & Angert, E. R. Extreme polyploidy in a large bacterium. Proc. Natl Acad. Sci. USA 105, 6730–6734 (2008).

    CAS  Article  Google Scholar 

  25. 25.

    Gloor, G. B., Macklaim, J. M., Pawlowsky-Glahn, V. & Egozcue, J. J. Microbiome datasets are compositional: and this is not optional. Front. Microbiol. 8, 2224 (2017).

    Article  Google Scholar 

  26. 26.

    Aitchison, J. On criteria for measures of compositional distance. Math. Geol. 24, 365–379 (1992).

    Article  Google Scholar 

  27. 27.

    Martino, C. et al. A novel sparse compositional technique reveals microbial perturbations. mSystems 4, e00016–e00019 (2019).

    Article  Google Scholar 

  28. 28.

    Aitchison, J., Barceló-Vidal, C., Martín-Fernández, J. A., Pawlowsky-Glahn, V. & Logratio Analysis and compositional distance. Math. Geol. 32, 271–275 (2000).

    Article  Google Scholar 

  29. 29.

    Breitwieser, F. P., Lu, J. & Salzberg, S. L. A review of methods and databases for metagenomic classification and assembly. Brief. Bioinform. 20, 1125–1136 (2019).

    CAS  Article  Google Scholar 

  30. 30.

    Legendre, P., Borcard, D. & Peres-Neto, P. R. Analyzing beta diversity: partitioning the spatial variation of community composition data. Ecol. Monogr. 75, 435–450 (2005).

    Article  Google Scholar 

  31. 31.

    Mantel, N. The detection of disease clustering and a generalized regression approach. Cancer Res. 27, 209–220 (1967).

    CAS  PubMed  Google Scholar 

  32. 32.

    Faith, D. P., Minchin, P. R. & Belbin, L. Compositional dissimilarity as a robust measure of ecological distance. Vegetatio 69, 57–68 (1987).

    Article  Google Scholar 

  33. 33.

    Legendre, P. & Gallagher, E. D. Ecologically meaningful transformations for ordination of species data. Oecologia 129, 271–280 (2001).

    Article  Google Scholar 

  34. 34.

    van der Maaten, L. J. P. & Hinton, G. E. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

    Google Scholar 

  35. 35.

    McInnes, L. & Healy, J. UMAP: uniform manifold approximation and projection for dimension reduction. J. Open Source Softw. 3, 861 (2018).

    Article  Google Scholar 

  36. 36.

    Dray, S., Chessel, D. & Thioulouse, J. Procrustean co-inertia analysis for the linking of multivariate datasets. Écoscience 10, 110–119 (2003).

    Article  Google Scholar 

  37. 37.

    Digby, P. & Kempton, R. Multivariate Analysis of Ecological Communities (Palgrave MacMillan, 1987).

  38. 38.

    Human Microbiome Project Consortium. Structure, function and diversity of the healthy human microbiome. Nature 486, 207–214 (2012).

    Article  Google Scholar 

  39. 39.

    Hsu, T. et al. Urban transit system microbial communities differ by surface type and interaction with humans and the environment. mSystems 1, e00018-16 (2016).

    Article  Google Scholar 

Download references

Acknowledgements

Research reported in this publication was supported by grant nos. R01AI141529, R01HD093761, R01AG067744, UH3OD023268, U19AI095219 and U01HL089856 from the National Institutes of Health. This work was also supported by IBM Research through the AI Horizons Network, UC San Diego AI for Healthy Living program in partnership with the UC San Diego Center for Microbiome Innovation.

Author information

Affiliations

Authors

Contributions

Y.-Y.L. and R.K. conceived and designed the analysis. Z.S. and S.H. led the analysis. M.Z., Q.Z., N.H., A.P.C., Y.V.-B, L.P. and H.-C.K. contributed evaluation strategies. All authors analyzed the results. Z.S., S.H., Y.-Y.L. and R.K. wrote the paper. All authors edited the paper.

Corresponding authors

Correspondence to Rob Knight or Yang-Yu Liu.

Ethics declarations

Competing interests

This work received support from IBM Research through the AI Horizons Network. Coauthors N.H., A.P.C., L.P. and H.-C.K. are employees of IBM. The authors declare no other competing interests.

Additional information

Peer review information Nature Methods thanks the anonymous reviewers for their contribution to the peer review of this work. Lin Tang was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Note, Figs. 1–6 and references.

Reporting Summary

Supplementary Data

An Excel file that includes the source data used in each of the Supplementary Figures.

Source data

Source Data Fig. 1

Source data used in Fig. 1.

Source Data Fig. 2

Source data used in Fig. 2.

Source Data Fig. 3

Source data used in Fig. 3.

Source Data Fig. 4

Source data used in Fig. 4.

Source Data Fig. 5

Source data used in Fig. 5.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sun, Z., Huang, S., Zhang, M. et al. Challenges in benchmarking metagenomic profilers. Nat Methods 18, 618–626 (2021). https://doi.org/10.1038/s41592-021-01141-3

Download citation

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing