Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

A multicenter study benchmarks software tools for label-free proteome quantification

Nature Biotechnology volume 34, pages 1130–1136 (2016)Cite this article

Subjects

Abstract

Consistent and accurate quantification of proteins by mass spectrometry (MS)-based proteomics depends on the performance of instruments, acquisition methods and data analysis software. In collaboration with the software developers, we evaluated OpenSWATH, SWATH 2.0, Skyline, Spectronaut and DIA-Umpire, five of the most widely used software methods for processing data from sequential window acquisition of all theoretical fragment-ion spectra (SWATH)-MS, which uses data-independent acquisition (DIA) for label-free protein quantification. We analyzed high-complexity test data sets from hybrid proteome samples of defined quantitative composition acquired on two different MS instruments using different SWATH isolation-window setups. For consistent evaluation, we developed LFQbench, an R package, to calculate metrics of precision and accuracy in label-free quantitative MS and report the identification performance, robustness and specificity of each software tool. Our reference data sets enabled developers to improve their software tools. After optimization, all tools provided highly convergent identification and reliable quantification performance, underscoring their robustness for label-free quantitative proteomics.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Study workflow.
Figure 2: Protein-level LFQbench results for data from TripleTOF 6600 and 64-SWATH-window setup.
Figure 3: Integrated analysis of the five software tools.
Figure 4: Differences in retention time and correlation of reported peak intensities among all software tools for the respective matching precursors.

Similar content being viewed by others

References

  1. Aebersold, R. & Mann, M. Mass spectrometry-based proteomics. Nature 422, 198–207 (2003).

    Article  CAS  Google Scholar 

  2. Mallick, P. & Kuster, B. Proteomics: a pragmatic perspective. Nat. Biotechnol. 28, 695–709 (2010).

    Article  CAS  Google Scholar 

  3. Distler, U., Kuharev, J. & Tenzer, S. Biomedical applications of ion mobility-enhanced data-independent acquisition-based label-free quantitative proteomics. Expert Rev. Proteomics 11, 675–684 (2014).

    Article  CAS  Google Scholar 

  4. Gillet, L.C. et al. Targeted data extraction of the MS/MS spectra generated by data-independent acquisition: a new concept for consistent and accurate proteome analysis. Mol. Cell. Proteomics 11, O111.016717 (2012).

    Article  Google Scholar 

  5. Geromanos, S.J., Hughes, C., Ciavarini, S., Vissers, J.P.C. & Langridge, J.I. Using ion purity scores for enhancing quantitative accuracy and precision in complex proteomics samples. Anal. Bioanal. Chem. 404, 1127–1139 (2012).

    Article  CAS  Google Scholar 

  6. Geiger, T., Cox, J. & Mann, M. Proteomics on an Orbitrap benchtop mass spectrometer using all-ion fragmentation. Mol. Cell. Proteomics 9, 2252–2261 (2010).

    Article  CAS  Google Scholar 

  7. Liu, H., Sadygov, R.G. & Yates, J.R. III. A model for random sampling and estimation of relative protein abundance in shotgun proteomics. Anal. Chem. 76, 4193–4201 (2004).

    Article  CAS  Google Scholar 

  8. Li, G.-Z. et al. Database searching and accounting of multiplexed precursor and product ion spectra from the data independent analysis of simple and complex peptide mixtures. Proteomics 9, 1696–1719 (2009).

    Article  CAS  Google Scholar 

  9. Michalski, A., Cox, J. & Mann, M. More than 100,000 detectable peptide species elute in single shotgun proteomics runs but the majority is inaccessible to data-dependent LC-MS/MS. J. Proteome Res. 10, 1785–1793 (2011).

    Article  CAS  Google Scholar 

  10. Gatto, L. et al. Testing and validation of computational methods for mass spectrometry. J. Proteome Res. 15, 809–814 (2016).

    Article  CAS  Google Scholar 

  11. Dufresne, C. et al. ABRF research group development and characterization of a proteomics normalization standard consisting of 1,000 stable isotope labeled peptides. J. Biomol. Tech. 25, S1 (2014).

    Article  Google Scholar 

  12. Yates, J.R. III et al. Toward objective evaluation of proteomic algorithms. Nat. Methods 9, 455–456 (2012).

    Article  CAS  Google Scholar 

  13. Leprevost, Fda.V., Barbosa, V.C., Francisco, E.L., Perez-Riverol, Y. & Carvalho, P.C. On best practices in the development of bioinformatics software. Front. Genet. 5, 199 (2014).

    Article  Google Scholar 

  14. Pak, H. et al. Clustering and filtering tandem mass spectra acquired in data-independent mode. J. Am. Soc. Mass Spectrom. 24, 1862–1871 (2013).

    Article  CAS  Google Scholar 

  15. The difficulty of a fair comparison. Nat. Methods 12, 273 (2015).

  16. Kuharev, J., Navarro, P., Distler, U., Jahn, O. & Tenzer, S. In-depth evaluation of software tools for data-independent acquisition based label-free quantification. Proteomics 15, 3140–3151 (2015).

    Article  CAS  Google Scholar 

  17. Sajic, T., Liu, Y. & Aebersold, R. Using data-independent, high-resolution mass spectrometry in protein biomarker research: perspectives and clinical applications. Proteomics Clin. Appl. 9, 307–321 (2015).

    Article  CAS  Google Scholar 

  18. Röst, H.L. et al. OpenSWATH enables automated, targeted analysis of data-independent acquisition MS data. Nat. Biotechnol. 32, 219–223 (2014).

    Article  Google Scholar 

  19. MacLean, B. et al. Skyline: an open source document editor for creating and analyzing targeted proteomics experiments. Bioinformatics 26, 966–968 (2010).

    Article  CAS  Google Scholar 

  20. Bruderer, R. et al. Extending the limits of quantitative proteome profiling with data-independent acquisition and application to acetaminophen-treated three-dimensional liver microtissues. Mol. Cell. Proteomics 14, 1400–1410 (2015).

    Article  CAS  Google Scholar 

  21. Reiter, L. et al. mProphet: automated data processing and statistical validation for large-scale SRM experiments. Nat. Methods 8, 430–435 (2011).

    Article  CAS  Google Scholar 

  22. Tsou, C.-C. et al. DIA-Umpire: comprehensive computational framework for data-independent acquisition proteomics. Nat. Methods 12, 258–264, 7, 264 (2015).

    Article  CAS  Google Scholar 

  23. Cox, J. et al. Accurate proteome-wide label-free quantification by delayed normalization and maximal peptide ratio extraction, termed MaxLFQ. Mol. Cell. Proteomics 13, 2513–2526 (2014).

    Article  CAS  Google Scholar 

  24. Elias, J.E. & Gygi, S.P. Target-decoy search strategy for increased confidence in large-scale protein identifications by mass spectrometry. Nat. Methods 4, 207–214 (2007).

    Article  CAS  Google Scholar 

  25. Navarro, P. et al. General statistical framework for quantitative proteomics by stable isotope labeling. J. Proteome Res. 13, 1234–1247 (2014).

    Article  CAS  Google Scholar 

  26. Bell, A.W. et al. A HUPO test sample study reveals common problems in mass spectrometry-based proteomics. Nat. Methods 6, 423–430 (2009).

    Article  CAS  Google Scholar 

  27. Schubert, O.T. et al. Building high-quality assay libraries for targeted analysis of SWATH MS data. Nat. Protoc. 10, 426–441 (2015).

    Article  CAS  Google Scholar 

  28. Rosenberger, G. et al. A repository of assays to quantify 10,000 human proteins by SWATH-MS. Sci. Data 1, 140031 (2014).

    Article  CAS  Google Scholar 

  29. Shteynberg, D., Nesvizhskii, A.I., Moritz, R.L. & Deutsch, E.W. Combining results of multiple search engines in proteomics. Mol. Cell. Proteomics 12, 2383–2393 (2013).

    Article  CAS  Google Scholar 

  30. Yuan, Z.-F., Lin, S., Molden, R.C. & Garcia, B.A. Evaluation of proteomic search engines for the analysis of histone modifications. J. Proteome Res. 13, 4470–4478 (2014).

    Article  CAS  Google Scholar 

  31. Distler, U. et al. Drift time-specific collision energies enable deep-coverage data-independent acquisition proteomics. Nat. Methods 11, 167–170 (2014).

    Article  CAS  Google Scholar 

  32. Fonslow, B.R. et al. Digestion and depletion of abundant proteins improves proteomic coverage. Nat. Methods 10, 54–56 (2013).

    Article  CAS  Google Scholar 

  33. Wis´niewski, J.R., Zougman, A., Nagaraj, N. & Mann, M. Universal sample preparation method for proteome analysis. Nat. Methods 6, 359–362 (2009).

    Article  Google Scholar 

  34. Escher, C. et al. Using iRT, a normalized retention time for more targeted measurement of peptides. Proteomics 12, 1111–1121 (2012).

    Article  CAS  Google Scholar 

  35. Eng, J.K., Jahan, T.A. & Hoopmann, M.R. Comet: an open-source MS/MS sequence database search tool. Proteomics 13, 22–24 (2013).

    Article  CAS  Google Scholar 

  36. Perkins, D.N., Pappin, D.J., Creasy, D.M. & Cottrell, J.S. Probability-based protein identification by searching sequence databases using mass spectrometry data. Electrophoresis 20, 3551–3567 (1999).

    Article  CAS  Google Scholar 

  37. Reiter, L. et al. Protein identification false discovery rates for very large proteomics data sets generated by tandem mass spectrometry. Mol. Cell. Proteomics 8, 2405–2417 (2009).

    Article  CAS  Google Scholar 

  38. Lam, H. et al. Development and validation of a spectral library searching method for peptide identification from MS/MS. Proteomics 7, 655–667 (2007).

    Article  CAS  Google Scholar 

  39. Deutsch, E.W. et al. TraML–a standard format for exchange of selected reaction monitoring transition lists. Mol. Cell. Proteomics 11, R111.015040 (2012).

    Article  Google Scholar 

  40. Kunszt, P. et al. iPortal: the swiss grid proteomics portal: Requirements and new features based on experience and usability considerations. Concurr. Comput. 27, 433–445 (2015).

    Article  Google Scholar 

  41. Ning, K., Fermin, D. & Nesvizhskii, A.I. Comparative analysis of different label-free mass spectrometry based protein abundance estimates and their correlation with RNA-Seq gene expression data. J. Proteome Res. 11, 2261–2271 (2012).

    Article  CAS  Google Scholar 

  42. Röst, H.L. et al. TRIC: an automated alignment strategy for reproducible protein quantification in targeted proteomics. Nat. Methods 13, 777–783 (2016).

    Article  Google Scholar 

  43. Fenyö, D. & Beavis, R.C. A method for assessing the statistical significance of mass spectrometry-based protein identifications using general scoring schemes. Anal. Chem. 75, 768–774 (2003).

    Article  Google Scholar 

  44. Kim, S. & Pevzner, P.A. MS-GF+ makes progress towards a universal database search tool for proteomics. Nat. Commun. 5, 5277 (2014).

    Article  CAS  Google Scholar 

  45. Nesvizhskii, A.I., Keller, A., Kolker, E. & Aebersold, R. A statistical model for identifying proteins by tandem mass spectrometry. Anal. Chem. 75, 4646–4658 (2003).

    Article  CAS  Google Scholar 

  46. Shteynberg, D. et al. iProphet: multi-level integrative analysis of shotgun proteomic data improves peptide and protein identification rates and error estimates. Mol. Cell. Proteomics 10, M111.007690 (2011).

    Article  Google Scholar 

  47. Silva, J.C., Gorenstein, M.V., Li, G.-Z., Vissers, J.P.C. & Geromanos, S.J. Absolute quantification of proteins by LCMSE: a virtue of parallel MS acquisition. Mol. Cell. Proteomics 5, 144–156 (2006).

    Article  CAS  Google Scholar 

  48. Vizcaíno, J.A. et al. The PRoteomics IDEntifications (PRIDE) database and associated tools: status in 2013. Nucleic Acids Res. 41, D1063–D1069 (2013).

    Article  Google Scholar 

Download references

Acknowledgements

We thank R. Spohrer for sample preparation and L. Burton, A. Lau and G. Ivosev for their support with SWATH 2.0. P.N. and S.T. are supported by grants from the Bundesministerium für Bildung und Forschung (BMBF) (Express2Present 0316179C), the Deutsche Forschungsgemeinschaft (DFG) (ST599/1-1, ST599/2-1) and Mainz University (Research Center for Immunotherapy (FZI)). H.L.R. is supported by Swiss National Science Foundation grant P2EZP3_162268. Y.P.-R. is supported by the Biotechnology and Biological Sciences Research Council (BBSRC) 'PROCESS' grant (BB/K01997X/1). A.I.N. is supported by US National Institutes of Health grant 5R01GM94231. R.A. was supported by European Research Council (ERC) AdG 233226 (Proteomics version 3.0) and ERC-2014-AdG 670821 (Proteomicxs 4D), the PhosphonetX project of SystemsX.ch and the Swiss National Science Foundation grant 3100A_166435.

Author information

Author notes

  1. Pedro Navarro and Jörg Kuharev: These authors contributed equally to this work.

Authors and Affiliations

  1. Institute for Immunology, University Medical Center of the Johannes-Gutenberg University Mainz, Mainz, Germany

    Pedro Navarro, Jörg Kuharev, Ute Distler & Stefan Tenzer

  2. Department of Biology, Institute of Molecular Systems Biology, Eidgenoessische Technische Hochschule (IMSB-ETH) Zurich, Zurich, Switzerland

    Ludovic C Gillet, Hannes L Röst, George Rosenberger & Ruedi Aebersold

  3. Biognosys AG, Schlieren, Switzerland

    Oliver M Bernhardt & Lukas Reiter

  4. Department of Genome Sciences, University of Washington, Seattle, Washington, USA

    Brendan MacLean

  5. AB Sciex, Concord, Ontario, Canada

    Stephen A Tate

  6. Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, Michigan, USA

    Chih-Chiang Tsou & Alexey I Nesvizhskii

  7. PhD Program in Systems Biology, University of Zurich and ETH Zurich, Zurich, Switzerland

    George Rosenberger

  8. European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Hinxton, UK

    Yasset Perez-Riverol

  9. Department of Pathology, University of Michigan, Ann Arbor, Michigan, USA

    Alexey I Nesvizhskii

  10. Faculty of Science, University of Zurich, Zurich, Switzerland

    Ruedi Aebersold

Authors
  1. Pedro Navarro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jörg Kuharev
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ludovic C Gillet
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Oliver M Bernhardt
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Brendan MacLean
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hannes L Röst
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Stephen A Tate
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Chih-Chiang Tsou
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lukas Reiter
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ute Distler
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. George Rosenberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Yasset Perez-Riverol
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Alexey I Nesvizhskii
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Ruedi Aebersold
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Stefan Tenzer
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

P.N. and S.T. designed and supervised the study; U.D. and L.C.G. prepared the samples and performed the MS measurements; L.C.G., G.R. and H.L.R. executed and supervised the OpenSWATH analyses; P.N. and S.A.T. executed and supervised the SWATH 2.0 analyses; P.N. and B.M. executed and supervised the Skyline analyses; P.N., O.M.B. and L.R. executed and supervised the Spectronaut analyses; C.-C.T. and A.I.N. executed and supervised the DIA-Umpire analyses; J.K., P.N. and Y.P.-R. developed LFQbench; P.N., S.T., J.K., B.M. and O.M.B. performed the benchmark analyses; L.C.G., O.M.B., B.M., H.L.R., S.A.T., C.-C.T., L.R., G.R., A.I.N. and R.A. provided critical input into the project; P.N., J.K. and S.T. wrote the manuscript.

Corresponding authors

Correspondence to Pedro Navarro or Stefan Tenzer.

Ethics declarations

Competing interests

S.A.T. is employed by SCIEX, and O.M.B. and L.R. are employed by Biognosys AG.

Supplementary information

Supplementary Text and Figures

Supplementary Tables 1–9, Supplementary Figures 1–25 and Supplementary Notes 1–6 (PDF 28116 kb)

Supplementary Code

LFQbench (R-package) source code (ZIP 19693 kb)

Supplementary Data Set 1

List of peptides exclusively identified by DIA-Umpire in the HYE124 64 variable windows experiment (XLSX 131 kb)

Supplementary Data Set 2

Ion library for 32 fixed SWATH windows experiments (ZIP 14442 kb)

Supplementary Data Set 3

Ion library for 32 variable SWATH windows experiments (ZIP 15376 kb)

Supplementary Data Set 4

Ion library for 64 fixed SWATH windows experiments (ZIP 15413 kb)

Supplementary Data Set 5

Ion library for 64 variable SWATH windows experiments (ZIP 14695 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Navarro, P., Kuharev, J., Gillet, L. et al. A multicenter study benchmarks software tools for label-free proteome quantification. Nat Biotechnol 34, 1130–1136 (2016). https://doi.org/10.1038/nbt.3685

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nbt.3685

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research