Data-independent acquisition (DIA) has been gaining momentum in label-free protein quantification by mass spectrometry (MS). The technique has the potential to provide a full and precise picture of peptide abundance because all precursor ions in a defined m/z ratio are fragmented regardless of intensity. But the MS data generated are highly complex, and the robustness and accuracy of DIA rely in part on the computational tools used to analyze such data.

Pedro Navarro and Stefan Tenzer at Johannes-Gutenberg University Mainz in Germany, along with their colleagues, benchmarked computational tools for analyzing sequential window acquisition of all theoretical fragment-ion spectra (SWATH-MS) data, a particularly promising DIA method.

But objective comparison is easier said than done. A method's performance depends to some extent on the experience level of the user. Reference data sets used to evaluate the software should represent the spectrum of samples commonly analyzed and be complex enough to test the methods in depth. Measures and statistical tests to assess performance must be carefully chosen and applied.

Navarro, Tenzer and colleagues took these factors into account when designing a comparison of five tools for SWATH-MS data analysis: DIA-Umpire, SWATH2.0, OpenSWATH, Skyline and Spectronaut. They started by generating two proteomic hybrid reference data sets from defined ratios of bacterial, yeast and human proteomes, which they analyzed using different instruments and SWATH acquisition modes. Then the data went to the software developers, who knew the ins and outs of their tools and could optimize their performance. To standardize the evaluation of data analysis, the researchers developed LFQbench, an R package that calculates and graphically represents the precision and accuracy of label-free quantification based on hybrid data sets.

After a first round of analysis, differences in measured protein abundance among the tools helped reveal potential pitfalls. The developers then used these insights to improve their tools. In the end, the benchmarking exercise did not yield a clear winner; instead, the performance of the methods converged on a high point, with all methods enabling accurate identification and quantification at the peptide and protein levels. Where methods abound, carefully thought-out and executed comparisons can benefit both the user and the developer.