Please visit methagora to view and post comments on this article.

Researchers planning a project often have to choose between several technical options. Although the choice is partly dictated by cost or availability, in some cases understanding the performance and limitations of available tools really does matter. Most often, the only way to find out is through a comparative experimental analysis. The recent literature includes a few examples of such studies that have benefited large groups of investigators.

DNA microarrays are the typical example of a technical platform on which several useful comparative analyses have been performed. Initially, researchers using microarrays to study gene expression were keen to address doubts that had been cast on the technology's reproducibility. Three large comparisons involving a grand total of 15 array platforms and 17 laboratories (Nat. Methods 2, 337–344, 345–350 and 351–356, 2005) established conditions under which microarray results could be compared between platforms and laboratories. At the initiative of the US Food and Drug Administration, the Microarray Quality Control (MAQC) consortium later completed another large analysis aimed at establishing to what extent microarrays were ready to be used as the basis for regulatory decisions (Nat. Biotechnol. 24, 1103, 2006). Very recently, oligonucleotide arrays have been compared for their suitability as readouts for chromatin immunoprecipitation assays (ChIP-chip), taking into account cost and sensitivity (Genome Res. 18, 393–403, 2008; and Research Highlight page 288).

All but one of these analyses were conducted jointly by several labs to assess the platforms' robustness. But single labs, if they have the instrumentation and expertise required, can also bring important insights to other aspects of performance. For example, in 2005, Steven Gygi and his colleagues performed a direct quantitative comparison of the two mass spectrometry instruments and the two sets of analysis software most commonly used at the time (Nat. Methods 2, 667–675, 2005). This analysis showed that the relatively low reproducibility between replicate data acquisitions can be exploited to increase sensitivity and confidence in large-scale protein identifications. Last year, Ruedi Aebersold's laboratory showed that the three most common phosphoprotein enrichment methods selectively enrich different portions of the phosphoproteome, and thereby concluded that no single method could be used for a comprehensive phosphoproteomics analysis (Nat. Methods 4, 231–237, 2007).

This type of comparison can also be very informative when focusing on reagents. For example, the International Stem Cell Initiative (ISCI) has characterized 59 human embryonic stem cell lines using standardized methods. This survey established that despite their disparate origins, the lines can be used interchangeably, at least to some extent (Nat. Biotechnol. 25, 803–816, 2007). In the laboratory of Roger Tsien, fluorescent proteins were subjected to a series of standardized tests to compare key properties (Nat. Methods 2, 905–909, 2005). Judging by the number of times this paper is still accessed online, this analysis continues to help investigators pick the protein that is best suited for their work.

Of course, such comparative analyses demand substantial time and effort from participating labs, and they are often costly. Although specific funding directed to this purpose is not always easy to obtain, recent notable grants have shown that some funding bodies take an interest in studies that can save time and money down the line. The NHGRI ENCODE project, for example, has funded the recent ChIP-chip analysis, and the International Stem Cell Forum has supported the ISCI cell line survey.

Companies that supply technologies and reagents also have a vital role to play in supporting comparative studies. Some can provide free reagents, as several companies did for the MAQC and ChIP-chip studies. Others can contribute to the preparation and distribution of well controlled test samples. (The MAQC test samples are now available from two manufacturers.)

To help disseminate informative comparative studies, Nature Methods has now adopted an additional article format called Analysis. We will be looking for more comparative studies that tackle important technologies (or methods or reagents) and quantitatively evaluate their performances and the source of their limitations. It is crucial that the methods or technology platforms tested be 'at their best'—the most recent versions, the most optimized protocols, the most experienced hands. Obviously, such analyses are useful only if their design permits a meaningful statistical analysis and if the test samples are carefully controlled and representative of 'real life' samples. Test samples, in fact, constitute in themselves an important resource for future benchmarking experiments (of new labs or new technologies) and should be prepared accordingly.

All the examples cited here illustrate how comparative analyses can provide critical insights into technologies, from establishing best practices to ensure comparability in collaborative projects to selecting the tools best suited for large-scale projects. The initial investment these analyses require does pay off with the ability to design better and more cost-effective experiments further down the line.