Tumor samples consist of subclones with different mutations and evolutionary trajectories. Inference of tumor subclonality using DNA sequencing data is challenging. “There are a remarkable number of methods for doing this, and effectively no good benchmarks of which work well and which do not,” says Paul Boutros, from the University of Toronto and the University of California, Los Angeles.

Boutros shared the idea of building a framework for benchmarking subclonal reconstruction methods with others, which led to a community undertaking. Inferring tumor subclonality needs to estimate global characteristics of tumor composition, assign individual mutations to each subclone, and reconstruct phylogenetic relationships. For each task, the team assessed a set of metrics and selected the most suitable ones. Although this sounds straightforward, Boutros was surprised by how much experts sometimes disagreed. “In one part of the study we showed experts in the field a true evolutionary history for a tumor, and then several different types of mistakes that might be made in reconstructing it. We then asked them, ‘Which mistake is worst?’, and while there was agreement in many situations, in others experts really prioritized different aspects of tumor evolution.” This is where a community-based approach helped greatly.

There are other components of the framework, including a simulator of tumor genomes and a tool for efficiently executing a library of methods for subclonal reconstruction. Using these resources, Boutros and colleagues found the performance of a method is often influenced by factors such as read depth and the detection pipeline for single nucleotide variants. The framework is being used in the ICGC-TCGA DREAM Somatic Mutation Calling Tumor Heterogeneity Challenge for evaluating subclonal reconstruction methods. Beyond that, “I hope that people will use this strategy to develop more tightly tailored benchmarks for specific biological situations of interest not touched in our work (e.g., comparing primary and metastatic disease),” says Boutros.