Reply to Updating benchtop sequencing performance comparison

Loman et al. reply:

We were pleased to see this useful update from Jünemann et al.¹ to our article 'Performance comparison of benchtop sequencers'². Progress in sequencing technologies is driving genomic research at an astonishing rate. More than 14 months have elapsed since we submitted our manuscript based on data generated in the summer of 2011. There have been impressive changes in throughput (up to fivefold) and read length (up to fourfold) during this time, easily outperforming Moore's Law. However, we note that despite these improvements, our overall conclusions on the relative performance of the 454 GS Junior, Ion Torrent Personal Genome Machine (PGM) and Illumina MiSeq benchtop sequencers remain unchanged.

One anomalous issue in this article is the large discrepancy between the reported insertion and deletion (indel) rates from our two runs, of 316 chips, in July 2011 and those reported by Jünemann et al.¹. Without access to the data, we can only speculate about the reason, but it seems probable that the discrepancy is related to the different read-trimming procedures used. More stringent read-trimming algorithms are likely to result in an improvement in error rate, as there is a strong correlation between quality score and actual error rate (as noted in our original study)². We note that other, contemporaneous studies describe error rates for the PGM equivalent to those that we reported: in one study³ the total error rate was 1.78%, and in a second study⁴ an insertion rate of 0.693% and deletion rate of 0.965% were reported. We also note that the 100-base-pair data set generated by Jünemann et al.¹ on the 316 chip, contemporaneously with our study, performed particularly badly during de novo assembly with an N50 <1.5 kb and did not allow the vast majority of coding sequences in the Escherichia coli Sakai genome to be reconstructed without errors. Such poor assembly statistics at high coverage are hard to reconcile with the low error rates quoted by Jünemann et al.¹.

There is no sign that progress in genome sequencing technologies is slowing. Publication delays have the potential to limit the use of such platform comparisons, but we believe these comparisons are nonetheless more useful than marketing literature or anecdotes. We would welcome a community-led, open-access project to provide trustworthy benchmarking in a timely and objective fashion.