Time for the data to speak

Journal name:
Nature Biotechnology
Volume:
35,
Page:
689
Year published:
DOI:
doi:10.1038/nbt.3938
Published online

Retraction of a study claiming gene editing via an Argonaute enzyme illustrates the importance of post-publication peer review in the age of 24/7 media.

In this issue, Chunyu Han and colleagues retract a paper published in May 2016 claiming that an Argonaute protein (NgAgo) from the archaea Natronobacterium gregoryi can be guided by short 5′ phosphorylated single-stranded DNAs to generate double-strand breaks and edit the human genome (Nat. Biotechnol. 34, 768773, 2016). Although the paper was initially greeted with enthusiasm from researchers and intense media interest, speculation as to its reproducibility quickly grew, fueled by Twitter, blogs and other social media. Last November, this journal issued an Editorial Expression of Concern to alert the community to these reproducibility questions. Final resolution of the controversy necessitated the generation of additional experimental data from several groups over many months. That a retraction is now issued is testament to the considerable time, effort and funds invested by many laboratories around the globe that have sought to clarify NgAgo's function.

It is hard to overstate the impact of the Han paper following its publication last year, especially in China, where the paper originated. Coverage in the Chinese media was extensive, with headlines heralding the discovery of an entirely new gene editing system. The NgAgo report was easily the most widely covered paper in China last year; according to media monitor Meltwater, nearly 4,000 Chinese news stories cited the Han paper in just the first two months after publication.

The excitement generated by NgAgo centered on its potential to complement, or perhaps even supersede, the CRISPR–Cas9 gene editing system. NgAgo promised gene editing that required only a single target sequence (Cas9 needs both the target sequence and an additional adjacent recognition (PAM) sequence). What's more, initial data suggested advantages in terms of enhanced stability of the guide (DNA compared with RNA for Cas9), improved specificity, reduced off-target editing of the genome and improved activity in GC-rich regions of the genome; and the reagents used were easier to synthesize and handle.

If all this sounded too good to be true, the failure last summer of an increasing number of laboratories to reproduce the genome editing activity reported in the Han paper started to raise doubts. The paper became a hotly discussed topic at genome editing conferences, news groups and e-mail lists. It didn't take long before the press took notice. Claims and counterclaims regarding the validity of the initial report were exchanged. Nature Biotechnology's internal image integrity screening process found no obvious anomalies in the Han paper, a finding echoed by three external reviewers who reexamined the data.

Meanwhile, Nature Biotechnology kept in contact with the community about ongoing efforts to replicate the paper. Ultimately, the editors were able to coordinate the work of three independent groups into a single peer-reviewed refutation paper (Nat. Biotechnol. 35, 1718, 2017). With these data in hand, we then had sufficient cause to alert our readers to potential problems with the paper by publishing the Editorial Expression of Concern, which now appears alongside the original paper online—a step that was supported by two of the authors, including Han.

We also asked the authors if they could shed light on why the community was having difficulties reproducing their results. Accordingly, last December, Han and colleagues and several additional independent groups who contacted the journal provided new data claiming to have reproduced NgAgo gene editing activity. At the time, these data were judged too preliminary by the editors and an external reviewer to warrant publication. We decided to give the original authors and new groups more time to gather additional experimental evidence to bolster their claims.

Now, more than a year after the publication of the original report, we have learned that the independent groups that reported initial success in reproducing the results have not been able to bolster their preliminary data to a publishable level. Similarly, after seeking feedback from expert reviewers, we have concluded that the latest data from Han and his colleagues are insufficient to counter the substantial body of evidence that contradicts their initial findings. We are now convinced that the decision of Han and colleagues to retract the paper is the best course of action to support the integrity of the published record.

Publication of the NgAgo paper was not the end of the scientific process, it was the start. Like any other report that appears in the literature, it is the wider research community that tests methods, identifies potential sources of error, validates reagents and optimizes assays. In this case, it took dozens of dedicated individuals to work through the details of the published protocol and produce well-documented and controlled refutation studies (Protein Cell 7, 913915, 2016; Nat. Biotechnol. 35, 1718, 2017; Cell Res. 26, 13491352, 2016; PLoS One 12, e0177444, 2017).

The NgAgo controversy also illustrates the pros and cons of social media. Clearly, these platforms were valuable for rapidly alerting the wider scientific community to problems with the paper. But they also raised expectations that issues with this paper were straightforward and could be solved quickly. Unraveling all the problems with the NgAgo editing claim didn't happen in weeks or a few months for a reason. Even simple experiments take weeks to prepare, perform, analyze and troubleshoot. It does not help that the efforts of those carrying out replication studies often go unrewarded—it is unglamorous, unfunded and thankless work.

Little wonder then that to a 24/7 media and public that desire quick, definitive answers, the process of post-publication peer review can seem frustratingly slow. But when it comes to biology, answers are often not definitive. And when it comes to replication studies, the one thing we know is that it takes time. In the case of NgAgo, the time has come and the data have spoken.

Additional data