Nature | Editorial

Replication studies offer much more than technical details

They demonstrate the practice of science at its best.

Article tools

Purists will tell you that science is about what scientists don’t know, which is true but not much of a basis on which to develop new cancer drugs. Hence the importance of knowledge: how crucial this mutation or that cell-surface receptor really is to cancer growth. These are the findings that launch companies and clinical trials — provided, of course, that they have been published in research papers in peer-reviewed journals.

As we report in a News story this week, a systematic effort to check some of these findings by repeating an initial five published cancer studies has reported that none could be completely reproduced. The significance of this divergence — how the specific experiments were selected and what the results mean for the broader agenda of reproducibility in research — is already hotly contested.

Perhaps the most influential aspect of the exercise, called the Reproducibility Project: Cancer Biology, has nothing to do with those arguments. It lies beneath the surface, in the peer reviews of the project teams’ replication plans, which were published before the studies began. These reviews can be read as part of the editorial decision letters linked to each replication plan, or ‘registered report’ (see go.nature.com/2jte08a).

What, one might ask, could be less interesting? What insights can emerge from technical feedback on plans to repeat what other groups of scientists have done? Plenty. The decision letters reveal the practice of science at its best: probing whether an experiment truly tests a particular idea; identifying shortcomings in the original set-ups; and proposing further, sounder tests. To those committed to improving science, reading these insights — the fruit of voluntary donations of time and effort — will prove a moving experience. Journal clubs would do well to read the original papers, and then compare their own analyses with those of the scholars who have reviewed the replication reports.

Again and again, the peer reviewers and the replicators clash. The reviewers are eager to produce the best experiment to test a publication’s conclusions; they want to correct deficiencies in the design of the original high-impact studies. The replicators do, on several occasions, agree to add an extra measurement, particularly of positive and negative controls that had originally been neglected. Often, however, they resist calls for more definitive studies. Testing “the experiment’s underlying hypothesis,” they insist, “is not an aim of the project.”

“Grand conclusions are built on myriad laboratory experiments.”

This is a frustrating, if understandable, response. It is easier to compare the results of highly similar experiments than to assess a conclusion. Thus, the replication efforts are not especially interested in, say, the big question of whether public gene-expression data can point to surprising uses for new drugs. They focus instead on narrower points — such as whether a specific finding that an ulcer drug stalls the growth of lung cancer in mice holds up (it did, more or less; see I. Kandela et al. eLife 6, e17044; 2017). Even so, the results are not definitive. One set of dissimilar results can’t establish that the original result was a false positive; nor do similar results show that the original conclusion is correct.

Yet a project that sticks to direct replication and eschews broader ‘truth’ can still raise bigger issues. After all, grand conclusions are built on myriad laboratory experiments. How well (or not) such experiments work outside the original lab is too often communicated through gossip rather than open discourse. For just US$2 million in private funding — less than a typical 5-year grant from the US National Institutes of Health to a single investigator — this replication project shines a very public light on the sticking points of experiments.

One aspect that merits increased focus is how markedly the results of control experiments varied between the original work and replications. In one case, mice in the control group of an original study survived for nine weeks after being engrafted with tumours, whereas those in the replication survived for just one. In another, a bioluminescent signal used to track tumour burden differed markedly between the original and replication studies. Such situations occur frequently in biology, but are often overlooked because researchers focus only on results in their own laboratories. There is great value in highlighting variability in ‘established’ experimental systems.

Such results deserve more attention, both technically and philosophically. Researchers routinely optimize experimental conditions to see the signals that they are looking for. But how far can this proceed until confirmation bias makes results uninterpretable? How does one do pilot studies without overly encouraging favourable results?

More than 50 years ago, the philosopher Thomas Kuhn defined ‘normal science’ as the kind of work that faithfully supports or chisels away at current hypotheses. It is easy to dismiss this as workmanlike and uninteresting. But only by doing such normal science — and doing it well — can we recognize when revolutions are necessary.

Journal name:
Nature
Volume:
541,
Pages:
259–260
Date published:
()
DOI:
doi:10.1038/541259b

For the best commenting experience, please login or register as a user and agree to our Community Guidelines. You will be re-directed back to this page where you will see comments updating in real-time and have the ability to recommend comments to other users.

Comments for this thread are now closed.

Comments

1 comment Subscribe to comments

  1. Avatar for mihai nadin
    mihai nadin
    Why replication is failing The reproducibility crisis is not of methodology, but of perspective. The realization that the living dynamics (cancer is an outcome of change) and that of the physico-chemical are different. Empirical evidence from evolutionary outcomes led to the conclusion that the phase space of the living is continuously changing (Longo, 2017). In the replication attempts made public so far, scientists took note of the cell line genetic drift and the variability of the microbiome. The variability of tumor formation assays in vivo is also related to the variability of the phase space of the living (healthy or not). Furthermore, the complexity threshold for the living is decidability (Nadin, 2016). Therefore the expectation of experimental reproducibility is a goal incompatible with the nature of the change processes examined. When tumors grow too quickly or too slowly—given the non-deterministic nature of the processes examined—replication becomes uninterpretable. The only replications that partially met the expectations referred to the physico-chemistry level—peptides that penetrate tumors-- which is below the decidability threshold that characterizes the living. None of the experiments—five Replication Studies have been published so far—and attempts at replication paid any attention to the holistic nature of living dynamics. The Reproducibility Project, representative for the crisis of reproducibility, generated a large body of evidence for alternative views. Instead of feeling sorry for themselves because the expectation of repeatability cannot be met, those scientists still in the grip of the “theology” of the reductionist-deterministic view of the world should use their impressive knowledge in order to advance an understanding of the living—in particular the cancer affecting it—grounded in its condition. The Nature editorial contrasts normal science and revolution in science. The time of the normal is way over. Nature | Editorial, 541:7637, 18 January 2017 Longo, G, How future depends on past and rare events in systems of life, Foundations of Science, 2017, IEA Nantes. http://www.di.ens.fr/users/longo/files/biolog-observ-history-future.pdf Nadin, M. Medicine; The decisive test of anticipation. In: Nadin, M. (ed.) Anticipation and Medicine. Cham: Springer, 2016, 1–27.
sign up to Nature briefing

What matters in science — and why — free in your inbox every weekday.

Sign up

Listen

new-pod-red

Nature Podcast

Our award-winning show features highlights from the week's edition of Nature, interviews with the people behind the science, and in-depth commentary and analysis from journalists around the world.