Many journals have introduced checklists to comply with NIH guidelines for reproducibility in research. But some societies are not happy with the "one-size-fits-all" approach. Credit: Chris Ryan

Guidelines from the US National Institutes of Health (NIH) that aim to improve the reproducibility of preclinical research are premature, burdensome and may be ineffective, an influential group of scientific societies says.

The NIH introduced its guidelines last November in the wake of reports that only a fraction of landmark biomedical findings could be reproduced1,2. The guidance includes aims such as making crucial data sets available on request, and providing thorough descriptions of cell lines, animals, antibodies and other reagents used in experiments. Authors must also state whether they have taken steps such as blinding and randomization. Journals including Nature have adopted the guidelines in the form of checklists to ensure good reporting standards in experiments.

But last week, the president of the Federation of American Societies for Experimental Biology (FASEB), Joseph Haywood, called for a step back from “one size fits all” guidelines in an e-mail to NIH principal deputy director Lawrence Tabak. FASEB, based in Bethesda, Maryland, represents 27 societies that together have some 120,000 members and publish 62 peer-reviewed journals. Haywood argues that the guidelines could make preparing, submitting and reviewing research papers more burdensome, and that the checklists and other requirements imposed by journals could make it more difficult to recruit high-quality peer reviewers.

Exceptions to the rule

There can be a good case not to blind or randomize some experiments or to have smaller sample sizes, especially when studies are labelled as ‘exploratory’ and will not be used as evidence to launch clinical trials, says Jonathan Kimmelman, a biomedical ethicist who studies clinical-research requirements at McGill University in Montreal, Canada. The NIH guidelines do not dictate how experiments should be designed, however: they just stipulate that experimental methods be explicitly stated, he notes.

FASEB members are worried that they might be pressured to report details that would not affect the experiment just for the sake of reporting, says Yvette Seger, FASEB's director of science policy, adding: “Would people be so beholden to a checklist that they would forget what kind of science they were looking at?”

Much of the concern, she says, is because it is unclear exactly how the guidelines will be implemented and how much flexibility will be allowed. There is also uncertainty around what kinds of experiment count as preclinical research. FASEB leaders think that action should be taken to improve reproducibility, and are grateful that the NIH is tackling the issue, but there should have been more discussion with the scientific community, Seger says. The NIH released the guidelines after a workshop last June with journal editors from more than 30 titles (including Science, Nature, and Cell). “We would have loved to have been at the table,” she says.

More than 80 journals, including at least three run by FASEB member societies, have endorsed the NIH guidelines. But many journals have not. The American Chemical Society (which publishes titles such as ACS Chemical Biology) and the Royal Society Journals (which include Biology Letters) have decided not to endorse them, saying they have their own procedures for ensuring reproducibility. The Biophysical Journal, published by the Cell Press group, published an editorial last week saying that while it agreed with the guidelines’ intent, it felt they were “not pertinent or applicable to the types of science" the journal published. The Proceedings of the National Academies of Science is still deliberating about whether to sign up.

Adapt as appropriate

The NIH's Tabak says he is encouraging journals to sign the guidelines, and that he has already seen improvements in the scientific literature because of them, such as authors explicitly stating that an animal study did not have statistical power to assess a hypothesis. The journals who came together to draft a set of general principles for improving transparency and reproducibility should be commended, he says. But he adds that developing guidelines is an iterative process that requires continual refinement. "Journals that sent us the letters should be commended also. They are paying attention, they are thinking about it. This is a good thing."

Neuroscientist Ulrich Dirnagl at the Charité Medical University in Berlin, who is editor-in-chief of the Journal of Cerebral Blood Flow and Metabolism, thinks that the part journals have to play is relatively minor. “The cleanup that is needed must come from scientists themselves, nudged by the funders,” he says.

The crucial factor is that researchers take on the spirit of the guidelines, says Glenn Begley, chief scientific officer at pharmaceutical firm TetraLogic in Malvern, Pennsylvania, who co-authored a 2012 Nature commentary calling for more rigorous standards in biomedical research1. “If ‘improved rigour’ merely becomes a box-checking exercise, rather than a fundamental embrace of good scientific method that is inculcated into every experiment, we will have failed,” he says.