A debate this week on ways to improve the practice of science quickly spread to social media. The event at University College London, called ‘Is science broken? If so, how can we fix it?’, included claims that some dubious laboratory practices, such as tweaking statistical analyses to make results seem significant, are widespread. One suggested solution — requiring scientists to register their experimental design and planned analysis with a journal before running any tests — received general support. Audience member Michael Markie, an associate publisher at the open-science platform F1000Research, shared his thoughts about preregistration on Twitter:
Chris Chambers, a neuroscientist at Cardiff University, UK, opened the discussion with a call for widespread preregistration, a step that would essentially require scientists to stick to their original protocols throughout the experiments and present all results, even those that failed to reach significance. In 2013, Chambers helped to implement a Registered Reports format at the journal Cortex, where he is now the Registered Reports editor. He adds that at least 13 other journals, including Experimental Psychology and AIMS Neuroscience, have adopted the preregistration model, and Royal Society Open Science is scheduled to do so later this year.
Scientists who wish to publish in Cortex first submit a detailed plan for the experiment and the statistical analysis. If the proposal passes peer review, the journal promises to publish the results, even if they do not show an effect, provided that the experiment actually follows the original plan. Chambers says that the traditional publishing model generally encourages scientists to chase exciting results instead of focusing on sound experimental practice. “Journal-based preregistration seeks to solve these problems by making publishing decisions before the results are known,” he says. “I want to look back on this period in 20 years and be able to say that we put in place a publishing model that rewarded transparency and reproducibility,” he says.
Some panellists, including Sam Schwarzkopf, an experimental psychologist at University College London (UCL), wondered whether a strict preregistration system might impose too many limits on scientific analysis. In an online comment about the event, Schwarzkopf said that he is “wary” of any system that prevents scientists from thoroughly examining their results. “I think except for the simplest designs there will always be things you can think of only when you see the data,” he wrote.
Schwarzkopf posted his comment on a blog written by Aidan Horner, a UCL cognitive neuroscientist who attended the event. In an interview, Horner said that it remains to be seen whether preregistration will actually make experimental results easier to verify and replicate. “The only way to collect that data is to introduce preregistration as an option and assess its impact,” he says. Overall, he says he left the debate feeling optimistic that scientists could find a way to address its core problems. “New problems emerge in science, but they’re dealt with over time.”
- Journal name:
- Date published: