Research that cannot be reproduced has become one of the most debated issues in science. In April, a group of influential UK biomedical funding agencies held a meeting to discuss the problem — and have just released their findings. Dorothy Bishop, a researcher in developmental neuropsychology at the University of Oxford who led the April meeting, talks to Nature about the reproducibility problem and how to solve it.
Your report notes that it is difficult to quantify the scale of the reproducibility problem. Do you think it is a major problem?
It is very clear that there are things that are not being done optimally and that people are quite often absolutely unaware of this. In psychology, we have recently obtained quantification of this from the reproducibility project [a collaborative effort which found that the results of more than half of psychology studies could not be replicated]. Those results are a bit disturbing. It’s not just a minor problem on the fringe of the subject.
The group that is most interesting are the early-career researchers who seem to be both aware of the problem and terribly worried about it. If they are bold and do some of the things we have recommended in the report — such as adopting preregistration, being more collaborative, using open data and open methods — then I think they feel that they might be putting themselves at a career disadvantage. They have a sense that a lot of very senior figures either are unaware of the problem or disapprove of these solutions.
Is it true that senior scientists disapprove of these solutions?
I can only speak here for psychology. I’m very enthusiastic about the idea that you preregister your studies — you make a quite detailed protocol of your design and intended analysis, and get that reviewed before you do the study. But I think a lot of other scientists felt under personal attack from those advocating preregistration, that others were saying they weren’t doing science properly. There were scientists talking about the ‘reproducibility police’ or even ‘reproducibility Nazis’. That has to some extent calmed down. But it’s still the case that you get kickback.
Have you come across reproducibility problems in your research?
Yes. I think I had a sort of epiphany when I was doing electrophysiology research in children with reading and language problems. I tried to do a meta-analysis of the literature and realized that nobody ever did reproduce anybody’s results. Everybody did things differently.
How much of a problem is the pressure on people to get published in high-profile journals?
That is something that needs to be addressed, as the report notes. People doing slow, careful science, possibly over several years, and checking every step of the way, may find themselves left behind because they haven’t got a paper in a top journal. One of the things that we might be able to do is to start giving credit for practices that are associated with reproducibility. Things like having open data, having papers preregistered, having open methods. Those sorts of things could be rewarded, not just by employers but also by funders in deciding who to give grants to.
What else would you like to see done?
We’ve identified the 'top-down' and the 'bottom-up' approaches. The top-down approach would be for funders, institutions and journals to change practices to favour reproducible research, rather than sexy research. But the bottom-up is that we can train researchers better. We’re aware that there are instances of people actually being told by their superiors to do things that are not right in terms of statistical analyses and so on.
Will this report have impact?
The bodies behind the report represent the key people in biomedical research in the United Kingdom. We’ve got the Academy of Medical Sciences, the Wellcome Trust, the Biotechnology and Biological Sciences Research Council, the Medical Research Council, all on board and all very enthusiastic about this. That gives us a certain degree of clout. In Britain, this is the first time there has actually been a body and a group of influential funders who have said ‘this matters to us’. People will be more likely to take notice than if it’s just me or somebody else saying you should go for preregistration.
All of the problems are quite depressing. Is there a positive message?
I don’t think this is all so negative. I feel quite upbeat about it. I think it’s a case where we’ve become aware of a problem. We have to avoid pointing fingers and making it sound like it’s somebody’s fault. Now that we’re aware of it, we have all sorts of ideas about how to deal with it. These are doable things. I feel that the mood is one of making science a much better thing. It might lead to slightly slower science. That could be better.
- Journal name: