The purpose of this editorial is to flag to clinicians, researchers and consumers an important document published by The Academy of Medical Sciences (in collaboration with other large international scientific organisations and funders of science).1 The document is titled: Reproducibility and reliability of biomedical research: improving research practice. It tackles the problem of why there are so many initial studies with positive findings that are subsequently shown to be false. While the document focuses on basic science and not specifically SCI-related basic science, the issues are equally applicable to all areas of basic and clinical SCI research.
The document was compiled after a 2-day meeting by some of the world’s top scientists, research methodologists and funders of research. They were from all areas of science including neurophysiology. The impetus for the meeting came from the growing concern that a large amount of research is not reproducible. That is, researchers are declaring far more false positives than chance alone would dictate. This is a concern because time, money and effort is wasted pursuing false leads, and in the meantime, consumers of healthcare are given false hope. The document not only summarises the extent of the problem but it also summarises the possible causes and drivers of the problem. It then outlines some possible solutions that are very relevant to the SCI-research community.
False positive findings are often associated with new and exciting interventions that promise better outcomes for patients. They are invariably announced with great fanfare, media attention and excitement. This is understandable but often these findings are quietly disproven or left to sink into the sea of obscurity. It can take many, many years to disprove initial findings and the scientific community can devote a lot of time, cost and effort trying to replicate initial results. Alarmingly, often the promising initial findings become embedded into clinical practice making it even more difficult to remedy.
The Academy of Medical Sciences argues that false positives in the scientific literature are predominantly due to poor scientific practice. This includes amongst many things: p hacking, data dredging, opportunistic stopping of studies, failure to publish negative findings, selective reporting of outcomes, removal of uncooperative data, use of non-blinded assessors, failure to control for natural recovery, failure to control for repeated exposure to testing protocols, poor randomisation procedures, elevation of positive secondary outcomes to primary outcome status, inappropriate sub-group analyses and more. Most of these practices are well documented which raises the question as to why they still persist and what our scientific community can do to eradicate them.
Disturbingly, The Academy of Medical Sciences argue that some of the poor scientific practices arise from highly competitive academic environments which generate pressures to publish and to make discoveries. However, poor scientific practice can also arise from more innocuous factors such as loyalties to professions, ideas or concepts. All of these factors encourage researchers to put spin on findings and to be selective about what they report and the papers they cite.
The international community of SCI researchers, clinicians and consumers all share one concern: to advance the care of people with spinal cord injuries. If we cannot be confident about the reproducibility of our research findings, we will be chasing our tails. After all, a novel and exciting discovery is of no value until it can be independently reproduced. It is therefore upon all of us to better understand the forces behind false positives and do everything possible to minimise their occurrence.
The Academy of Medical Sciences Reproducibility and reliability of biomedical research: improving research practice: Symposium report, October 2015.
About this article
Cite this article
Harvey, L. Reproducing the research results of others: a problem for all including researchers within the SCI world. Spinal Cord 55, 223 (2017). https://doi.org/10.1038/sc.2017.17