Illustration by David Parkins
The history of science shows that irreproducibility is not a product of our times. Some 350 years ago, the chemist Robert Boyle penned essays on “the unsuccessfulness of experiments”. He warned readers to be sceptical of reported work. “You will meet with several Observations and Experiments, which ... may upon further tryal disappoint your expectation.” He attributed the problem to a 'lack of skill in the scientist and the lack of purity of the ingredients', and what would today be referred to as inadequate statistical power.
By 1830, polymath Charles Babbage was writing in more cynical terms. In Reflections on the Decline of Science in England, he complains of “several species of impositions that have been practised in science”, namely “hoaxing, forging, trimming and cooking”.
In other words, irreproducibility is the product of two factors: faulty research practices and fraud. Yet, in our view, current initiatives to improve science dismiss the second factor. For example, leaders at the US National Institutes of Health (NIH) stated in 2014: “With rare exceptions, we have no evidence to suggest that irreproducibility is caused by scientific misconduct”1. In 2015, a symposium of several UK science-funding agencies convened to address reproducibility, and decided to exclude discussion of deliberate fraud.
To dismiss the role of research misconduct is mistaken and unfortunate. At best, ignoring deliberate misconduct in efforts to reduce irreproducibility is a wasted opportunity, like tilling a field without clearing it of rocks. At worst, it permits destructive behaviour to persist and flourish.
Scale of evidence
Only 10–12 individuals are found guilty by the US Office of Research Integrity (ORI) each year. That number, which the NIH used to dismiss the role of research misconduct1, is misleadingly low, as numerous studies show. For instance, a review2 of 2,047 life-science papers retracted from 1973 to 2012 found that around 43% were attributed to fraud or suspected fraud. A compilation of anonymous surveys3 suggests that 2% of scientists and trainees admit that they have fabricated, falsified or modified data. And a 1996 study4 of more than 1,000 postdocs found that more than one-quarter would select or omit data to improve their chances of receiving grant funding.
Admittedly, many causes of irreproducibility do not involve dishonesty. The NIH has promoted responsible research for 25 years by funding studies on research integrity, creating educational resources and backing the ORI.
Nonetheless, we contend that when scientific leaders minimize “hoaxing, forging, trimming and cooking” as contributors to irreproducibility, they choose to ignore the problem rather than confront it. This mechanism is what psychiatrists term denial, when an individual faces what they believe to be an insoluble problem. Deliberate misconduct is a reality that government funders can and must address. In 2012, an article in this journal declared that “the time is right to confront misconduct”. We agree; it is even more urgent now. We recommend five key approaches (see ‘Preventing misconduct’).
To diminish the threat that misconduct poses to science, scientists and society:
- Authorities should acknowledge that deliberate misconduct is an important contributor to irreproducibility.
- Mentors should be evaluated to assure quality; those who contribute to misconduct should be penalized.
- Institutions and government agencies should have procedures to protect whistle-blowers from retaliation.
- Senior faculty members who are found guilty of misconduct should face severe penalties.
- Institutions that fail to establish and follow policies and processes to prevent misconduct should be sanctioned.
In the 1990s, the NIH mandated that all of the trainees it funds must receive a course on the responsible conduct of research. Not surprisingly, it failed in its goal of reducing research misconduct5 — which it defines as fabrication, falsification or plagiarism. Presumably, the ethics proscribing such practices are established long before people enter science. Instead, we propose interventions to address the psychological factors that motivate individuals to commit misconduct, depending on their role in the research hierarchy.
Those found guilty of misconduct by the ORI fall into three categories in roughly equal measure: trainees, support staff and senior scientists. Each has its own motivations6.
Trainees. Many trainee missteps can be traced to a fear of failure and a lack of quality mentorship. One study7 of trainees who were found guilty of misconduct revealed that 62% of their mentors had not established adequate procedures, such as providing clear rules on data ownership and recording, safety, materials transfer or scheduling regular meetings, and 73% had not reviewed trainees' raw data. A survey8 at a major US cancer centre found that nearly one-third of 140 trainees felt pressure to “prove” a mentor's hypothesis, even though results did not support it.
Some trainees who commit misconduct are perfectionists and are unable to cope with failure. Mentors should intervene with perspective, encouragement and even referral to counselling. They should assure trainees that there are respected careers outside a tenure-track appointment. Instead, junior scientists report that they are treated as cheap labour; their professional development is a low priority.
Funders should craft policies to ensure that mentors act as an adviser, teacher and role model, and should limit the number of trainees per mentor by discipline. Each year, trainees should be required to complete anonymous questionnaires evaluating their mentors, and results should be sent to funding agencies as well as to research deans.
Institutions should reward mentors for outstanding performance and provide adequate training. When justified, mentors should be held responsible for misconduct by their trainees and appropriately sanctioned.
Support staff. Workers such as laboratory technicians, phlebotomists and data collectors represent about one-third of the individuals annually found guilty by the ORI of deliberate misconduct. They may have falsified data to boost their income or reduce their workload in response to an investigator's unrealistic productivity goals.
Treating support staff as valued members of a team could go a long way. They should be made aware of the study's goals, and how invalid publications harm scientific progress and patient care.
Senior researchers. Established scientists would be less likely to commit misconduct if they were more concerned about being detected and punished. Currently, they conclude that the risk is low: few cases are referred to the ORI and few of their colleagues want to be enmeshed in a conflict.
More than 80% of faculty members say that they would be reluctant to report potential misconduct for fear of being ostracized and damaging their own reputations9. One ORI study found that 47 out of 68 people who reported misconduct experienced an adverse consequence. Clearly, concerns about making allegations are justified.
Well-articulated policies are key to helping whistle-blowers come forward. So too is a well-trained research integrity officer (RIO), ideally a respected faculty member or administrator. The potential whistle-blower must be confident that the institution's RIO and its policies will protect them from retaliation.
Institutions. Research centres should build a culture and infrastructure that encourages integrity. For example, peers can emphasize their commitment to robust data in everyday interactions and by supporting random audits; and data systems can date-stamp and track who accesses files to protect them from manipulation. Leaders should make it clear that they will not tolerate misconduct and that perpetrators will suffer severe consequences.
An effective RIO is crucial for leading the educational and enforcement effort and in building trust in the integrity of the institution. Unfortunately, studies have shown that many RIOs are poorly trained and do not manage allegations and investigations of research misconduct effectively. Perhaps most significantly, they might fail to adequately prepare and protect the whistle-blower10. The RIO must be selected thoughtfully and provided with sufficient authority and support.
Any institution that receives US federal research funds should be required to have at least one designated, trained and certified RIO who has been assessed by the ORI. Moreover, research funds should not be released to an institution that cannot demonstrate current certification.
Institutions that fail to establish and execute policies to assure integrity should be held responsible when misconduct occurs. For example, in July 2014, Iowa State University agreed to repay US$496,000 and forego $1.4 million in grants after one of its researchers was found guilty of fraud. However, this penalty, as well as a prison sentence for the fraudster, happened only because a senator intervened. That should not be necessary.
Government officials should be prepared to pursue repayments. The threat of such penalties should have a chilling effect on investigators contemplating research misconduct, and motivate institutions to establish and implement policies that reflect their commitment to institutional integrity.
We believe that these system-wide interventions are essential to have an impact on the irreproducibility produced by research misconduct.
- Journal name:
- Date published: