NEWS FEATURE

The science institutions hiring integrity inspectors to vet their papers

Some researchers have their manuscripts screened for errors before they go to journals.
Alison Abbott writes from Munich, Germany.

Search for this author in:

Illustration by Ana Kova

On 15 June 2017, scientists at a respected biological institute in Germany were thrown into crisis by an alarming announcement. An investigation into the Leibniz Institute on Aging had found that its director, cell biologist Karl Lenhard Rudolph, had published eight papers with data errors, including improperly edited or duplicated parts of images.

Investigators didn’t find deliberate fraud, but Rudolph wasn’t able to present original data to explain the problems. The Leibniz Association, which runs the institute in Jena and had commissioned the probe, concluded that Rudolph hadn’t supervised his lab group properly, and so was guilty of “grossly negligent scientific misconduct”. It applied the strictest sanctions it could, barring the institute from applying for research funding from the association while under Rudolph’s leadership for three years. It also ordered the centre to undergo an international review, even though the last one had been completed only a couple of years earlier. Rudolph resigned as director.

It was the second calamity in a year for the centre, which is also known as the Fritz Lipmann Institute (FLI). Police had raided it in 2016 after allegations that the centre had violated European regulations on animal experiments. The experiments were suspended, and although the FLI was cleared of the allegations, not all of the experiments had been re-authorized when the Rudolph affair broke. “The second crisis sent us into shock — it seemed more personal,” says molecular geneticist Christoph Englert, a group leader at the FLI, which employs 270 scientists. Most researchers at the centre hadn’t even known their director was under investigation.

FLI leaders set about restoring the centre’s reputation. They began by phasing in mandatory electronic databases and creating a system of thesis advisory committees to replace single PhD supervisors. The FLI’s head of core facilities, Matthias Görlach, had a less conventional idea. He contacted Enrico Bucci, a molecular biologist who had visited the FLI for some PhD work 18 years earlier, and with whom he’d kept in touch. Bucci was now in the business of checking research papers, Görlach knew; in 2016, he’d founded a science-integrity firm called Resis, based in Samone, Italy. Could the company perhaps help the institute to avoid errors in future?

So began a remarkable system of outside vetting, in which researchers at the FLI must send every paper and doctoral thesis across to Resis for screening before they submit them for publication. It’s an unusual step. Some journals check papers for errant statistics or manipulated images before publishing, but most research institutions say it’s up to the scientists themselves to ensure their manuscripts are correct. “I am not aware of any US institute doing this,” says Lauran Qualkenbush, director of research integrity at Northwestern University in Chicago, Illinois, and president of the US Association of Research Integrity Officers.

And some researchers disapprove. “The moment an institution needs to constantly question the moral integrity of its scientists by double-checking submitted figures, the leadership should resign,” says Giulio Superti-Furga, director of the Research Center for Molecular Medicine in Vienna.

But amid rising concern about the quality and reproducibility of research, particularly in the biomedical sciences, a handful of European institutions have told Nature that they have now hired external companies or dedicated in-house experts to check research manuscripts. The institutions say the cost of the endeavour is worthwhile, not only for the immediate benefit of the checks, but also because it can help them to spot areas in which their scientists need extra training.

The Leibniz Institute on Aging

The Leibniz Institute on Aging, also known as the Fritz Lippmann Institute (FLI), in Jena, Germany.Credit: Jörg Hempel, Aachen

Scientists at the FLI and other institutions see the extra layer of checks as protective, not intrusive. “Because of the manuscript check, I sleep at night,” says one FLI group leader, Björn von Eyss. “I had started to worry about whether I had done something wrong in my papers, maybe missed a label: a mistake can become misconduct,” adds Lilia Espada, a postdoctoral researcher at the centre. “Now that we submit to external checking, I have more confidence.”

Science under scrutiny

Across the research world, there is growing suspicion about sloppiness and outright misconduct in the scientific literature. The number of retractions of research papers has risen to around 1,400 a year, compared with about 40 at the turn of the millennium, notes Ivan Oransky, a journalist in New York City who co-founded the website Retraction Watch, which monitors and reports on retractions.

In 2016, Elisabeth Bik, a microbiologist then at Stanford University in California, reported that around 4% of more than 20,000 biomedical papers she had examined contained inappropriately duplicated images. (Bik is now a full-time research-integrity consultant.) And last year, Bucci reported that about 6% of a sample of 1,364 papers he had looked at contained at least one instance of image manipulation.

Increasingly, fraud-busters are starting to hunt down manipulated images in published papers and flag them widely. Rudolph’s work is an example: the faults were exposed by an external whistle-blower, who sent the findings to Rudolph, then to the DFG, Germany’s main national funding agency, and to its independent Ombudsman Commission. The Leibniz Association has declared a zero-tolerance approach, and young scientists at the FLI say they feel under pressure. Some have told Nature privately that they are worried because of the way in which even unintended errors in papers are flagged publicly online. It can be easy to make a mistake when handling massive and complex biological data sets, they say — and they fear their papers might be publicly picked apart, derailing their careers before they get started.

Matthias Görlach

Matthias Görlach helped to set up a manuscript vetting system at the FLI in Jena, Germany.Credit: FLI/Anna Schroll

In this atmosphere, the idea of the sort of pre-submission screen that Bucci’s company was offering appealed to Görlach. Bucci had been drawn into the world of research integrity after founding an image-search company called BioDigitalValley in Pont-Saint-Martin, Italy, in 2008 that aimed to sell a service to biomedical scientists who wanted all images relevant to a particular tissue or disease extracted from the literature. Bucci had first made a giant database of accessible biomedical papers and cleared it of retracted articles. He then checked the images in all publications by the authors of those retracted papers. He found serious problems in the work of many of them, particularly that of Alfredo Fusco, a then-prominent cancer researcher at the University of Naples Federico II. Fusco has now had 24 papers retracted and 10 corrected. The affair, which implicated scientists in Fusco’s network at other institutes in Italy and beyond, sent shock waves through the scientific community. Bucci was so disturbed by what he saw that he switched career path, founding Resis, to try to do something about it.

Restoring reputation

After Görlach contacted him, Bucci gave the FLI’s group leaders a presentation of his work. His company’s proprietary software scans images in a manuscript for duplication or unlikely composition, he told them. Resis has just two employees, but brings in consultants for particular contracts. In late 2017, FLI group leaders sent Bucci some sample papers and theses to check — and were impressed by the results. He picked up some small errors they hadn’t spotted. The institute signed a contract with Resis to analyse the images in all papers, to do random checks on statistics and also, in doctoral theses, to look for plagiarism. Resis screens all manuscripts within 24 hours of receipt, although if the screen flags problems, further analysis can take up to three more days. The institute budgets up to €50,000 (US$55,000) per year to cover both the service and its handling of the information that Resis supplies.

Enrico Bucci

Enrico Bucci founded a science-integrity firm, Resis, to find errors in research papers.

The new system began in April 2018, and the first results proved its value, says molecular geneticist Alfred Nordheim at Germany’s University of Tübingen, who became the FLI’s interim scientific director when Rudolph stepped down. Resis found no serious problems in the first 40 manuscripts that it analysed for the institute, but it did flag at least one issue in 17 of them, Nordheim says. “Most of these issues were to do with the use of statistics — things like undersampling or use of not fully suited statistical procedures,” he says. “The Resis analysis has been important for us because it allowed us to identify patterns of errors, and act accordingly.” Now, for example, the institute has introduced mandatory statistical workshops for all of its scientists.

FLI researchers see the system as a positive step that is helping to protect them from error. Rudolph himself says that had the checking system had been in place earlier, he would have caught the problems in his papers. (Five have been corrected, one remains under discussion at a journal, and in two cases, journal editors decided no correction was needed, he says.) Rudolph remains a lab leader at the FLI, but his group has now shrunk to seven scientists, half the size it was before the scandal broke.

In June this year, Marco Foiani, the scientific director of IFOM, a molecular oncology research institute in Milan, Italy, learnt about the initiative during a meeting of the FLI’s international scientific advisory board, of which he is a member. It struck an immediate chord with him: IFOM was itself reeling from research misconduct investigations involving a former director, Pier Paolo Di Fiore, who had co-authored some papers with Fusco that have been retracted. Di Fiore says he agrees with the retractions, but wasn’t involved in putting the figures together for the papers. IFOM had introduced electronic notebooks and other measures to promote good scientific practice, and Foiani decided to add on external checking, also using Resis. “It is very important for our image as an institute to get back on track,” says Foiani.

As at the FLI, young researchers at IFOM welcome the screens. “Having a research scandal can affect the credibility of the whole institute,” says Ylli Doksani, one of IFOM’s 24 research group leaders. “We are mostly funded by a charity, and I am happy if the institute does whatever is needed to maintain trust and show we take integrity issues very seriously.”

Other organizations have decided to do publication checks internally. After the Beatson Institute in Glasgow, UK, had to deal with a retraction in 2012, it hired a dedicated integrity offer, former molecular biologist Catherine Winchester, to check all papers destined for publication by eye. “It took only a short time for the more junior scientists to shed their fear that they were being policed, but there was immediate buy-in from senior PIs,” she says. “Now everyone is really grateful for the service.”

The cost of checks

Some research organizations rule out external checks for themselves. The president of Germany’s Max Planck Society, Martin Stratmann, says that the society — which runs 78 elite research institutes — does not need to commission outside checkers because research directors themselves have the mandate and responsibility to check every paper before it goes out. Some institutes Nature talked to for this story were unwilling to comment on the topic; others said only that they found it interesting. “We will monitor the process and discuss with our faculty,” says Bruce Stillmann, director of the Cold Spring Harbor Laboratory in New York.

Nor do all institutes that have been hit by a scandal see the need for screening. In 2012, the DFG judged that Silvia Bulfone-Paus, a senior scientist at another Leibniz institute, the Research Center Borstel, had failed in her supervisory duties after data manipulation was discovered in more than a dozen of her papers. Centre director Stefan Ehlers doesn’t think that paying for independent checks is the right way to approach these problems: rather, he says, it’s important to foster “a culture of trust and fearlessness to report mistakes and to discuss questionable data”.

And pre-submission checks wouldn’t stop all types of fraud, adds Shinya Yamanaka, a Nobel laureate and director of an institute that has recently experienced such a case, the Center for iPS Cell Research and Application at Kyoto University in Japan. There, in 2018, stem-cell researcher Kohei Yamamizu was found guilty of fabricating and falsifying images in a high-profile paper in Stem Cell Reports. Yamanaka implemented measures such as electronic notebooks and mandatory storing of all experimental data — but did not opt for pre-submission checks, a method that “does not investigate whether experiments were truly carried out and recorded appropriately”, he told Nature in an e-mail.

Still other institutions say that the checks are beyond their budget. The Italian National Research Council (CNR), which runs 102 research institutes, would like to offer a full — but voluntary — screening service to its institutes, but says it can’t afford to. After the Fusco affair, it established a technical unit to use licenced Resis image-analysis software to check published papers. The unit provided formal comments on the report of the University of Naples’ investigation into Fusco’s papers, and now focuses on allegations of misconduct by CNR researchers. If an allegation surfaces, the unit examines all the papers the institute in question has published over the previous five years. Any manipulated images are recorded in a growing database.

Last year, the CNR unit started preventive work on a modest scale: it has so far done a few pre-submission checks, responding to individual CNR researchers who were concerned, for example, about joining in as co-authors on particular manuscripts. “Prevention is the critical step,” says Cinzia Caporale, who leads the organization’s research integrity activities from its headquarters in Rome. After the scandals in Italy, “scientists don’t always trust their colleagues any more”, she says. Caporale thinks the CNR’s work has increased scientists’ awareness: the council’s database suggests that its scientists are already publishing fewer inappropriate images, she says. A higher budget would allow more systematic pre-checking, but Caporale says there is no prospect of that right now.

Not many image-checking services have the capacity to rapidly screen a high volume of papers, as an institute — or a journal — might require. But some say they are interested. Sheridan, a large publishing-services firm in Hunt Valley, Maryland, already offers image forensics to journals, and told Nature that it is “open to the idea” of setting up such a service for institutions. Mike Rossner, who runs a small consultancy firm called Image Data Integrity in San Francisco, California, says he’d prefer to train someone from an institution’s research integrity office to do screening using his own manual system. Rossner is known for his expertise in spotting problems in papers by eye: as a former managing editor at the Journal of Cell Biology, he introduced checks of images in all papers accepted for publication — making the journal the first major life-sciences publication to institute the practice.

Nurturing trust?

Rossner thinks that investing in pre-checking could save money in the long run. “Prophylactic screening makes financial sense, because any case brought against an institution for publishing misleading data could cost an institute even more in legal fees,” he says. It might even become a selling-point for institutes, suggests Caporale. “Being able, for instance, to tell journal editors that a paper has been independently checked may nurture trust,” she says.

Even if that were true, it wouldn’t relieve journals of the responsibility to do their own checking, says Bernd Pulverer, chief editor of the EMBO Journal in Heidelberg, Germany. His journal checks images in all papers before they are accepted, and generally sees problems in around one in five manuscripts, a proportion that has not changed since the journal began the checks ten years ago, he says. Only a tiny minority (0.5%) of these involve outright fraud. Other journals now regularly check images too, although some (including Nature) do spot checks, not systematic ones.

Journals don’t have the same jurisdiction as a scientist’s employer does to investigate problems, so the institute has an important role in ensuring quality, Pulverer adds. “But it is important for the employer not to start over-policing, because that can backfire,” he says.

The FLI plans to continue working with Resis and thinks that the checks will make the institute more attractive in competing for the best scientists, says Nordheim. In June 2018, it reported its experience to a Leibniz Association leadership meeting on good scientific practice. Matthias Kleiner, the association’s president, was impressed. He is planning to test the possibility of introducing a certification system for good scientific practice for the association’s institutes. It’s possible that pre-submission checks could be an optional item on these certificates. For some Leibniz institutes, he adds, it could be a way “to protect scientists from being in danger of scientific misconduct”.

Nature 575, 430-433 (2019)

Updates & Corrections

  • Correction 26 November 2019: An earlier version of this story erroneously stated that Resis checked all master’s theses.

Nature Briefing

An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday.