Western blots — lab tests made on proteins in gels — that had inconsistent data were found in a number of published papers discussed by Paul Brookes on his now-discontinued blog, Science Fraud. Credit: Carol & Mike Werner/Visuals Unlimited/SPL

Airing concerns about data in research papers publicly is more likely to yield action than contacting authorities in private, says Paul Brookes, a whistle-blowing scientist. That is not just his opinion — it is the conclusion of a study in which Brookes, a researcher at the University of Rochester Medical Center in New York, compared public and private methods of discussing suspected flaws in life-sciences papers. Brookes found that online publicity is directly linked to a higher rate of retractions and corrections. The study is published today in PeerJ1.

Brookes ran the blog Science Fraud from July 2012 to January 2013, before closing it down in response to threats of legal action. For the PeerJ study, Brookes compared the outcomes of two sets of papers — 274 whose alleged data problems he chronicled on his blog, and 223 that he was e-mailed about but did not post before he shut the site down. Those private e-mails, he says, were also copied to the relevant journals, funding agencies and authors’ research institutions, so authorities would also have had the opportunity to review the allegations.

Of the 274 papers Brookes blogged about, 16 were retracted and 47 corrected by December 2013, he reports, meaning that action was taken in 23% of the cases. But of the 223 unpublicized papers, only two were retracted and five corrected — a rate of 3%.

Brookes argues that there were no substantial differences between the two sets of papers — they had similar data problems (most involving inconsistencies in data from western blots, which are used to detect the presence of proteins), and were published in journals having similar degrees of prestige.

Paul Brookes Credit: URMC

“Internet publicity is associated with a higher rate of retraction,” he says. “It’s difficult, obviously, not to draw the conclusion that the rate at which action is taken is a direct consequence of this public discussion.”

Provocative publicity

“This is a provocative study,” says Ferric Fang, a microbiologist at the University of Washington in Seattle and editor-in-chief of the journal Infection and Immunity, who studies retractions and irreproducible research. “The idea is that the current system, which relies on the integrity of journal editors, institutions and authors to do the right thing, doesn’t always work,” Fang says. And this is not a surprise, he adds: “Sometimes authors, and institutions, and editors, drag their heels.”

There are some limits to the study, as Brookes acknowledges. It is hard to know whether the unpublicized allegations, coming as the site grew more popular, were as well substantiated as the ones Brookes blogged about early on. The privately discussed papers were about three years older, on average, than the public ones, and Brookes speculates that there might have been less pressure to correct them, as the US Office of Research Integrity has a six-year statute of limitations for investigating allegations of misconduct. And papers in the private set might catch up with the public set in time, although this seems unlikely, he says.

It will be not be possible to independently check Brookes’ work — because of the sensitivity of the issue, he is not releasing a full data set of the private allegations. “It’s a real limitation,” says Fang, who adds that the same problem beset two recent, widely cited studies in which scientists at pharmaceutical firms said that much high-profile academic research could not be replicated2,3. “I don’t have any reason to question the interpretation, though it would be more persuasive to see it with one’s own eyes,” Fang says.

Still, the fact that seven-fold more of the publicly questioned papers were corrected or retracted should not be ignored, Brookes says. But he does not think would-be whistle-blowers should go public immediately. “A better strategy may be to start with private notifications, then go public,” he says, not least because, in cases where misconduct is involved, a public discussion might tip off the authors before an investigation can begin. Fang agrees that discussion should begin in private, but for a different reason — to protect the privacy of those involved until the truth is uncovered.

In his paper, Brookes also describes several instances in which journal editors did not deal satisfactorily with his queries, although he does not name the journals or the editors. “A wider point from what Brookes is saying is that journals and institutions are no longer operating in an environment where everyone just trusts them,” says Fang. “This should be a wake-up call to journals that people are not going to be satisfied by stonewalling. The public court of last resort is going to address these issues whether journals like it or not.”