Marcus Munafò just doesn't believe some of the stuff he sees in journals — especially in psychological sciences, which have come under pressure to prove that studies in the field can be replicated. "Too often my gut feeling is 'surely this can't be right'", says Munafò, who is an experimental psychologist at the University of Bristol, UK.
But when Munafò peer reviews such studies before they are published, he is often instructed not to express such personal opinions — something he finds frustrating. Instead, peer reviewers may be asked to stick to an objective assessment of whether a paper's methods look sound. And this, Munafò thinks, can be damaging: in a contentious field, a flurry of methodologically ‘sound’ results that support one explanation — though not conclusively — may create a self-fuelling illusion of consensus, preventing researchers from accurately assessing scientific truth.
Writing today in Nature1, Munafò and his collaborators assert that they can back up their concerns about peer review using cold mathematical logic. By modelling how scientific knowledge is disseminated through peer review and publication, they suggest that scientists who bring their own judgement to the review process can slow or reverse a misleading rush to consensus after a sequence of one-sided studies in a field. Yet other researchers say that the authors' model is too unrealistic to draw any real-world lessons about peer review.
The model — built by Munafò's colleagues Mike Peacey, a theoretical microeconomist, and In-Uck Park, a researcher in industrial organization — imagines a stream of scientists, each with private thoughts about which of two hypotheses they favour. They reveal that information in a step-by-step process. First, one scientist puts forward her favoured hypothesis in a paper. This is peer-reviewed by a second scientist, and rejected or accepted for publication based on that review. The second scientist updates his private beliefs on the basis of this public history, and puts forward his own manuscript, proffering his new thoughts. That is peer-reviewed by a third scientist, and so on.
As scientists update their beliefs based on peer-review history, they may quickly depart from their initial impressions if they are persuaded by the weight of accepted evidence. This process is called herding, because it evokes the self-reinforcing nature of clustering herds. The concept originates from economic models, says Peacey, in which people make decisions using a stream of information to influence their beliefs.
For the most part, herding is a good thing: if scientists are confident in their beliefs and papers are strong, why not follow the literature? The trouble, however, arises in areas in which the evidence seems to be inconclusive. Here, the model suggests, herding may lead scientists to converge on the wrong consensus, because the early papers give a false impression of what people really think.
If the idealized scientists could all gather together and simply share their thoughts, then everyone in the model would gain an accurate view, Peacey adds. It is because information trickles out paper by paper that false herding may occur.
But how quickly such herding forms depends on the way peer review is conducted, Munafò says. The authors compare what they call a 'subjective' version of peer review, in which reviewers' private thoughts inform their review, to an ‘objective’ version, in which researchers are not allowed to use their private information to influence their decision. The comparison revealed that the extra sharing of information can slow and sometimes reverse herding in contentious fields.
From mathematics to reality
The study “initiates an important discussion that I think people are having in many hand-waving ways”, says Luís Amaral, a physicist who studies complex systems at Northwestern University in Evanston, Illinois. It suggests that objective peer review may sometimes assist an emerging false consensus. “This is one of the fascinating things about models — they make you realise unintended consequences,” he adds. Moreover, the hypothetical researchers are working under ideal conditions: seeking the truth, rather than, for example, trying to publish to advance their careers.
But Andrew Gelman, a statistician at Columbia University in New York City, says that the model does not give much insight into the real world, because science does not really progress paper by paper in a series of opposing hypotheses. “Of course I believe herd behaviour exists. But I don’t see this mathematical model as relevant for characterizing the development of scientific literatures,” he says.
Daniele Fanelli of the University of Edinburgh, UK, who studies research fraud and bias, says that the assumptions are not realistic enough to give useful predictions. In the team's model, each paper gets only one reviewer, each scientist publishes only once, and every person hears about rejections, which in reality would not be made public.
The criticism does not deter Munafò. His team is merely contributing to the debate about peer review by offering a fresh view on its dynamics, he says. He thinks that peer reviewers should be asked subjective questions such as, “to what extent do you believe the results of this paper?”
But many argue the opposite: it is better to avoid subjective peer review based on personal bias and let post-publication peer review and discussions between scientists filter the best from the worst.
Munafò’s study does not mention whether post-publication peer review might help to prevent false consensus in contentious fields, because it doesn’t include that process. But Munafò has strong opinions on the topic: “People talk about putting everything out there and letting the cream rise to the top, but I don’t think they’ve thought through what the sheer volume of material would be,” he says. “There’s a real signal-to-noise problem, and having a stronger gatekeeper function as a check on that flow would be helpful.”
- Journal name: