A rare analysis of open peer review — in which reviews are posted alongside published papers — has overturned some common conceptions about the practice: notably, that it doesn’t put the reviewers off or affect their recommendations on whether to accept a paper.
The analysis, published on 18 January in Nature Communications1, also indicates that open reviewers mostly prefer to remain anonymous, and that they don’t take any longer to complete reviews than in the conventional process.
“I think the case for publishing peer reviews is quite clear in terms of transparency and accountability,” says Tony Ross-Hellauer, an information scientist at the Graz University of Technology in Austria who conducted a 2017 survey about open peer review. “In terms of clearing away some doubts about publishing peer reviews, I think this study is really good news.”
Some journals — including F1000Research and PeerJ — have used open peer review to some extent for years.
The study used data from journals published by Elsevier, which said that in light of the results, it would consider whether to use the practice at more of its journals.
Testing the fears
Some people fear that making reports openly available could compromise the peer review process in various ways, for example, by making academics less likely to agree to do reviews. There are also concerns that reports destined for publication might take longer because reviewers might be inclined to be less critical about studies, and therefore more likely to recommend accepting a paper. The study set out to test these concerns.
The authors evaluated more than 18,000 reviews from about 9,000 manuscript submissions to 5 Elsevier journals from 2010 to 2017. For some of this time, these journals posted referees’ reports openly online as part of a pilot project — a rare chance to study what happens to reviews when journals switch their practice. The trial also offered the open reviewers the choice to reveal their identity.
The authors compared the contents of the open reports with a set of traditional, private reports done for the same five journals.
The researchers also compared the likelihood that a scientist would agree to do an open review with data on the same likelihood at five comparable Elsevier journals that didn't offer open review.
The findings surprised the authors. Academics were just as likely to agree to review a paper if their report was going to be made public. And practising open peer review didn’t make a significant difference to the likelihood that a reviewer would recommend a paper for publication.
Only around 8% of referees chose to reveal their identity by signing their reports.
Reviewers typically chose to disclose their names when their feedback on a manuscript was positive, says study co-author Flaminio Squazzoni, a sociologist at the University of Milan in Italy. Squazzoni suggests journals wishing to roll out open peer review might be better off offering anonymity to reviewers.
That view chimes with the results of Ross-Hellauer’s 2017 survey, which polled 3,000 academics worldwide; most said that open peer review should be mainstream but without disclosing reviewer identities.
That makes sense, says Marcia McNutt, president of the US National Academy of Sciences and former editor-in-chief of Science. “Most of the concern I have seen with experiments in reviewing have been with revealing the names of reviewers to authors and others, beyond the editors,” she says. “Journals that have experimented with that option have had disappointing results in reviewer acceptance and quality of the reviews.”
Squazzoni points out that, although the latest study suggests that switching to open peer review doesn’t seem to have any downsides, it doesn’t necessarily improve the peer review process either.
So rather than rushing to make reports open, he thinks the publishing community should focus on providing incentives to encourage academics to participate in peer review in the first place.
A survey of more than 11,000 researchers published last year revealed growing ‘reviewer fatigue’ among researchers, with editors having to invite more reviewers to get each review done.