Credit: V. WEHRMAN/CORBIS

Almost all peer reviewers get worse, not better, over time, suggests a study presented on 10 September at the Sixth International Congress on Peer Review and Biomedical Publication in Vancouver, Canada.

Michael Callaham, editor-in-chief of the Annals of Emergency Medicine in San Francisco, California, analysed the scores that editors at the journal had given more than 1,400 reviewers between 1994 and 2008. The journal routinely has its editors rate reviews on a scale of one to five, with one being unsatisfactory and five being exceptional. Ratings are based on whether the review contains constructive, professional comments on study design, writing and interpretation of results, providing useful context for the editor in deciding whether to accept the paper.

The average score stayed at roughly 3.6 throughout the entire period. The most surprising result, however, was how individual reviewers' scores changed over time: 93% of them went down, which was balanced by fresh young reviewers coming on board and keeping the average score up. The average decline was 0.04 points per year.

“I was hoping some would get better, and I could home in on them. But there weren't enough to study,” says Callaham. Less than 1% improved at any significant rate, and even then it would take 25 years for the improvement to become valuable to the journal, he says.

Others are not so convinced that older reviewers aren't wiser. “This is a quantitative review, which is fine, but maybe a qualitative study would show something different,” says Paul Hébert, editor of the Canadian Medical Association Journal in Ottawa. A thorough review might score highly on the Annals scale, whereas a less thorough but more insightful review might not, he says. “When you're young you spend more time on it and write better reports. But I don't want a young person on a panel when making a multi-million-dollar decision.”

Callaham agrees that a select few senior advisers are always very useful. But from his own observation, older reviewers do tend to cut corners. He notes that psychological research shows that experts in complex tasks typically reach a plateau and then stay there or slowly deteriorate. Perhaps by the time researchers are asked to review a paper at his journal, they are already experts. He suspects the same would hold true for journals across all fields.

Callaham also found that a mentoring programme at the Annals, in which new reviewers are paired up with senior ones, has only a temporary effect. Young reviewers assigned a mentor typically scored half a point better than non-mentored colleagues, but when the mentor's watchful eye disappeared after a year or so, this advantage evaporated. And the Annals dropped a separate peer-review training course some time ago, he notes, because although people loved it, it wasn't helping their scores. “It's kind of depressing,” he says.