Towards the end of July this year, a lot of organic chemists were getting hot and bothered, and the cause certainly wasn't the weather; it was a paper reporting the oxidation of benzylic alcohols using sodium hydride (NaH) — the well-known base and reducing agent1.

This surprising result was picked up by and commented on in traditional chemistry publications, but perhaps the most unconventional assessment came from the blogosphere. Medicinal chemist Paul Docherty repeated one of the experiments reported in the paper and posted his observations live on his blog, 'Totally Synthetic'2. Docherty concluded (like many others in the community) that although oxidation did occur, it was more likely caused by the presence of adventitious oxygen or by another oxidant present as an impurity. Several comments, made both on his blog and in other venues, suggest that similar reactions had been reported some time ago in the chemistry literature3.

The existence of these prior reports might — perhaps should — have been caught by traditional peer review, but what of the other more fundamental problems with the chemistry itself, and what can the community do, if anything, to prevent similar situations arising? A recent report4 by the UK charity Sense About Science concludes that the majority (69%) of researchers are satisfied with the peer-review process, but that a large proportion of them think it can be improved — so what options are there?

Any system necessarily relies on the reviewers trusting the observations that are presented in the manuscripts they evaluate and, save for a few notorious examples, most results are presented in good faith, as seems to be the case with the NaH paper. This certainly isn't the first time that an impurity has ultimately been found to be the real cause of a chemical reaction — other recent examples include iron(III) catalysis5 and the metal-free Suzuki coupling6. It is worth noting, however, that the immediate and intense scrutiny applied in the NaH case may not have transpired had the reaction looked more feasible on paper. It is fair to say that oxidation mediated by a well-known reducing agent stood out like a sore thumb and was asking to be investigated further!

Docherty and his fellow bloggers should be commended for their contribution to the chemistry community, but it is unlikely that such a process (or something similar) could become a routine feature of peer-review. Is it reasonable to expect referees to repeat experiments, synthetic or otherwise, reported in a manuscript as a matter of course? In almost all circumstances, this is undoubtedly a step too far; the experimental priorities of researchers understandably lie elsewhere, with their resources (time, materials, equipment, workforce and so on) rightly dedicated to their own projects. For example, it is doubtful that Docherty would have been given leeway by his employers to repeat a total synthesis. Of course, some publications — such as Organic Syntheses — include experimental verification as part of the review process, but this approach is only worthwhile for compounds that are frequently in demand by a large group of researchers.

A further concern is the provenance and legitimacy of comments provided in some blog settings — claims made in such forums are not subject to the same level of scrutiny as the original research paper itself, or indeed the referee reports associated with it. So what other options are there for community input in a similar vein? There are advocates for open-notebook science7, where individual results are immediately posted online. However, although this approach has been shown to be very successful in some specific cases it does require all results to be presented openly. It's also hard to imagine more sensitive results being publicized like this — at least not without a sea-change in the way research funding is awarded. Open-notebook science may not be attractive to a researcher running a small group and trying to compete with the 'big guns' of the subject who have access to far greater resources.

Open peer-review experiments have generally not been very successful8, and the main reason seems to be that reviewers are less likely to make forthright comments in an open forum — particularly about the work of the aforementioned big guns. Double-blind peer review is another option, but one must consider the difficulties of effectively hiding the identity of authors in smaller fields from other experts — especially when many authors regularly cite and discuss their previous work in papers.

The Royal Society of Chemistry's Dalton and Faraday discussion meetings provide a unique mix of traditional peer review coupled with both comment (by peers) and responses from the authors, but require members of a particular research community to assemble at a conference. It is in some ways similar to the grant proposal review process at, for example, the US National Institutes of Health9. However, such a process is clearly not a viable option for every one of the vast number of papers submitted for publication. The journal Atmospheric Chemistry and Physics uses a system10 in which, after initial assessment by an associate editor, manuscripts are posted online for comment. After referee reports are received, these are also posted online with the manuscript along with author rebuttals. If eventually accepted, a paper is formally published in the journal, whereas those that are not remain available (and citable) as online 'discussions'. This differs from the preprint servers Nature Precedings and arXiv because there is an initial assessment of the suitability of the work (based on more than just scope).

Perhaps a hybrid system could be the solution. Traditional peer review, and a decision to publish, could be followed by a fixed period in which any interested party could post questions or comments and the authors are given the opportunity to respond — all moderated by an editor — before a final version of the article (including comments and responses) is preserved for the record. This would again require a large change in the habits of the community — authors, reviewers and publishers — and previous experiments with commenting on published papers have been far from conclusive. As it stands, to paraphrase Winston Churchill, peer review is the worst form of assessing science, except all the others that have been tried.