Peer review is slowly opening up. It is no longer exclusive of journals and the experts that they pre-select; and the single-blind process, although predominant, is not the only option. Comments on preprints, post-publication discussion and debate in social media, as well as double-blind, partially transparent and fully open processes, are becoming increasingly accepted and used.

Credit: momentimages/Getty Images

The vast majority of papers do not undergo post-publication peer review, however, and those that do are often published in high-ranked journals. After all, wider scrutiny depends on attention and interest, and hence those papers that attract more of both tend to accumulate a higher share of the discussion, before publication (when a preprint exists) and after. Nevertheless, online communication, search and discovery tools increase the democratization of the assessment of the soundness and reliability of scientific knowledge, and this ought to be nurtured and celebrated.

As with most human activities, peer review is imperfect; from time to time it fails to prevent unsound, preliminary or unclear manuscripts from being published. The recruited reviewers may lack some of the expertise needed, they may not be able to dedicate enough time to scrutinize the work fully, might burn out after an excessive number of revisions, or may have competing interests or be biased for or against the work. These shortcomings can sometimes compound with sloppy data presentation, incomplete methods, insufficiently clear discussion or hyped conclusions. And editors may apply bad judgement when selecting reviewers, taking decisions, and managing the communication between authors and reviewers and among reviewers. Yet, despite the many potential pitfalls, most papers undergo a fine peer review; otherwise, the overall advancement of science and technology would not have progressed at a rapid pace. Some have argued that the reliability of published research decreases with increasing journal rank (as measured by the flawed yet useful impact factor); this is unlikely to be the case for most scientific fields, because of undeniable progress in the accumulation of knowledge, the wider exposure and influence of papers published in well-regarded journals, and the typically more thorough manuscript selection and editing processes run by their dedicated editors.

The recent increased emphasis on replicability and reproducibility, especially in life sciences and medicine, has motivated many scientists to seek best practices in study design, statistics, methods and documentation, and also prompted journals to demand clear, careful and thorough reporting from their authors. Reporting summaries and checklists, plagiarism checks by publishers, and the scrutiny of discussion, figures, methods, data and code by reviewers and editors, as well as data and code sharing by authors, go a long way towards facilitating the reliability of published research.

Yet reliability in methodology does not ensure replicability or reproducibility. Even the most detailed protocol can fail in the hands of the non-expert, especially when experimental or environmental factors can introduce large variability. Samples can sometimes be flimsy or faulty, the reliability of laboratory equipment may differ in substantial yet imperceptible ways, and custom software may not be robust against compiler options, operating systems, hardware environments or software frameworks. Hence, although extremely important for scientific progress, reproducing others’ work can be difficult and laborious.

When peer review fails to ensure soundness, and further analysis determines serious flaws in a paper, journals should issue a correction or retraction of the work. Yet most queries and disputes on published work are solved via informal dialogue and do not require corrections of the literature. When this is not the case, journals should allow for post-publication commentary on their own published papers. To encourage post-publication debate and allow for due process in the peer review of scientifically relevant and timely comments and clarifications, this journal and the rest of Nature research journals are now encouraging that such commentary — which, if published, will be in the form of ‘Matters Arising’ — be posted in relevant preprint servers, as supported by our policies. Matters Arising is a new content type specific for challenges and clarifications to published work by the journal. The Correspondence article type, long used for refutations in Nature journals, will remain available for broader commentary not linked to a particular primary research paper.

As with pre-publication peer review, both trust (in that the authors did what they claim to have done and in their integrity) and scepticism (in the soundness and reliability of the data and in their interpretation) are necessary in critical post-publication debate. When the balance of trust and scepticism is heavily skewed towards the former or the latter, post-publication peer review can be unfruitful at best. In particular, hasty debate in social media and partisan blogs can spread rushed and groundless accusations of scientific misconduct that damage scientific reputations and make scientists wary of online public discussions. Some post-publication review sites have contributed to this, raising the voice of anonymous watchdogs. Yet such sites also make it easier for necessary and inclusive debate when peer review does not live up to high standards, as in the highly publicized STAP case.

Fully anonymous commenting is a double-edged sword. It avoids the younger or less powerful jeopardizing their careers, but lowers the threshold for civil discussions, careful language and grounded claims. The traditional single-blind peer review — and double-blind peer review when feasible — provide a tried-and-trusted middle ground: the editor (of a curated post-publication site or of a traditional journal) assures the anonymity of reviewers, supervises the discussion and vets the contributions. Although not a fail-proof or fast process (editors are not impervious to biases, and managing debate and allowing for due process and checks requires effort and patience), expert mediation by an independent party remains the best way to achieve the appropriate balance of trust and scepticism that makes scientific debate healthy and productive.