As early-career researchers (ECRs), we have experienced at first hand the challenges and consequences of navigating the peer-review system. By pursuing careers in academia, we agreed to subject ourselves to the scrutiny of fellow experts in our field, who evaluate the quality of our research and grant proposals. The decisions of these few, often-anonymous individuals have far-reaching effects on career progression, funding opportunities and reputations, both inside and outside the scientific community.
The peer-review process is often touted as a key arbiter of academic rigour — a means of ensuring that sub-par research doesn’t slip through the net. However, in recent years, its usefulness and relevance have been hotly debated. Supporters argue that peer review identifies errors, biases and misinterpretations. Opponents counter that the current process is based on repeated cycles of submission and rejection, which is time-consuming, expensive, lacks evidence of its efficacy and might hinder scientific progress1. The confidential nature of peer review also makes it challenging to evaluate and reform the process.
Papers are increasing in volume, complexity and interdisciplinarity, which means that the pool of available and qualified reviewers is becoming ever more shallow. It can often feel as if the current peer-review model is reaching its limits. Unsurprisingly, recent years have seen the emergence of alternative publishing approaches that aim to address some of these limitations, such as the peer-review platform Review Commons2 and the switch last year in the journal eLife’s editorial process and publishing model3.
Rethinking peer review
The COVID-19 pandemic triggered a seismic shift in the way research studies are shared. During the pandemic, immunologists took advantage of preprint servers to rapidly communicate their research findings. This wave of non-peer-reviewed studies presented an opportunity and a necessity to explore alternative ways to conduct peer review. For this reason, we — ECRs at the Icahn School of Medicine at Mount Sinai in New York City and the University of Oxford, UK — independently began to critically read and analyse many COVID-19-related preprints. We made our assessments publicly available in the comment sections under each preprint on the bioRxiv and medRxiv platforms to help researchers to contextualize this unprecedented flow of scientific data.
The enthusiasm for this community-centred peer-review approach encouraged us to expand it after the pandemic. Together, we established a cross-institutional journal club to evaluate emerging immunology studies that are posted before peer review. This led to the creation of the Preprint Club in 2020, and two other institutions, the Karolinska Institute in Stockholm and the University of Toronto in Canada, subsequently joined.
The Preprint Club builds on the fact that most academic institutions host journal clubs at which ECRs already spend time and energy reading and commenting on the work of others. Unfortunately, these efforts are often limited to discussing peer-reviewed manuscripts, and the analyses are rarely shared beyond the journal clubs. We harnessed this lost opportunity to repurpose our institutional journal clubs to focus on preprints that have not yet been peer reviewed, and to share our analyses publicly.
A case for community-based preprint clubs
We had three core ideas for the Preprint Club. First, the Preprint Club should have a cross-institutional format, ideally involving participants from different countries to broaden expertise and reduce bias. Our members meet online once a week, and two ECRs from different participating institutions present one preprint each that they discuss and evaluate with the Preprint Club community.
Second, it should provide a training ground for ECRs to develop their peer-review skills. These are rarely taught as part of a curriculum, leading to the exclusion of ECRs from the peer-reviewer pool. Thus, each presenter is encouraged to write a short, standardized review of the preprint they presented, which is then shared on our website (www.preprintclub.com).
Finally, the process should be rewarding. Our members can choose to sign their reviews and be credited publicly for them. We also established a collaboration with the editors at Nature Reviews Immunology, who took this opportunity to launch an article type called Preprint Watch. In these, our members are invited, together with a faculty member of their choice, to write highlights about the preprints that they collectively deem most promising.
Two years after starting the Preprint Club, we released a blueprint detailing its methodology, and performed an assessment of its efficacy4. By tracking the outcome of preprints, we concluded that our system can successfully identify the most impactful studies before they are published in peer-reviewed journals. By collecting feedback from our members and from the authors of evaluated preprints, we were particularly happy to find that our approach shows training benefits for ECRs — the majority reported a higher confidence in their peer-reviewing abilities after participating in the Preprint Club. Furthermore, the preprint authors themselves deemed our reviews to be helpful, fair and accurate5.
Encouraged by our own experience, we think that our approach is a valuable complement to the conventional peer-review process. Repurposing existing journal clubs into cross-institutional preprint clubs is easy to replicate in academic institutions and for disciplines beyond immunology. Preprint reviews have the potential to improve research culture, and the journal-independent approach allows peer review to focus on the quality of research rather than on a subjective fit for a given journal.
We hope that our experience inspires similar initiatives at other institutions and, ultimately, that it contributes to making science more democratic, accessible and impactful.