COVID is amplifying the inadequacy of research-evaluation processes

Researchers have influenced the course of the coronavirus pandemic, by revealing the biology of the virus and driving the development of vaccines. But the pandemic has also irrevocably altered the lives of many researchers.

Last year, a Nature poll of postdoctoral scientists revealed a generation concerned about shuttered projects, insufficient funding to cover the duration of the pandemic, and a fall in employment opportunities. There is also growing evidence that the fallout from the pandemic is worsening existing disparities — between genders, for example, which is already a significant problem for science. Female researchers with children seem to have been particularly badly affected. According to one survey of senior researchers carried out last July, women with responsibilities for looking after children experienced a substantial decline in the time they could devote to research¹.

And although in the past year there has been a noticeable spike in submissions to journals across all subjects, not everyone has been able to increase their output. A study of publishing patterns during the pandemic found that women, particularly young women, submitted proportionally fewer manuscripts than men during the early months of the outbreak². During March and April 2020, the number of male authors posting to the preprint servers bioRxiv and arXiv grew faster than did the number of female authors, compared with the same period in 2019. The proportion of women posting as first authors stayed the same, however.

Postdocs need urgent financial support amid COVID conditions

Some universities are beginning to offer financial support to researchers suffering because of the pandemic, but their funds are being stretched. In the United States, for example, half of a US$400,000 emergency equity fund established by Indiana University had been allocated within six weeks of its launch. Grants covering the costs of care-giving and tutoring were among the most popular.

But universities cannot be expected to shoulder this responsibility alone — research funders, too, have a part to play in addressing inequalities. One factor that risks exacerbating existing inequalities are the systems by which researchers and their institutions are assessed. As the pandemic’s toll becomes clear, funders, especially those that run national research-evaluation systems, must take care not to penalize those whose work has suffered.

In the United Kingdom, the pandemic hit just as researchers were preparing to submit their research outputs to the Research Excellence Framework (formerly the Research Assessment Exercise), which periodically assesses the work of academics. The nation’s four higher-education funding bodies, which organize the assessment, agreed to extend the submission deadline from November 2020 to March 2021. They also made adjustments to submission criteria. However, the 34 panels of peer reviewers — who will be assessing disciplines from art history to zoology — have not yet been sent guidance on the kinds of adjustment they will need to make when taking into account the impact of COVID-19, despite being due to start their work in May. This needs to be sent without delay.

The pandemic’s impact will be felt for years to come, however, which means it will also affect subsequent research assessments. As they prepare for this, funders must consider how they can more-equitably account for the harm caused to research and careers in 2020. A year of lost or deferred science could be much more damaging to those early in their careers than to their senior colleagues. It is important that funders learn lessons from the pandemic and allow these to inform the construction of future evaluation systems.

Are women publishing less during the pandemic? Here’s what the data say

Evaluation is designed, on balance, to meet funders’ needs. At the same time, funders do take care to ensure that evaluation criteria are not overly complex, because this adds to the burden on institutions. But broad-brush criteria also have implications for equality and diversity. If all researchers are being assessed by roughly the same criteria, those who are already disadvantaged are more likely to be outperformed by those who have had more advantages, which will further entrench inequality.

Researchers and research managers have been studying the comparative strengths and weaknesses of different systems for many years³ — work that provides insights into how to make evaluation more equitable. But more-detailed work is needed to understand the extent to which existing measures of evaluation — such as journal output or research income — are an accurate reflection of progress in science. And funders should consider designing systems that credit equally important aspects of the research process, such as mentorship, building and working in teams, and providing opportunities for researchers from under-represented groups.

The pandemic has showcased the stunning gains that science and scientists can achieve for society. At the same time, it has thrust some of academic research’s uncomfortable truths back into the spotlight — notably, the continuing, and now seemingly widening, gender gap and the potential neglect of a generation of researchers. Funders and those responsible for evaluation policy must reset how research is valued, measured and assessed. But, ultimately, inequalities need to be tackled at their roots and disparities erased, so that no researcher is at a disadvantage.