More carrot, less stick: how to make research assessments fairer

How do you measure something such as kindness, teamwork or quality of mentorship? UK researchers now have an opportunity to say what they think. The country’s research-funding organizations are planning changes to a nearly 40-year-old system for assessing research so that these qualities are measured and rewarded — and researchers have until 6 October to provide feedback.

The assessments that form part of the UK Research Excellence Framework take place around every seven years. Higher-education institutions are judged on the quality and impact of their research outputs, as well as the quality of the institutional environment.

The results show the proportions of an institution’s submitted research projects that reach a standard from 1* (research that is recognized nationally to be original, significant and rigorous) to 4* (world-leading on those criteria). In the latest assessment, in 2021, around 180,000 research outputs from 76,000 researchers at 157 institutions were reviewed by expert panels. More than 80% of submitted research was either 3* or 4* quality.

The REF is a big deal. The results are used to allocate at least £2 billion (US$2.6 billion) in government funding every year, almost all of which rewards institutions for their 4* and 3* research. In a tightening economic climate, the REF provides evidence to both the government and taxpayers that hard-earned money is going to the brightest and the best. For these reasons, the REF is valued more highly by people in managerial positions than it is by researchers, many of whom say that the process creates stress and anxiety, encourages game-playing and discourages novelty.

Support Europe’s bold vision for responsible research assessment

To prepare for the REF, university leaders search for what is known as ‘REF-able’ work — research that is most likely to achieve 3* and 4* results. This could mean that 1* or 2* work is less likely to be submitted for assessment even if it is highly relevant to a local issue and done by a research group whose members work well together as a team. That decreases the visibility of these researchers and their work. And less visibility in the REF means less chance of funding.

This is what the research-funding bodies for each of the four UK regions want to change for the next REF exercise in 2028, in line with global trends towards more-responsible research assessment and evaluation. The idea is to take the pressure off individuals, so that the exercise is not used to judge their performance. By extension, universities should not use REF compliance to manage the performance of their staff.

Funding bodies are keen to expand the definition of excellence. They also want all university staff members to be eligible for the assessment, rather than just people directly doing the research, as has been the case so far. In future, contributions to scholarship could also be included from an entry-level technician, or the person at reception who answers the phone. Peer-review panels will also take a more inclusive approach to assessing the quality of outputs. The assessment will take into account not only research results reported in journals, but also the replication studies, review articles, software, policy reports and the myriad other ways in which academics record their work and its impact. Moreover, the weighting attached to outputs (renamed ‘contribution to knowledge and understanding’) falls from 60% of the overall score to 50%. The environment weighting (renamed ‘people, culture and environment’) increases from 15% to 25%. Impact (renamed ‘engagement and impact’) stays the same at 25%.

External advice

Many of the proposed changes have been informed by the results of several surveys of researchers and advice from an international expert advisory group chaired by Peter Gluckman, New Zealand’s former chief scientific adviser. One of the advisory group’s recommendations, Gluckman says, is for the REF to value research ‘processes’ as much as final results. It means that more journals will need to offer publishing formats, such as Registered Reports, that don’t simply focus on a final outcome.

Peer review will remain the main method for judging quality. But other measures are being considered for evaluating aspects such as data on equality and diversity, with metrics for career progression as proxies for a supportive research culture. There clearly is value in monitoring these data, but it is debatable whether they are a true reflection of a supportive research culture, or just a box-ticking exercise.

Research assessment exercises are necessary — but we need to learn to do them better

Similarly, the ambition that the REF should not be used as a performance-management tool is reassuring. But it’s not clear how the changes will take pressure off individuals. Researchers’ identities will still need to be known to those doing the assessments. It might be better to explore incentives such as rewarding researchers who submit their work voluntarily. If funding bodies genuinely want to reward a diversity of scholarship, most of the funding should not be going to 4* and 3* research.

Research is as much a craft as it is a science. It takes time to acquire and hone skills. Being a researcher involves continuous learning and improvement. Publishing articles in well-known journals and generating and curating data are not the only signs of scholarship — and being a good mentor to a PhD cohort involves more than passing a set number of graduate students. Research-assessment systems will always be less-than-perfect measures of the totality of the work that happens at universities. The planned reform of the REF is a welcome start. It is an opportunity to make assessment better, deeper and more inclusive, and, ideally, without doing any harm.