Nature | News

Major review calls time on 'gaming' in UK research assessment

Report tries to curb bad behaviours incentivized by academic audit system.

Article tools

Rights & Permissions

Felix Clay/eyevine

Economist Nicholas Stern has suggested changes to the UK's research audit system.

The United Kingdom’s mammoth periodic audit of its research is set for a shake-up that could change the way universities hire academics – and might also encourage scientists to spend more time on public engagement.

Every five to seven years, the UK runs an evaluation of the quality of research produced by its universities, and uses it to allocate around £2 billion (US$2.7 billion) in annual research funding.

The exercise, called the Research Excellence Framework (REF), has had a huge influence on the behaviour of British researchers and university managers – not least by encouraging a frenetic academic transfer market, in which universities try to improve their quality rating just before the next REF by rushing to sign up star scholars, sometimes on temporary contracts.

A government-commissioned report on the REF’s future – led by the economist Nicholas Stern, the chair of the Grantham Research Institute on Climate Change and the Environment – has broadly backed the system in its current form. But it calls for changes designed to cut costs and prevent 'gaming’ of the system. One of those would prevent universities from claiming credit for research papers written by staff before they joined.

If implemented, that change could cause problems for early-career researchers. They are often forced to move between short-term contracts and will be “worrying about their marketability if they can’t take their hard-won papers with them to incentivize universities to take them on”, says Richard Jones, pro-vice chancellor for innovation at the University of Sheffield, UK. On the other hand, he says, “I think it’s really important that universities nurture their own talents rather than rely on a chequebook to hire some people in at the last minute.”

Oliver Robinson, a neuroscientist at University College London, tweeted that the move would be a “huge blow” to early-career researchers, adding that it would probably exacerbate an expected post-Brexit brain drain. But others suggested the move could be positive: encouraging institutions to look for academic potential when making new hires, rather than making decisions based on past records.

Stern’s other major suggestion is that universities must submit all of their staff to the audit process – not just a selection of staff, as the REF currently allows. That might save costs, as universities currently spend a lot of time working out which staff will win them the highest ratings. (The costs are significant: the latest iteration of the REF, in 2014, cost £246 million to run, compared to £66 million for a simpler version of the process in 2008.) And it might also be more inclusive, avoiding the demotivation of academics who aren’t picked for the REF.

Another recommendation is that the audit should encourage universities to boast of their research’s wider impacts on public engagement, teaching or cultural life. That could solve what’s known as the “Brian Cox problem", says Stephen Curry, a structural biologist at Imperial College London. Cox, a physicist at the University of Manchester, UK, has become a television star, but his public engagement work wasn’t adequately recognized in the last audit. “I’m hoping that change creates new incentives to support public engagement work in universities,” Curry says.

The report also calls for future REFs to do more to recognise interdisciplinary work, by including university-wide assessments that look at interdisciplinary and cross-institutional initiatives.

Overall, universities have welcomed Stern’s ideas – not least his conclusion that the principles of the REF, including its assessment of research primarily on the basis of peer review by panels of academics rather than on metrics, should remain unchanged. “I think it’s fairer and more sensible – very much an evolutionary, rather than a revolutionary, change,” says Curry.  

Stern’s report acknowledges that work will be required to test the impact of the proposals on the scope for gaming, and to mitigate against unintended consequences.

The UK government said it planned to consider the recommendations before issuing a formal response. A full consultation on the next REF, which is scheduled to begin in 2020, will be published later this year.

Journal name:
Nature
DOI:
doi:10.1038/nature.2016.20343

For the best commenting experience, please login or register as a user and agree to our Community Guidelines. You will be re-directed back to this page where you will see comments updating in real-time and have the ability to recommend comments to other users.

Comments

Commenting is currently unavailable.

sign up to Nature briefing

What matters in science — and why — free in your inbox every weekday.

Sign up

Listen

new-pod-red

Nature Podcast

Our award-winning show features highlights from the week's edition of Nature, interviews with the people behind the science, and in-depth commentary and analysis from journalists around the world.