Experts fear irrelevance for assessment of research relevance

Australian academics will need to prove that they have demonstrated the usefulness of their work to communities, industry and government.

28 March 2018

akindo/Getty

 

Australian universities are preparing submissions for a programme that will assess not just the impact of their work but also their engagement.

But science policy experts fear the world-first initiative may turn out to be little more than a box-ticking exercise, because there is no funding attached. Many researchers are already doing the work they will now be expected to account for, so analysts doubt it will lead to much change in behaviour.

Yet the world is watching, says Andrew Gunn, a higher-education researcher at the University of Leeds. “If the assessment of engagement in Australia is successful, there will be a new benchmark out there,” he says. “Countries are always on the lookout for tried and tested new schemes to take off the shelf.”

Impact is the influence of research on society, while engagement means the efforts that researchers have taken to demonstrate the value of their work.

The latest iteration of Australia’s national impact assessment was announced in 2015, soon after Prime Minister Malcolm Turnbull unveiled a AU$1.1 billion innovation plan, which would rely on strong ties with industry. “The Australian system is unique in that it measures both impact — the end results — and engagement — the means to those ends,” says Gunn.

The Australian Research Council (ARC) has launched the Engagement and Impact Assessment (EI) across 42 universities following a one-year pilot, which determined the final metrics. Submissions are due in July 2018, and will complement the fourth round of evaluations of research quality under the Excellence in Research for Australia (ERA), which ends in May. Australia has committed Au$11.2 million over five years from 2015 to implement the new initiative.

Panellists will rate university submissions in both tracks as high, medium or low. To prove impact, universities have to provide a detailed description of the contribution of their research to the economy, society, environment or culture in specific fields, and how they helped bring about those outcomes.

Engagement will need to be demonstrated through a 1,000-word narrative, and a 750-word statement elaborating on quantitative outputs, including income from research commercialisation, as well as the amount of cash and grant sponsorship they received from those who will use or directly benefit from their work.

The engagement track includes activities currently not acknowledged by the UK system, says Gunn, such as public lectures, media appearances, and policy debates.

By acknowledging these activities, Australia will incentivize research teams to engage with communities, says Michael Mintrom a public-sector management researcher at Monash University in Melbourne. “Researchers are going to be respected for their outreach in a way that wasn’t in the past.”

No cash prize

But some researchers question whether the assessment will lead to any visible shifts in academic behaviour. The current scheme is not tied to funding decisions, and it is unclear to what extent it will inform the ERA.

Without financial incentives for universities, it won’t be worth their while to take part, says Gunn. “You need to have the money coming from Canberra as a reward for doing well or the scheme won’t work.”

Universities already keep track of their engagement, says a research unit staff member at an Australian university involved in the pilot, who requested anonymity due to the confidentiality of the process. “The benefit of an external benchmarking exercise for this purpose is still somewhat unclear.”

In 2015, a government-commissioned review of research policy and funding arrangements in Australia recommended that some funding should be influenced by the EI assessment. The government accepted that recommendation, and will consider funding implications after completing the full assessment this year, says Leanne Harvey, acting chief executive officer of the ARC.

“Regardless of funding, the EI assessment will help provide clarity to the government and the Australian public about how their investment in university research translates into tangible benefits beyond academia,” says Harvey. “It is likely that universities will see reputational benefits and it may help high-performers raise their profile with possible industry partners.”

The ERA’s assessment of research quality also does not directly inform funding allocations for universities, says Mintrom, though many expect financial incentives for high performance will be strengthened.

“In some ways, how to measure the engagement of academics is the next big thing,” says Gunn. The UK government, for example, is currently developing a Knowledge Exchange Framework, to go alongside the Research Excellence Framework (REF), which will assess how university knowledge is shared with businesses. “It will be very interesting to watch it in Australia.”

Impact agenda

Impact has been on the agenda in Australia since the early 2000s, when then prime minister John Howard’s government proposed the Research Quality Framework to measure non-academic research outcomes.

“The idea of the knowledge economy was very fashionable in the early 2000s,” says Gunn, who recently co-authored a paper with Mintrom on models for measuring impact. “Governments started to look for ways to maximize the wider economic benefits of research, feeding into things like industry and innovation policy.”

Australia’s proposal was jettisoned along with Howard’s party in 2007, but it informed national evaluations developed in Europe. Today, 20% of the UK’s REF accounts for impact, and Dutch institutions are assessed on the ‘relevance’ of their work under the Standard Evaluation Protocol.

Funding bodies have also moved away from a narrow focus on excellence. Applicants for Research Council grants in the UK, and ARC grants in Australia must describe how their proposed research will lead to tangible societal benefits. Anne Kelso, chief executive officer of Australia’s National Health and Medical Research Council, recently announced that the agency will also look at applicants’ track record in achieving impact in the next round of funding.