Q&A Trish Greenhalgh: Linear impact narratives encourage easy science

The REF’s approach to impact incentivizes research that can be obviously linked to outcomes.

19 April 2018

z_wei/Getty

Primary health scientist Trish Greenhalgh of the University of Oxford is a member of the public health, health services and primary care sub-panel in the United Kingdom’s Research Excellence Framework (REF) 2021. She spoke with Smriti Mallapaty.

The impact agenda is gaining traction globally. Impact case studies will account for a quarter of the score for universities in the REF 2021, up from a fifth in 2014. What is driving this growth?

When I first went to university in 1977, universities were seen as a public good, something worth funding in their own right. These days, there is an expectation that public money should be spent responsibly, efficiently and transparently. In the UK, this can be traced back to a policy approach introduced in the late 1980s known as the new public management. The idea was to get rid of the layers of fat in the public sector, and ensure that it was delivering value for money.

More generally, society is also moving towards a rationalist approach to project management — this idea of setting objectives and targets. My kids learned it when they had to present their homework diaries every week.

Trish Greenhalgh

Is it possible to create a cohesive and all-encompassing measure for research impact?

Impact doesn’t have a single consistent meaning across countries and sectors. In a way, we are reifying the concept of impact.

There is a trade-off between breadth and depth. If you want to measure the impact of every bit of research that everyone in a university has ever done, you have to use something that is easy to measure and probably automated. But if it is more important to get a rich and authentic picture of a sample of research programmes, you can afford to spend a bit of money looking at it in more detail.

How does the UK REF measure impact?

Impact in the REF 2014 was measured using case studies, which were four-page narratives describing a piece of research and the impacts that resulted from it. Those case studies were pretty successful, since they combined a logic model approach setting out the ‘inputs’ (research funding) and the ‘outcomes’ (societal impact) while allowing researchers to use narrative to add nuance to the facts and figures. Whilst the link between research and impact is never linear, the narratives are written retrospectively, which allows the researcher to draw a chain of causality (perhaps in a meandering way) between the research and its impact.

Is it problematic to take such a linear approach to measuring impact?

Requiring researchers to demonstrate a causal link between their research and its effects creates perverse incentives for people to do research that is easily, and obviously linked to an outcome. In reality, interactions between research and society are dynamic and interdependent, not linear.

Research into smoking offers a useful example. Over the past 50 years, a lot of groups have conducted research into the best ways to get people to give up smoking. Among them have been groups leading randomized controlled trials of antismoking drugs, research into policies such as government bans on smoking in public places, studies on how to conduct motivational interviews between counsellors and patients, and investigations of the bad behaviour of the tobacco industry. There has also been a lot of secondary research (systematic reviews and meta-analyses) summarizing this work.

These efforts have coincided with a drastic decline in smoking rates in the UK, from 46% in 1974 to 16% in 2016. How does one attribute this reduction to any one of the many lines of research, or even say how much of the drop was due to any research rather than a change in social attitudes?

We need to ensure that the difficulties in establishing causation do not avert people from smoking cessation studies. The problem is even more acute now that we have picked all the low-hanging fruit. Many of the remaining 16% of smokers have multiple co-morbidities and risk factors, including homelessness, mental health, drug and alcohol problems, and include a high proportion of ethnic minorities. There’s some excellent research going on into how we can reduce that 16% even further, but causality won’t be easy to demonstrate.

How does the REF account for gaps in its assessment of impact?

The REF recognizes that there are certain types of activity that cannot be directly attributed to a researcher or their work. It does this by only requiring institutions to submit a select number of case studies. In my department, for example, about a dozen people have potential impact case studies for the next REF, but we probably only need six. What the REF is saying is that they know some of our research will have impact in indirect and unmeasurable ways, don’t stop doing that, but they can’t give us points for it. Some things are inherently unmeasurable.

Is the narrative approach taken in the UK’s impact case studies a reliable method for establishing impact?

The idea that narrative is unscientific is a massive misrepresentation. Quantitative metrics without any sense-making narrative are unscientific. A group of researchers at the public policy institute RAND unpacked the claims made in the impact case studies from the REF 2014, with reassuring results. They confirmed that researchers, with few exceptions, weren’t lying or over-claiming. In general, claims were appropriately measured and cautious, as one would expect from academics.

Disciplines such as social policy, history, and political science are familiar with the case study as a genre and use standards of rigour in constructing narratives. In some of the universities that did well in the impact section of the REF 2014, the case studies were written by social policy academics.

What are some new ways being developed by researchers to measure impact?

We need to move away from a situation where impact metrics are driving academic work to one where academic work strategically uses particular impact measures. I have been working on an approach with Wilfred Mijnhardt from Erasmus University Rotterdam, whereby research groups pick from a menu of metrics that allow them to measure progress towards their strategic goals. For example, if my goal is to equip young researchers with the academic skills and moral qualities we want from professors, I might select metrics on the number of opportunities I have offered junior staff to develop in creative ways. There is sadly in some universities a culture of eating their own young, where the only objective given to early-career researchers is to write lots of papers, which will tend to make them pushy over-performers.

How would such an approach co-exist with national assessment programmes that might be encouraging perverse behaviours?

There will of course be clashes, because national policymakers are marching to a slightly different beat. But dialogue between academics and policymakers can lead to change, as seen with the REF. Many academics, including myself, were not happy with the REF 2014 process. It was too expensive, too top-down and some said it traumatized young researchers into leaving academia. The government asked economist Nicholas Stern to conduct a review, which many of us academics contributed to. As a result, the government decided to take the pressure off young academics in the REF 2021, by changing the unit of analysis from individual academics to research groups.