Q&A Koenraad Debackere: Impact assessment to become more complex

We need to define impact more clearly to be able to measure it effectively.

28 August 2018

mangsaab/Getty

We need to define impact more clearly to be able to measure it effectively.

Koenraad Debackere is an innovation and economics researcher at KU Leuven in Belgium. He is chairman of the advisory board of the Association for the Economic and Societal Impact of Science, which held its annual conference in June in Ottawa, Canada to discuss global best practices on the impact of science. Debackere spoke with Smriti Mallapaty.

How has the measurement of impact evolved?

Koenraad Debackere

Impact is a nimble concept. It is multidimensional and multifactorial. To use it, we need to understand not just how to measure it, but also how to incorporate it in the design of new policy instruments, and how to assess it.

Over the past two decades, the algorithms which are used to look into research outputs have become increasingly sophisticated. Coupled to developments in traditional bibliometric and scientometric indicators has been the emergence of the webometrics and altmetrics movements. The advent of big data and data analytics has allowed us to use more diversified data. These machine-learning and text-mining activities are not only counting documents and citations, but can look into the content of those documents and citations to find out, for example, whether a publication is novel, or a research report is more incremental.

There is also the whole area of narratives — case studies on particular breakthroughs and activities that have led to an impact on society.

What are the challenges to making research impact visible?

We don’t yet have a coherent, overarching framework for handling the various views and demands on scientific impact — one that will help us structure our thoughts, insights and questions about impact and the way we ought to answer those questions.

To develop this, we need to come to grips with the fact that impact means something different to different people. You can’t start measuring before knowing what it is you want to measure. A social scientist might want to better understand human behaviour, a medical researcher might want to save the life of a patient, while an engineer might want to build a car that uses less fuel. They all have radically different interpretations of impact. Once we clarify the definition, we can start thinking about developing better methods to assess and measure those constructs, design appropriate policies and assess their success.

How is the definition of impact changing?

Gradually, we are moving away from a very narrowly defined quantification of impact to a broadly defined qualification. This does not mean that we have to throw away the quantification, but we have to complement it with more qualification-type impact measures. The problem with going down that path is that you move from something which is measurable at the large scale, to something that is only measurable at the granular scale.

Everything started with the Journal Impact Factor, but it has become increasingly clear that the impact factor is limited in what it can tell us. For example, it can reveal the impact of a journal, but not of an individual article. It is essentially an average.

From the impact factor, we began looking at paper visibility, but realized that the citation of individual articles doesn’t tell the whole story. Further down the road emerged measures that bring us closer to research use, such as patent analysis, which can only assess a specific type of impact. Clinical impact indicators allow us to begin to ask how the research influences patient treatment, mobility, survival rates and quality of life.

How do you see impact assessments developing over the next decade?

Assessments will be more fine-grained but also more complex. You will no longer be able to use one number as a measure of impact, but have to select the most appropriate from a portfolio of techniques and levels and units of analyses that best suit the application, society, and societal problem you want to address.

For instance, assessing the economic impact of a new health technology will require a different set of measures and analyses than assessing the impact of a particular scientific discovery on the development of technology platforms to fight climate change more effectively. The decision on ‘what’ to apply ‘when’ will have to be taken and monitored by the scientists and developers of the knowledge, the users of the knowledge (including policymakers supporting their use), and the experts of those measurement methods.

How are funders and policymakers beginning to incorporate research impact in the decisions they make?

We are seeing changes in the design of new policy programmes to fund the diversity of research. This means not just focussing on discovering new knowledge but also thinking about how that knowledge can be applied.

You see this in Europe with the different instruments for funding science: the European Research Council, the European Commission’s Horizon programme, the European Innovation Council, and the European Institute of Innovation & Technology (EIT). In various ways these programmes are beginning to involve multiple stakeholders in the policy design and the measuring and mapping of science.

What kinds of stakeholders are being incorporated in the decision-making process?

We come from an era in the 1960s and 1970s where science policy was about what the world of science judged to be very important. Policymakers would often decide what to fund based on a broad, bottom-up approach. We need such philosophy-driven science to survive, but as governments step up their research and innovation budgets, there is room for other stakeholders, including industry and patient communities, to co-define at least part of the allocation.

There has been a lot of writing, research and discussion on the topic of the triple helix of scientific impact, where collaborations between universities, industry and governments create economic added value. But we are now moving beyond purely economic assessments toward the ‘quadruple helix’ of scientific impact, which brings civil society and the public sector into the equation.

How will the focus on impact change the type of science that is produced?

The science of science impact will increase the focus on interdisciplinarity. You cannot create an outcome for the benefit of society or other uses that just stems from one discipline. Interdisciplinary work will be a driver of impact and a corollary of impact studies.