Australia and New Zealand are experimenting with ways of assessing the impact of publicly funded research.
When it comes to research, governments the world over are asking more questions about whether they are getting value for money — and there is nothing wrong with that, says Peter Gluckman, chief science adviser to the prime minister of New Zealand. “It is what the whole of a nation's science policy process is about: how much to allocate to public sector support; how much to industry sector support; how much to invest in health versus relative amounts for environment, for instance,” he says. “Whether it is done implicitly or explicitly, everyone in that process is thinking about impact.”
And whereas large economies have the capacity to invest in a wide range of scientific endeavours, from nanotechnology to cosmology, smaller countries such as Australia and New Zealand do not have that luxury. “The smaller the country, the more limited the choice,” says Gluckman. “So when looking at science and innovation systems, you start to be more conscious in the prioritization process; it becomes a much more strategic issue.”
Determining the impact of research on wider society has the potential to assist decision-makers within organizations and institutions. But what is troubling people like Gluckman are the definitions. “You have to be really clear about the word: there are many different kinds of impact and perceptions differ,” he says. “Governments have to decide what impacts they are looking for.” Questions surround what constitutes impact and at what point during or after the research process it should be evaluated. Can something that is subjective and qualitative ever be appropriately measured?
Gluckman's office in Auckland serves as the administrative headquarters for the science, technology and innovation stream of the Small Advanced Economies Initiative (SAEI) — a network for the discussion of challenges, opportunities and policies that are of particular relevance to small developed nations. For these countries, prioritizing the areas of science and innovation in which they invest is crucial to economic prosperity.
The SAEI has begun to develop an 'impact taxonomy' to help categorize the range of impacts that can arise from research. These include not only the direct economic effects but also intangible factors — for example, a Nobel prizewinner's role in enhancing a country's scientific reputation. Gluckman says that a holistic science system has to consider all the different kinds of impact that matter to people; a taxonomy, he contends, will facilitate that discussion. “What is the 'value' of doing the kind of biosecurity research that makes foot and mouth disease less likely or a country more resilient to an earthquake?” he asks. “This type of research could easily get forgotten if you focus on only one form of impact.”
But measuring the breadth of impacts arising from research that often takes place over many years poses tough challenges. Assessors need to identify proxies and intermediate outcomes to gauge the direction a study is taking. That is where Adam Jaffe, director of New Zealand's Motu Economic and Public Policy Research think tank in Wellington, comes in. Jaffe is working on an evaluation framework covering five categories of impact: financial, social, environmental, public policy and capability. The purpose of this framework is to give decision-makers an idea of the full range of potential outcomes and help them decide which to track — and how.
Many scientific discoveries are serendipitous, and critics suggest that such a goal-oriented approach could lead to less blue-sky research funding. Gluckman disagrees, saying that knowledge advancement is in itself a goal. “To say impact assessment moves you away from basic science is to narrowly use the term 'impact'.” Jaffe, an American, uses a baseball analogy to defend impact assessments. “The fact that sometimes you strike out and sometimes you do well doesn't stop us from thinking about who is better on average,” he says. “We can look at which models on average generate the greatest outcomes and impacts across a number of different measures.” The inevitable randomness in the process, he adds, “doesn't invalidate this approach”. Using a framework such as his, which incorporates multiple measures across many dimensions, will minimize the chance of missing important effects.
Such a framework will probably be attractive to New Zealand's recently launched National Science Challenges (NSCs), which provides designated pots of money to research areas deemed to be of national significance. The NSCs have a funding horizon of ten years — part of the government's move to support science over a longer term than most other funding bodies, which typically give three- to five-year grants. In their proposals, applicants are required to describe their ten-year vision, anticipated outcomes and impact.
Auckland University physicist Shaun Hendy and his team have recently submitted an application for one NSC, called Science for Technological Innovation, which covers work that “enhances the capacity of New Zealand to use physical and engineering sciences for economic growth”. Hendy, who is also the director of the Te Pūnaha Matatini Centre for Complex Systems and Networks, hosted by Auckland University, found the requirement to discuss impact hampered his application. “Our proposal covers a very broad range of disciplines and industry stakeholders,” he says. “Manufacturing technologies are changing so rapidly that we've struggled to design a research programme that will deliver short-term impact but will also be relevant in a decade.”
Research teams that are awarded NSC funding will be required to develop a 'pathway to impact' plan, complete with monitoring and evaluation procedures. Hendy does not believe this sort of assessment is particularly helpful in the long run, because it does not measure the opportunity costs. “To determine the real benefits of the National Science Challenges, an economist would need to know what we chose not to fund as well as what we did,” he explains. “These sorts of exercises are much more about bureaucratic box-ticking than any real attempt to measure the value of science to society”.
The Commonwealth Scientific and Research Organisation (CSIRO), whose headquarters is in Canberra, is the largest of the Australian government's portfolio-funded research agencies. CSIRO is one of the few public research and development entities in Australia — and possibly the world — that formally and transparently plans, monitors and evaluates the impact of its research, according to CSIRO's Mark Johnson. He is project manager of Impact 2020, launched four years ago with the aim of developing a framework to assess the economic, environmental and social impacts of CSIRO's Flagship programmes for use across the organization. For CSIRO, impact is used to influence its “4As”: allocation (of resources), advocacy, accountability (to government and other key stakeholders) and analysis (for performance improvement).
CSIRO uses an impact pathway model that describes a project's inputs, activities, outputs, expected outcomes and eventually impact — for example, the adoption of new research protocols that improve productivity (See 'Pathway to impact'). Within the Flagship programme, project leaders can modify these pathway stages depending on changes in resources and goals of either CSIRO or its clients and partners.
CSIRO's research impact planning is a dynamic process; the organization is continually evaluating changes beyond the bounds of a study to see whether the focus of that study is still appropriate. This eye to the wider world also helps with internal engagement. “A lot of people get hung-up on the long-term nature of most impacts, which they see as so far removed from their actions today that they are reticent to engage,” says Mark Bazzacco, CSIRO's executive manager of performance and evaluation. He says that “monitoring progress towards impact” as opposed to “monitoring impact” helps scientists to keep an eye on their project's goals and get a greater sense of the role they play in effecting change.
Bazzacco gives an example. Some research impacts are the culmination of decades of work. The Murray-Darling Basin management plan was a large collaborative research project that aimed to measure and model water flows within this vital river basin. In addition to collecting data, the researchers involved also helped policy-makers understand the modelling and the plan's findings to enable creation of new regulations concerning water usage. But once the new policy had been developed, it was up to state and federal governments to adopt it — and then for many other actors to implement it. It will take another decade at least for the final impacts (environmental, agricultural, social and economic, for example) to be realized; in the meantime, the researchers have moved on to other projects. Monitoring progress towards impact recognizes where the researcher's role is important (for example, in conducting the work and helping communicate the results) and avoids any negative implications of assigning responsibility for behaviours that influence the final impact yet are beyond the researchers' control (in this example, the implementation of the policies).
Over its lifetime, CSIRO has conducted tens of thousands of projects. Johnson is working on ways to show these in an easy to appreciate, visual manner. They have come up with an impact map, shown , designed to be a conversation starter rather than a precise diagnostic tool. It shows 286 projects — only those that meet a minimum realized or projected financial return criterion and where data are available to evaluate the delivered or intended impact. Impacts can be social, environmental and/or economic. Projects are assigned a primary impact category based on the same 17 socioeconomic objectives used by the Organisation for Economic Co-operation and Development.
The size of the bubbles is based on a four-point scale that allows projects with different types of outcome to be compared with each other. The placement of the bubbles are indicative of when projects have delivered, or will likely deliver, a significant milestone. From 2004, the timescale changes from five- to two-yearly, as there are more data are available for recent projects. Here are some examples, highlighted on the image.
Wireless LAN: CSIRO scientists solved the main problem impeding fast wireless networking of electronic devices — that of reverberation within rooms. The organization applied for several patents and, in 1996, was granted a US patent for wireless local-area network (WLAN, or wifi). This outcome has led to major social and economic impacts and has revolutionized communication. The technology is now used in an estimated five billion devices worldwide. CSIRO has licence agreements, worth more than AUS$430 million (US$400 million) with more than 20 international companies.
Murray-Darling basin plan: The Murray-Darling is the largest river system in Australia. It supplies water to approximately 10% of the population and produces 40% of national agricultural output. Growing demands on its increasingly variable flows have caused widespread concern among communities and industry. CSIRO scientists measured the available water within the basin and created a model. In 2011 they produced a report on resource planning, management and investment, as well as modelling scenarios for future catchment development, groundwater extraction and climatic conditions out to 2030. The report and models directly informed government investment; the project's impact has been an estimated saving of at least AUS$2.8 billion (US$2.6 billion) through better use of funds and water infrastructure efficiencies.
Tiger prawn and Aquaculture feeds: In 2010, CSIRO announced that its ten-year collaboration with Australian prawn farmers had led to successful selective breeding of Black Tiger prawns with improved growth and survival rates that could be sustainably farmed in salt-water ponds. The estimated economic impact on the industry is AUS$120 million (US$112 million) per annum. Simultaneously, another CSIRO research team developed a new aquafeed ingredient (Novacq), derived from marine microbes, which increases Black Tiger prawn growth rates by 30%. The combined value of the increase in prawn yield is estimated to be AUS$430 million (US$400 million).
The Cooperative Research Centres (CRCs) programme in Canberra, set up in 1990 by the federal government, also looks at the broader impact derived from its applied research. CRCs support multidisciplinary teams, which often include groups from CSIRO, and are carefully managed to deliver impacts, says Tony Peacock, chief executive of the CRC Association, the CRC umbrella advocacy body. All CRCs include participation of end-users from the outset, ensuring that projects are always addressing real-world situations. Any proposal for CRC funding involves completing an impact tool similar to CSIRO's impact pathway approach.
Over the past decade, there have been three independent retrospective analyses of the CRC programme. The methodologies were agreed up front with government officials and use a counterfactual point of view — that is, they compare the impact of the CRCs with a scenario in which each project had not taken place. All three studies showed that CRCs have had a positive impact, including on economic indicators such as GDP. Peacock is open about the intended target audience. “There is no problem convincing the public of the value of research — they want to hear our stories,” he says. “These retrospective studies were directed solely at those who were likely to determine whether the CRC programme continues to get money.”
Australia's higher, or tertiary, education sector, which includes all the country's universities and almost two-thirds of its scientists, does not routinely conduct research impact assessments. These institutions are, however, compelled to monitor research quality through the Excellence in Research for Australia (ERA) initiative (see page S64).
In 2003, the then Liberal government started developing an assessment system — called the Research Quality Framework (RQF) — that looked at the impact of research as well as its quality. But in December 2007, just weeks before the RQF was due to be implemented, an incoming Labor government scrapped it (see page S52). Matt Brown, senior policy analyst in research at the Australian Technology Network of Universities (ATN), in Adelaide, believes that an opportunity to provide a useful decision-making tool was lost. “We've seen car manufacturing suddenly disappear from Australia and now the country is asking what its future industries will be,” he says. “Research impact assessment would have helped guide today's decisions.”
Another attempt to push the research agenda came in 2012, when 12 Australian universities took part in the Excellence in Innovation for Australia (EIA) trial. The EIA demonstrated how a case-study approach could be used to systematically assess the impact of university research across a wide range of disciplines and areas. “The time was right to have another go at putting impact on the agenda,” says Brown. “And when trying to convince policy-makers, you need proof by demonstration.” In the EIA trial report, the authors note that they met their objective to “measure the innovation dividend of research generated by Australian universities” and advanced the methodology to do so.
Australia's chief scientist, Ian Chubb, seems undecided as to the benefits of an impact assessment system for the tertiary sector and has concerns around the retrospective case-study approach, as recently implemented in the United Kingdom's new Research Excellence Framework (which, ironically, was influenced by the aborted RQF). This approach is also highly selective, with universities submitting only those studies that demonstrate the best results. Chubb echoes Gluckman in the need to articulate the goal of any assessment system. “If the purpose of measuring impact is to show politicians just how much scientific research underpins sectors of the economy, then we should own up to it,” he says. “But if impact evaluation is to be tied to funding, then you have to be careful that you're not just being fashionable. You're not going to give a university funding because 15 years ago someone had an idea that turned out to be profitable.”
In an effort to ensure better coordination of activity and investment, without succumbing to the whims of fashion, the Australian Research Committee (ARCom), based at the Department of Industry in Canberra and chaired by Chubb, put together a national research investment plan to guide government. The plan was released in November 2012. As part of its plan, ARCom, which includes the Canberra-based chief executives of the Australian Research Council (ARC), the National Health and Medical Research Council (NHMRC) and CSIRO, recommended the development of a university-focused impact assessment mechanism as a companion to the ERA. ARCom released a discussion paper in June 2013 and the submissions window was closed in August — just in time for another change in government. The incumbent Liberal party has not endorsed the plan.
In the absence of government leadership, one Australian university has decided to go it alone. “We are in a different political environment now,” says Warren Payne, pro vice-chancellor in research and research training at Victoria University, Melbourne, “and we need to be ready for a measure of impact that might come on top of the current ERA framework.”
Individual research programmes at Victoria University are assessed for retrospective and prospective impact; economic, social and environmental outcomes are combined with qualitative surveys that gather the views of people who are socially or financially invested in the project on potential future impacts. The university is currently trialling the system and has brought in an independent assessor to evaluate progress. So far, 11 projects from Victoria University's social science and science technology departments have been successfully assessed via a rating scale that considers the significance and reach of each of the impact claims.
Payne sees Victoria University's impact tool as a useful way to guide its research decisions. But he says that perhaps the greatest attribute of the system is to focus the minds of researchers as to what expectations they are raising and whether they are delivering on them. And, as an added benefit, they are more motivated because they understand how their work feeds future impacts.
One of the strongest arguments for assessing impact is the inclusion of research that falls outside the traditional criteria for academic excellence. “Work that previously might have been seen as highly applied — almost in a pejorative sense — might then be recognized as being important,” says Tim Wess, executive dean of science at Charles Sturt University in Wagga Wagga. Wess gives the example of research into changing a nursing procedure that reduces post-operative mortality, compared with fundamental research in particle physics. Assessing both types of research by their excellence and impact “would level the playing field”, he adds. University promotion committees and grant application reviewers could also take this information into consideration.
Unfortunately, no single group of measures may be able to indicate both the excellence and the impact of research. Publishing a paper, even an 'excellent' one, may not have an impact outside academia without additional effort — often undertaken by others — to translate that knowledge into practice. This raises the issue of timeframe. The wider benefits of research might not appear for years or decades, by which point many individuals and organizations may have contributed. How can an impact measure tease out separate contributions?
Indeed, coming up with a suitable methodology is a big sticking point. Payne has discussed Victoria University's approach with the ARC, which administers the ERA assessment exercise and is the larger of Australia's two research-funding councils. “ARC feels that although our method can probably be scaled quite nicely within the university, it isn't sure it can be done systemically,” he says. The other funding body, the NHMRC, requires some mention of impact in its grant proposals, but the guidelines are not explicit and vary by scheme. “Consensual views as to what can be claimed as impact and what evidence should be provided would be very helpful,” says NHMRC chief executive Warwick Anderson.
So far, it is a discussion that has been disparate — and one that, given its membership, ARCom seems well placed to continue. “I'm not saying it shouldn't be done — or that a whole bunch of smart people can't find a way,” says Chubb. “We just can't afford to be simplistic about it.”
About this article
BMC Medicine (2016)