Despite its limitations, Excellence in Research for Australia was the right assessment tool at the right time, says Margaret Sheil.
In the early part of this century, the Australian government made a decision to improve the country's research effort. It made targeted investments in research agencies, built infrastructure and created a number of world-class centres of excellence. Yet some argued our research efforts were spread too thinly and performance was uneven. By 2007, only two of Australia's 40 universities were in the top 100 of the Academic Ranking of World Universities (ARWU). That same year, an Australian Productivity Commission report on public support for science and innovation identified several areas of the innovation system requiring improvement, which included a lack of effective support for industry-based research and development, deficiencies in the scientific workforce and inadequate methods of evaluation.
The government determined that Australia's dual funding system, comprising individual competitive grants and block grants to institutions, was not providing the right incentives. In particular, block funding was distributed according to formulae that took into account only the number of scholarly publications, with no consideration for their quality. The results were, in hindsight, predictable: the number of publications increased but not their overall quality. The government needed a new framework: one that would encourage universities to focus their endeavours and build on their strengths; to address weaknesses in research performance; and to provide researchers with an incentive to target quality. All of that had to happen within a system that was streamlined and cost effective.
From 2007 until 2012, I was chief executive of the Australian Research Council (ARC), the body tasked with developing the new framework. We weren't starting from scratch; the previous government spent nearly four years working on a new design, and we could draw inspiration from other assessment systems, including the United Kingdom's Research Assessment Exercise (RAE), and New Zealand's Performance-Based Research Fund (PBRF).
None of the approaches, however, were entirely appropriate for Australia. The proposed design from the last government relied on case studies to assess research impact. Case studies are expensive and time-consuming to prepare and assess, yet were not, the new minister felt, sufficiently robust to inform funding allocations. Similarly both PBRF and RAE — to different degrees — are selective exercises, focusing on just the best output, allowing universities to hide poor performance. RAE used panels of experts to assess the quality of publications — a costly and lengthy process. And the units of assessment for both schemes were unsuited to Australia: PBRF evaluates individual portfolios, making it difficult to scale up; RAE evaluates departments. But our universities have varied and complex organizations of departments, schools, faculties and/or research centres which do not lend themselves to a simple comparison.
These schemes did, however, provide useful leads. In particular, the 21 years of the RAE saw a growing correlation between the quality ratings assigned to departments and their citation performance. This suggested that metrics alone could be used as indicators for many scientific disciplines, although less so for others, such as the humanities and mathematics where citation data were less reliable or books were more important than journal articles.
In coming up with our own solution, the ARC consulted widely with institutions, learned academies, research leaders and bibliometric experts. We settled on 'discipline' as the best unit for evaluation because it avoided focusing on either university structures or individuals (thereby minimizing the value and prospect of poaching individual stars). Where citation analyses were not appropriate, peer review of selected publications or outputs was used instead. This approach also had the benefit of enabling evaluation of the creative arts — important for Australia where most conservatoria and visual and performing arts schools had been incorporated into universities.
Excellence in Research for Australia (ERA) was born in 2010. By including all outputs within a discipline, rather than just the premier efforts of a select group of researchers, ERA ensured that the attention of every researcher was on quality rather than quantity. And because its discipline-specific measures of achievement accorded with established academic practice, the results were largely accepted with only a small number of disputed outcomes.
There have been difficulties. ERA initially ranked around 22,000 journals in four bands (A*, A, B and C) to provide another set of indicators for the committees to use. However, some institutions used the journal ranks out of context, potentially harming the careers of young academics and those working in cross-disciplinary areas, since both groups are unlikely to publish in top tier journals. Bandings were removed in 2012.
Overall, ERA has been of considerable value — despite the fact that it drives only a small proportion of the block funding allocations. Crucially, it has demonstrated that citation analysis can be used as a principle indicator of quality in many disciplines, producing finely grained information about research strengths and weaknesses. Governments, universities, industry and the academic disciplines themselves have welcomed this information and make regular use of it. This success shows in our international standing: Australia now has 5 universities in the ARWU top 100, and 19 in the top 500.
Any comprehensive evaluation system is bound to have its critics. But widespread consultation and the use of discipline-based solutions has helped minimize opposition to the ERA. Though there is widespread acceptance that there are benefits in competitive processes for teams and individuals that have winners and losers, the political dimensions associated with 'losing' loom larger when applied to institutions, for example, negatively affecting newer institutions in areas serving growing and diverse populations. These considerations are beyond the control of the ARC and are unrelated to the ERA itself, which, while not perfect, has nonetheless achieved its goals.