Concerns that research-assessment systems are too narrow in what they measure are no longer new. Existing approaches favour individuals or teams that secure large grants, publish in journals with high impact factors — such as Nature — or register patents, at the expense of high-quality research that does not meet these criteria.
According to a report in November 2020 by the Research on Research Institute (RoRI) — a network of experts who study how research is done — this method of assessment puts pressure on the research community to succeed for the sake of performance metrics. It also increases the risk of violations of research ethics and integrity. At the same time, it acts as a systemic bias against all those who do not conduct — or choose not to prioritize — research that meets criteria that can be measured with a number.
Concerns about the distorting effects of commonly used assessment procedures have already led to initiatives such as the San Francisco Declaration on Research Assessment (so far signed by more than 2,500 institutions, including Nature’s publisher Springer Nature, and 19,000 individuals); the Leiden Manifesto for research metrics; the SCOPE principles established by the International Network of Research Management Societies; and the Metric Tide report, commissioned by UK funding bodies. There are, in fact, at least 15 distinct efforts urging policymakers, funders and heads of institutions to ensure that assessment systems minimize harm.
Many of the architects of these projects are becoming concerned that each subsequent initiative amounts to more (no doubt, valuable) talk, but less by way of practical action.
The Agreement on Reforming Research Assessment, announced on 20 July and open for signatures on 28 September, is perhaps the most hopeful sign yet of real change. More than 350 organizations have pooled experience, ideas and evidence to come up with a model agreement to create more-inclusive assessment systems. The initiative, four years in the making, is the work of the European University Association and Science Europe (a network of the continent’s science funders and academies), in concert with predecessor initiatives. It has the blessing of the European Commission, but with an ambition to become global.
Signatories must commit to using metrics responsibly, for example by stopping what the agreement calls “inappropriate” uses of journal and publication-based metrics such as the journal impact factor and the h-index. They also agree to avoid using rankings of universities and research organizations — and where this is unavoidable, to recognize their statistical and methodological limitations.
Signatories must also pledge to reward more-qualitative factors, such as the standard of leadership and mentorship, including PhD supervision; as well as open science, including data sharing and collaboration. It is absolutely the case that the final research paper isn’t the only indicator of research quality — other forms of outputs such as data sets, new article formats such as Registered Reports (Nature 571, 447; 2019) and more-transparent forms of peer review are equally important.
What makes this more than just another declaration of good intent is that the signatories are committing to creating an organization that will, in effect, hold themselves to account. In October, they will meet in a United Nations-style general assembly to review progress and to create a more permanent structure. Central to that structure will be the idea of giving researchers, especially early-career researchers, an influential voice. They need to be around the table with their institutions, with senior colleagues and funders — those whose assessment systems have been the source of much stress at the present time.
The agreement focuses on three types of research assessment, covering organizations, such as universities and departments; individual researchers and teams; and specific research projects. Each assessment type will almost certainly need different kinds of arrangements, and these, in turn, will vary from country to country.
But the point of this exercise is not to create one uniform method of assessing research. It is to enunciate principles that everyone can agree on before they embark on their assessments. Assessments must be fair, the reasons for decisions transparent, and no researcher must be disadvantaged or harmed. If excellence is to be the criterion, then this should not be confined to a narrow set of indicators (such as funding raised or publications in journals with high-impact factors), as Nature has argued consistently (Nature 435, 1003–1004; 2005). There is excellence in mentorship, in sharing data, in spending time building the next generation of scholars, and in identifying and giving opportunities to under-represented groups.
As the authors of the RoRI report say, the time for declarations is over. Research assessment must now start to change, to measure what matters.