Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The maze of impact metrics

A Correction to this article was published on 23 October 2013

This article has been updated

In deciding how to judge the impact of research, evaluators must take into account the effects of emphasizing particular measures — and be open about their methods.

So much science, so little time. Amid an ever-increasing mountain of research articles, data sets and other output, hard-pressed research funders and employers need shortcuts to identify and reward the work that matters. They have plenty of options: research impact is now recognized as a multidimensional affair.

The conventional measures of scholarly importance — citation metrics, publication in influential journals and the opinion of peers as expressed in letters and interviews — still loom large. But to those are now added metrics such as article downloads and views, and measures of importance beyond the academic realm, including influence on policy-makers or health and environment officials, effects on industry and the economy, and public outreach.

It has never been easier for scientists to show off the various ways in which their work deserves attention — and funds.

Researchers at the Center for the Study of Interdisciplinarity, part of the University of North Texas in Denton, this year came up with 56 measures of impact (see Nature 497, 439; 2013), including influence on curriculum creation, authorship of textbooks and success in surveys of colleagues’ esteem. Some of these measures are a little fanciful, but they demonstrate that it has never been easier for scientists to show off the various ways in which their work deserves attention — and funds.

That variety is worth celebrating, but it can lead to dizzying confusion. How are researchers and evaluators to choose between measures? In this issue, Nature looks at some traditional and emerging ways to track research quality (see page 287). Ultimately, it is for institutions and funders to choose their preferences, but in doing so they should take two important considerations into account.

First, it is important to be aware of the positive and negative effects of privileging certain measures.

For example, emphasizing that research is considered especially important if it is published in one of a few historically influential journals — Cell, Nature, Science— could be a laudable attempt to get scientists to think ambitiously about their research goals. But it can also result in excessive pressure to publish big claims, leading to problems of irreproducibility, for example. (Nature’s position is that it has been publishing research using essentially the same criteria for decades; it is up to the scientific community and evaluators to decide how much importance they want to place on papers that appear in the journal).

Nature special: Impact

It is a mistake to consider a research paper important because it is published in a journal with a good citation record, as measured by its impact factor. As this publication has highlighted many times (see in particular Nature 435, 1003–1004; 2005), two articles in the same journal may have very different citation records. It is much better to focus on the citations, views or downloads of an individual article — and to recognize that these metrics vary between research disciplines.

In another example, emphasizing the economic impacts of research may force scientists to think about justifying their taxpayer-funded work, but it also runs the risk of distracting them with the lure of meaningless patents and ill-considered spin-out companies.

Nature special: Science metrics

The second important consideration is the need for research evaluators to be explicit about the methods they use to measure impact. Openness is an essential part of earning trust. Evaluators should publish worked examples showing how they score assessments, and the reasoning behind such scores; even better would be, where possible, to publish the full data. Otherwise, researchers might rightfully feel suspicious (see, for example, writer Colin Macilwain’s scepticism towards performance metrics: Nature 500, 255; 2013).

When scientists rail against the ‘impact agenda’, their arguments sometimes founder on irrelevant confusion between terms: too often, such discussion devolves into attacks on misuse of the impact factor, rather than looking at the range of possible metrics. The journal citation measure gains misleading prominence because its name happens to include the word impact — a semantic synergy that can cloud debate.

Arguments against impact metrics are strongest when they reference cases in which evaluators do not heed the considerations we mention above: in which evaluators choose metrics blindly, without sufficient thought for pernicious effects, or are secretive or inconsistent about their methodologies. If evaluators are to earn the acceptance — rather than the scorn — of the scientists whose work they want to fund, they had better pay attention to these concerns.

Change history

  • 21 October 2013

    This article originally gave the wrong location for the University of North Texas — it is in Denton not Dalton. The text has now been corrected.

Related links

Related links

Related links in Nature Research

Assessing assessment 2010-Jun-16

The ratings game 2010-Mar-03

Not-so-deep impact 2005-Jun-22

Nature special: Impact

Nature special: Science metrics

Related external links

Rights and permissions

Reprints and Permissions

About this article

Cite this article

The maze of impact metrics. Nature 502, 271 (2013). https://doi.org/10.1038/502271a

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/502271a

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing