Nature | Research Highlights: Social Selection

NIH metric that assesses article impact stirs debate

The US biomedical funding agency is using the metric to evaluate funded research.

Article tools

Rights & Permissions

Some researchers love to hate the journal impact factor, because it can be used inappropriately to judge the influence of articles. As part of a push for better article metrics, researchers at the US National Institutes of Health (NIH) have developed the Relative Citation Ratio (RCR), which they describe in a paper posted on the preprint website bioRxiv1. The RCR normalizes citation numbers across disciplines, so that articles from biomedical fields with different citation rates can be compared. The NIH says that it is using the metric to assess the influence of work that it has funded. But some scientists debated on social media how accurately the metric reflects an article’s impact.

Researchers have developed a variety of impact measures that focus on an individual researcher or paper. The RCR is calculated by dividing the number of citations a paper received by the average number of citations an article usually receives in that field. That number is then benchmarked against the median RCR for all NIH-funded papers. This allows articles to be assessed on the basis of their relevance in their own field, and highly influential articles will be recognized even if they are published in an obscure journal, says George Santangelo, director of the NIH Office of Portfolio Analysis in Bethesda, Maryland, and a co-author of the RCR paper. “We want to identify influential papers wherever they might be published,” he says. “We can provide a measure of how each paper has influenced the field.”

According to Santangelo, the NIH is now using the metric as one of many tools for strategic planning. For example, the RCR can identify influential areas as well as ones that are important but need more funding to have a greater impact. Santangelo stresses that the algorithm is still at an early stage and needs improvement. He says that the RCR will not replace experts when it comes to making decisions about funding. “Data-driven methods need to be tempered by subject-matter expertise,” he says. “The bottom line is that human administrators will continue to make decisions.”

Stefano Bertuzzi, executive director of the American Society for Cell Biology in Bethesda, praised the new metric in a blog post, calling it “stunning”. Bertuzzi applauds the NIH for moving away from the journal impact factor (JIF). He wrote that the metricevaluates science by putting discoveries into a meaningful context. I believe that the RCR is a road out of the JIF swamp.”

But Ludo Waltman, a bibliometrics researcher at Leiden University in the Netherlands, criticized the metric in a blog post that was shared by many researchers on Twitter. He wrote that “the RCR metric doesn’t live up to expectations,” adding that it is complex and not very transparent. In an interview, Waltman said that this might stop researchers adopting it. “In general, metrics that are relatively straightforward, simple and transparent will be easily accepted by researchers,” he says.

Another limitation is that the RCR has been applied only to papers in the biomedical field, which excludes other disciplines from making use of the metric, says Phil Davis, a publishing consultant in Ithaca, New York, who analyses citation and publication data. This could also skew the scores for certain papers. The RCR calculation includes the number of citations received by papers that have been cited by the article in question. If that article cites ones from fields outside of biomedicine, that could lead to a lower RCR score, meaning that interdisciplinary articles could be unfairly penalized, says Davis.

In an interview, Santangelo said that the RCR does accurately account for interdisciplinary citations within biomedicine, adding that the metric was designed for biomedical papers because that is the NIH’s focus. He would be happy for other researchers to improve the algorithm so that they can use it in their own fields, he says. “We don't suggest [the RCR] is the final answer to measuring. This is just a tool. No one metric is ever going to be used in isolation by the NIH.”

Bertuzzi said in an interview that the RCR is a good start. “Maybe it won't work, but we're starting a conversation,” he says. “We're seeing different ways of looking at impact.”

Journal name:
Nature
DOI:
doi:10.1038/nature.2015.18734

References

  1. Hutchins, B. I., Yuan, X., Anderson, J. M. & Santangelo, G. M. Preprint at bioRxiv http://dx.doi.org/10.1101/029629 (2015).

For the best commenting experience, please login or register as a user and agree to our Community Guidelines. You will be re-directed back to this page where you will see comments updating in real-time and have the ability to recommend comments to other users.

Comments

Commenting is currently unavailable.

sign up to Nature briefing

What matters in science — and why — free in your inbox every weekday.

Sign up

Listen

new-pod-red

Nature Podcast

Our award-winning show features highlights from the week's edition of Nature, interviews with the people behind the science, and in-depth commentary and analysis from journalists around the world.