Italy’s rise in research impact pinned on ‘citation doping’
Citation of Italian-authored papers by Italian researchers rose after the introduction of metrics-based thresholds for promotions.
23 September 2019
Italy is climbing international rankings of research impact, but that doesn’t mean the country’s science has improved or become more influential.
An analysis suggests the upward trend could be largely the result of Italian academics referencing each other’s articles more heavily to satisfy controversial management targets at universities.
“The rise of Italy in impact rankings appears to be the fruit of a collective ‘citation doping’ induced by the new policies,” says Alberto Baccini, an economist at the University of Siena in Italy who led the study, published on 11 September in PLoS ONE.
Italy, Baccini says, has been held up as an example of a nation that has boosted its research quality, even as its science funding took a hit after the 2008 financial crisis.
In 2010, concerned by accusations of cronyism, the country passed a law that requires researchers to meet thresholds in certain metrics — such as the number of journal articles they publish and the number of citations they receive — before they can be hired or considered for promotion to particular levels.
It also began research assessments that relied partly on citations.
Italy’s research impact — as measured by its citations — has since risen. But Baccini says his study suggests that this has been caused, in part, by an increase in self-citation, and possibly by “citation clubs”, in which scientists strategically reference each other’s works.
“Behind the Italian paradox there is not some best practice to be considered as a model by other countries, but a great bibliometric masquerade,” Baccini says.
Inward-looking
Researchers have already documented a rise in self-citations among some Italian scientists since the 2010 legislation.
But Baccini and his colleague Eugenio Petrovich worked with Giuseppe De Nicolao at the University of Pavia in Italy, to examine this on an international scale.
They used a citation-analysis tool called SciVal, built on Elsevier’s Scopus database, to look at all the citations that Italian papers received from 2000 to 2016. The proportion of those citations that came from papers with at least one Italian co-author provided an indicator of what they call “inwardness”.
After 2009, Italy's inwardness rose rapidly compared with other major countries, including Germany, France and the United Kingdom, even though its rates of international collaboration did not.
The team couldn’t analyse in detail what lay behind the jump, but De Nicolao says he’s “quite sure that self-citation accounts for most of the increase”.
Ludo Waltman, a bibliometrics researcher at Leiden University in the Netherlands, told Nature that after seeing the study, he successfully reproduced the analysis using a different database.
In a further analysis, he showed that self-citations by individual authors contributed a large amount to the observed rise in Italian authors citing Italian-authored papers.
He says that Baccini and his team “offer pretty strong evidence to support their claim that the Italian evaluation system leads to ‘strategic’ citation behaviour”. Moreover, “the effect seems to be quite substantial. I find this disturbing,” Waltman says.
Not everyone is so convinced. John Ioannidis, a specialist in metascience studies at Stanford University in California, says the change in national self-citations doesn’t seem particularly extreme, and might be due to chance.
He added that if the increase is real, it could have been driven by a minority of researchers. But Marco Seeber, a sociologist at Ghent University in Belgium who previously reported a rise in Italian self-citations, thinks that Italy’s policies bear a lot of responsibility for the growth of inwardness that Baccini’s team found.
Baccini says academics in every country that relies partly on bibliometric targets might be feeling similar pressures, but the effects have been highlighted in Italy by the sudden introduction of metrics-based promotion thresholds.
The country could consider excluding self-citations from its evaluations, or lowering citation or productivity thresholds to put more focus on the quality of research, he says.
This article was first published on nature.com. Read the Original article.
Read next:
Italian scientists increase self-citations in response to promotion policy
Frequent collaborators are less likely to produce replicable results