Has a research paper just criticized your work? Take it as a compliment. This kind of formally-expressed disagreement is rare and only the more highly-cited studies tend to attract it, according to the first large-scale effort to analyse 'negative citations' and their effects1.

Citations are generally craved by authors, but may be less welcome when they come as part of a critical comment or contradictory finding. The rate at which these negative citations appear in the literature is low but not negligible, according to the study, led by Alexander Oettl, an economist at the Georgia Institute of Technology in Atlanta.

The top 100 papers: Exploring the most-cited research of all time

Oettl’s team checked through more than 760,000 citations (referring to around 150,000 papers) from nearly 16,000 articles in the Journal of Immunology. They asked a team of immunologists to help them by manually classifying 'negative' citations from their context in around 15,000 examples. Wording along the lines of “The data therefore contrast with reports that …” or “This conclusion appeared inconsistent with …” was regarded as negative, for instance. Then, the researchers used automated text analysis, based on an algorithm trained with the manually-classified dataset, to analyse the rest.

They found that 2.4% of the citations were “negative”, and that some 7.1% of the more than 150,000 papers cited received at least one negative citation.

Criticize what's important

The papers that tended to receive criticism were ones that received a relatively large number of citations overall — which is usually taken as an indicator of a paper’s quality and significance. And the “penalty” incurred by a negative citation seems only slight in bibliometric terms. A paper that attracts a negative citation goes on to receive slightly fewer citations than does a similar paper that is not criticized, but the difference only becomes apparent after eight years. The study is published1 in the Proceedings of the National Academy of Sciences.

Is your most cited work your best?

That negative citations are less likely to be directed at low-impact papers fits within philosopher of science Thomas Kuhn’s framework of "normal science”, says Oettl — a mode in which science advances incrementally through peer assessment and filtering of claims. “Scrutiny should be directed towards the most important ideas so that they can be further refined”, he says. “Low-impact ideas receive less scrutiny, as overturning a trivial or minor finding does not drastically shift the scientific frontier.”

The analysis also suggests that negative citations are most likely to come from other researchers who are close in disciplinary affiliation and “social distance” to the critiqued authors — which is probably no surprise — but relatively far in geographical distance. The latter suggests that social factors — the awkwardness of contradicting someone whom you are likely to bump into in person — may play a part in science’s self-scrutiny.

Michael Schreiber, a physicist and specialist in bibliometrics at the Technical University of Chemnitz in Germany, says that the figures are in line with what he would have expected from earlier informal estimates, although he has not previously seen them quantified in this manner.

However, Schreiber thinks that the study’s definition of a negative citation might be too broad. It would include some apparent negative ­self-citations — for example, situations in which authors discuss the methodological limitations of their own earlier papers, to highlight their new advance.

Not necessarily negative

Ludo Waltman, who studies bibliometrics at Leiden University in the Netherlands, agrees that the definition of negative citations used by Oettl’s team is broad and need not imply disagreement with earlier work. “Having differences in empirical results is a natural phenomenon in science and shouldn’t necessarily be interpreted negatively,” he says.

‘Sleeping beauty’ papers slumber for decades

Oettl, however, thinks that his figures are likely to represent the lower bounds of how often later studies disagree with previous work. Researchers will be wary of explicitly criticizing a paper in a formal publication, he notes, as making an incorrect criticism could be socially and professionally costly. What is more, he says, critiques can also appear in other channels, such as at conferences and in private exchanges.

Oettl is planning to use the same approach to look at negative citations in other fields. And Schreiber suggests that such studies might be extended to automatically examine which papers attract the most negative citations. “This might indicate which papers are false or at least controversial,” he says.

“By having a tool for identifying controversial work in science,” adds Waltman, “we could learn more about the nature of this type of work and the mechanisms that cause publications to become controversial.”

10 principles to guide research evaluation