Instant social-media interest in a new paper does not predict its impact on future publications. Credit: esenkartal/iStock Vectors/GETTY IMAGES

The most popular research articles mentioned on social media in 2013 are studies on caesium contamination of fish off the coast of eastern Japan, on how a Mediterranean-type diet prevents cardiovascular disease, and on the direct imaging of atomic bonds.

But the top-100 list, published yesterday by the company Altmetric, may not reveal the papers that scientists will find most influential in future as some had hoped. A recent study reveals that the amount of buzz a paper gets on Twitter bears little resemblance to the impact it will have in terms of academic citations in later years.

“Twitter citations do not reflect traditional research impact,” says Stefanie Haustein, an information scientist at the University of Montreal in Canada who is lead author on the study, published two weeks ago in the Journal of the Association for Information Science and Technology1.

Dyslexia linked to brain communication breakdown NASA funding shuffle alarms planetary scientists Thin, weak layer of clay was behind Japan earthquake and tsunami

Dissatisfaction with the use of traditional metrics of academic value has led to increasing interest in alternative metrics, or 'altmetrics', which gauge attention on research papers by means such as downloads, social-media mentions and collections in online libraries. Companies such as Altmetric or Plum Analytics now provide these measures commercially. Some researchers who look at such indices had hoped that they may provide a real-time view of the most exciting papers, something that citations, which take time to gather, cannot offer.

Haustein's team used Altmetric’s software to analyse more than 340,000 tweets from 2010 to 2012 that mentioned around 135,000 unique biomedical research articles, around 9.4% of the 1.4 million biomedical papers published in the past three years.

Among papers tweeted about in 2011, Haustein found very weak correlation between the number of tweets papers received and the number of citations they would later accrue. “The top-tweeted articles are a mix of curious stories, funny articles or those with medical implications,” she says.

Euan Adie, the founder of Altmetric — which is part-owned by Digital Science, a sister company of Nature Publishing Group — says that he is not surprised. “I don't think that tweets by themselves are a robust leading indicator of citations. I don't think it's a good idea to spin them as such either. I do think that in combination with other data sources they're a good indicator of attention and dissemination into different audiences,” he says.

A similar study could produce different results for later years. In 2011 Twitter was still in its infancy, but the picture is changing fast: by 2012, 20% of biomedical articles had been tweeted at least once.

Users of altmetrics also recommend looking at a wide variety of measures, not just Twitter cites. Haustein says that in a further study, which has not yet been published, her team found that reader views for papers on Mendeley, an online academic reference manager, were more closely linked with later citations. The correlation was still only moderate, but this finding suggests that analysing tweets by scientists, rather than the general public, might produce a different outcome.

Adie adds that people tweeting or sharing articles are not necessarily broadcasting the relevance of the articles to ongoing research, as citations are thought to do. “That's the point of collecting altmetrics; the hope that they provide some insight beyond the things we usually look at. It's interesting to look at top-tweeted articles because that's evidence that there was something about them worth sharing,” he says.