Neuropsychopharmacology (NPP): update on relationships between online attention and citation counts

Recognizing that online attention can enhance journal visibility, Neuropsychopharmacology (NPP) leadership launched social media activities in 2017. The NPP Social Media Editor’s primary role is to engage our readership and the ACNP community by promoting journal content. In 2019, we published a report that examined relationships between online attention metrics and citations in NPP [1]. Specifically, we explored whether there are strong correlations between a paper’s Altmetrics scores, or some of the individual metrics that Altmetrics scores are designed to reflect, and its citation counts. Altmetrics scores are calculated based on a proprietary algorithm that proportionately weights online attention derived from an amalgamated variety of sources, including Twitter, news platforms, Reddit, LinkedIn, Wikipedia and Facebook. We initially examined content published in print in 2017, which was the most recent year for which all articles had at least one full year (i.e., all of 2018) to accumulate citations. Our results revealed that there were no statistically significant relationships between overall Altmetrics scores and citations. However, there were statistically significant positive correlations between Twitter mentions and citations of NPP source materials (i.e., original content for which an abstract was available). Specifically, Twitter mentions accounted for ~26% of the variance in the total number of citations of these types of articles. News mentions—another highly desirable form of visibility—were not correlated with citation counts for source materials in the data set used for our initial report [1].

Limitations of our initial report [1] included the relative newness of Altmetrics as a metric and the fact that citations had only 1 year to accumulate. Here we present a follow-up report designed to examine if relationships between online attention and citations change over time, as both the metric and the content mature. To provide these insights, we performed two types of analyses. First, we performed a re-analysis of the 2017 articles using data from 2019—reflecting the second full year after publication in print—to examine if relationships strengthened or weakened with the passage of time. Second, we performed an analysis of content published in print in 2018, using data from 2019, to determine if the 1st-year relationships discovered in our initial report [1] show stability from year to year.

Analyses were restricted to NPP content published in print in 2017 or 2018. Data were harvested on the same calendar date that was used to record data for our initial analysis [1], to ensure consistency in methodology. As before, online attention metrics were collected for the top 50% highest Altmetrics-scoring content qualifying as source materials, which we defined as original articles for which an abstract is available. Analyses were limited to the top 50% because Altmetrics scores and citation counts trended towards 0 for articles below this limit, potentially introducing skew in the data that could confound our interpretations. In addition, the Top 50% is a data set that can be readily collected for journals of any size, facilitating year-to-year tracking and/or comparisons with other journals. Individual article titles were cross-referenced to the Institute for Scientific Information (ISI) Web of Science search engine (http://www.webofknowledge.com) to determine citation counts in each year. Data were collected separately for all “front-half” materials, which we defined as content not associated with an abstract (e.g., Editorials, Perspectives, Research Highlights, Commentaries, “Hot Topics” articles). Pearson’s product-moment correlations were applied to quantify relationships between Altmetrics scores, Twitter mentions (including re-Tweets and quoted Tweets), news media mentions and citation counts. To compare the strength of the relationships between the two sets of analyses (i.e., initial [1] and current), Fisher’s r–Z transformations were performed on correlation coefficients.

In our “re-analysis” of NPP source materials (i.e., articles with an abstract) published in print in 2017, we saw for the first time a statistically significant relationship between citation count and Altmetrics scores (Pearson’s r = 0.185, P < 0.05; Fig. 1A), and the correlation with Twitter mentions remained strong (r = 0.52, P < 0.001; Fig. 1B). We continued to see no relationship between citation counts and news mentions (Fig. 1C). Despite the emergence of a significant effect with Altmetrics scores, Fisher’s r–Z transformation revealed that the strength of the relationships for 2017 source materials did not change significantly between the initial report [1] and the current analysis for any of the measures. For our new analysis of NPP source materials published in print in 2018, there were significant correlations between citation counts and Altmetrics scores (r = 0.28, P < 0.001; Fig. 2A) and Twitter mentions (r = 0.27, P < 0.001; Fig. 2B), although for this year a correlation with news mentions also emerged (r = 0.28, P < 0.001; Fig. 2C). Correlations between citations, Altmetrics scores, Twitter mentions and news mentions remained statistically significant after removal of a putative outlier (boxed data point, Fig. 2; Altmetrics: r = 0.25, P < 0.001; Twitter r = 0.28, P < 0.001; news: r = 0.21, P < 0.004). Fisher’s r–Z transformation revealed that there was no difference in the strength of the relationship between overall Altmetrics scores when the initial analysis of the 2017 articles was compared with the initial analysis of the 2018 articles, even though only the 2018 correlation was statistically significant. However, there were statistically significant differences in the strength of the relationships between Twitter mentions and citations when comparing 2017 versus 2018 source materials (Z = 2.5 (2017 initial versus 2018 initial); Z = 2.6 (2017 re-analysis versus 2018 initial); P’s < 0.01). Correlations for Twitter were stronger with the 2017 in-print content, regardless of whether the correlation coefficients from the new analysis of the 2018 articles were compared to the initial report [1] or to the re-analysis of the 2017 articles. Differences in the strength of relationships between news mentions and citations also trended towards significance for 2017 versus 2018 source content (Z = −1.9 (2017 initial versus 2018 initial), P = 0.07; Z = −1.8 (2017 re-analysis versus 2018 initial), P = 0.06). The relationship between news mentions and citations was stronger for 2018 in-print content than for 2017 in-print content, regardless of whether the correlation coefficients from the new analysis of the 2018 articles were compared to the initial report [1] or to the re-analysis of the 2017 articles.

Fig. 1: Citations and online attention metrics for NPP source materials (original reports and reviews) published in print in 2017 (n = 118).

A Correlation between Altmetrics scores and total number of citations, as identified on the ISI Web of Science. B Correlation between mentions on Twitter and total number of citations. C Correlation between mentions on news platforms and total number of citations. Closed data points, black solid lines and text represent re-analysis of 2017 in-print source materials. Open data points, grey dotted lines and text represent relationships between 2017 in-print source materials (n = 118) published in our original report, for comparison.

Fig. 2: Citations and online attention metrics for NPP source materials (original reports and reviews) published in print in 2018 (n = 190).

A Correlation between Altmetrics scores and total number of citations, as identified on the ISI Web of Science. B Correlation between mentions on Twitter and total number of citations. C Correlation between mentions on news platforms and total number of citations. Closed data points, black solid lines and text represent initial analysis of 2018 in-print source materials. Open data points, grey dotted lines and text represent relationships between 2017 in-print source materials (n = 118) published in our original report, for comparison. Boxed data point indicates a putative outlier; removal of this data point did not change statistical outcomes.

For re-analysis of 2017 front-half materials (i.e., content without an abstract), we also saw for the first time a statistically significant relationship between citation count and Altmetrics scores (r = 0.42, P < 0.01; Fig. 3A). We continued to see no relationships between citations and Twitter (Fig. 3B), whereas the previously observed correlation between citations and news mentions was now absent (Fig. 3C). Despite changes in the statistical significance of effects with Altmetrics scores and news mentions, Fisher’s r–Z transformations revealed that the strength of these relationships did not significantly change for any of the measures. For our new analysis of 2018 front-half materials, there was a significant relationship between citations and Altmetrics scores (r = 0.87, P < 0.001; Fig. 4A) and Twitter mentions (r = 0.82, P < 0.001; Fig. 4B), but not news mentions (Fig. 4C). Correlations between citations, Altmetrics scores and Twitter mentions remained statistically significant after removal of a putative outlier (boxed data point, Fig. 4; Altmetrics: r = 0.58, P < 0.001; Twitter r = 0.44, P < 0.006), and a significant relationship further emerged between citations and news mentions (r = 0.5, P < 0.002). There were statistically significant differences in the strength of the relationships between Altmetrics scores and citations when comparing 2017 versus 2018 front-half materials (Z = −9.2 (2017 initial versus 2018 initial); Z = −7.5 (2017 re-analysis versus 2018 initial); P’s < 0.001) and Twitter mentions and citations (Z = −10.64 (2017 initial versus 2018 initial); Z = −7.3 (2017 re-analysis versus 2018 initial); P’s < 0.001). The associations between these online attention metrics and citations for in-print front-half content were stronger for 2018 than 2017, regardless of whether the correlation coefficients from the new analysis of the 2018 articles were compared to the initial report [1] or to the re-analysis of the 2017 articles.

Fig. 3: Citations and online attention metrics when content is restricted to NPP front-half materials (editorials, perspectives, commentaries and hot topics) published in print in 2017 (n = 36).

A Correlation between Altmetrics scores and total number of citations, as identified on the ISI Web of Science. B Correlation between mentions on Twitter and total number of citations. C Correlation between mentions on news platforms and total number of citations. Closed data points, black solid lines and text represent re-analysis of 2017 in-print front-half materials. Open data points, grey dotted lines and text represent relationships between 2017 in-print front-half materials (n = 36) published in our original report, for comparison.

Fig. 4: Citations and online attention metrics when content is restricted to NPP front-half materials (editorials, perspectives, commentaries and hot topics) published in print in 2018 (n = 38).

A Correlation between Altmetrics scores and total number of citations, as identified on the ISI Web of Science. B Correlation between mentions on Twitter and total number of citations. C Correlation between mentions on news platforms and total number of citations. Closed data points, black solid lines and text represent initial analysis of 2018 in-print front-half materials. Open data points, grey dotted lines and text represent relationships between 2017 in-print front-half materials (n = 36) published in our original report, for comparison. Boxed data point indicates a putative outlier; removal of this data point did not change statistical outcomes with the exception that a significant relationship also emerged between citations and news mentions.

These analyses provide several take-home messages relevant to the function of NPP. First, the strong correlation between citations to source materials (i.e., articles with abstracts) and Twitter mentions seems replicable from year to year. Second, this relationship does not appear to change rapidly over time, such that an analysis performed after articles have been in print for one full calendar year seems as conclusive as an analysis performed after two full calendar years. Third, the relationship between citations and both overall Altmetrics scores and news mentions seems less reliable, reaching statistical significance in some years but not others. As previously acknowledged [1], it is not possible to infer cause-effect relationships from these data sets: it seems equally plausible that online visibility drives citations, or alternatively, that online visibility simply reflects the destiny of articles that would be highly cited anyway. In addition, it is unknown if these types of relationships would be seen with other journals. Regardless, social media is one avenue by which discoveries published in NPP, as well as awareness of issues pertaining to diversity and equitability, can be promoted to enhance engagement and nurture dialogues among the journal, investigators and the broader community.

Funding and disclosures

All of the authors have roles at NPP: CJJ is an Editorial Intern, KM is the Social Media Editor and WAC is the Editor-in-Chief.