An alternative metric for Chinese social media chatter

Researchers develop a tool for tracking academic activity and impact on WeChat.

27 November 2017

BJI/Blue Jean Images/Getty

A new metric seeks to address the disadvantage to Chinese researchers caused by their government’s block on global social media platforms such as Facebook and Twitter. Chinese researchers cannot compete with their Western counterparts on impact measures which count social media interactions, so two Chinese researchers are attempting to balance the scores using WeChat, China’s largest social media platform.

In their Scientometrics study, Zhao Rongying, an informatics researcher in the Research Center for Chinese Science Evaluation of Wuhan University, and colleague, Wei Mingkun, set out to assess the impact of public academic accounts on the platform, which is used by some scientists and scholars to disseminate their own or colleagues’ scientific achievements.

Although WeChat had 889 million monthly active users by the end of 2016, it is not recognized by Altmetric, the leading alternative metric. Altmetric tracks Wikipedia citations, social media attention, mainstream media reporting and academic blogs and other digital sources to produce a non-citation metric for research outputs. Almetric announced in March 2014 it would track Sina Weibo, the Chinese equivalent to Twitter. However the coverage of Sina Weibo was discontinued in July 2015.

The prestigious Zhejiang University revealed in September that its faculty members’ highly popular posts (with at least 100,000 readings) on social media platforms like WeChat can be counted as an academic journal publication. The policy has been greeted with both praise for encouraging scientists’ outreach, and concern over the difficulty of formulating appropriate metrics for social media impact.

The two bibliometrics scholars, Zhao and Wei, decided to use the number of readings and the number of ‘thumbs-up’ responses to measure the impact of academic accounts on WeChat. They chose the top 20 active academic accounts from accounts monitored by WeChat data measurement site Qingbo Index. For their impact metric, the researchers gave an 80% weight to readings and 20% weight to thumbs-up responses.

For the readings, they assigned 40% value to average daily views of all articles in an account, 45% to average views of each article during the studied period (two weeks), and 15% to the number of views of the most read article in the account in the study period. For the praise measure, the same proportion was given to the average daily thumbs-up for all articles, average thumbs-up for each article in the studied period, and 15% to the number of thumbs-up of the most praised article.

The impact measurement revealed big distances between the top 20 WeChat academic accounts, from the top-ranked Lilac Doctor (Wechat ID: dingxiangyisheng) at 1,367 to the Machine Learning Research Society at number 20 with a score of 380. The scholars also found a high association between reading and clicking the thumbs-up sign in the top 20 academic WeChat accounts.

More popular topics attract more attention, increasing their influence still further, say the scholars. “The study result can also be helpful to integrate WeChat to Altmetric evaluation, though unlike Weibo, WeChat accounts are more operated by a group instead of individual academics,” Zhao told the Nature Index.

Li Yan, a Wiley editor based in Beijing who has long studied Altmetric and other alternative metrics, says that while an impact index for WeChat might reflect the impact of the research reported on the platform, it could also be easily manipulated, “which might distort impact measurements.” Popular software used to ‘game’ audience figures registers false WeChat IDs to ‘read’ articles.

“Moreover, people tend to read and forward eye-catching research, but the integrity and accuracy of the original data can be easily overlooked,” Li says. He says the WeChat index could be used as supplementary data, “but should not be treated as the primary metric to evaluate the impact of an academic paper or the author(s).”

Wang Dapeng, an assistant professor at China Research Institute for Science Popularization, who studies the relationship between science and media, added that many academic WeChat accounts publish popular science articles rather than the latest research papers. More work is needed if the goal is to reliably evaluate the impact of scientists and their research output, he says.