Introduction

Artificial intelligence (AI) is going to fundamentally reshape how we work and live. The transformative impact of AI has in fact already begun—from emotion-recognition applications and sensors in advertising and marketing, to autonomous cars and home assistants, to workplace recruitment and automated management, to mood self-tracking devices and healthcare robots, to predictive policing and algorithmic sentencing, to border control and autonomous weaponry in war. Two of the key countries driving the research and development of AI are the US and China. However, despite similar AI objectives and vision both nations appear, at least politically and publicly, to be engaged in a cold war and as a result, are growing less cooperative and more adversarial (Martin, 2021). Indeed, if the techno-geopolitics of the twentieth century can be defined by the Arms Race and Space Race between the US and Russia, the early twenty-first century may be remembered as the AI race between the US and China. Importantly, while the history of research and development of AI can be attributed to Western pioneers such as Alan Turing and John McCarthy or American-based think-tanks such as RAND, China’s meteoric economic and military rise is paralleled by their prolific output in cutting-edge AI research and development (Wu et al., 2020). Although the US was the frontrunner of the AI race in supercomputing, it is clear that China’s AI research initiatives are beginning to outstrip all of its Western competitors (Martin, 2021). At the same time, neither country, or for that matter members of the international community has reached a consensus on the values that should be embedded in AI decisionism. This issue is even more complicated due to fundamental ideological, political, and cultural differences between the liberal West and authoritarian-leaning China, especially in relation to data privacy, public welfare, human security, and civil rights (ÓhÉigeartaigh et al., 2020; Thornhill, 2019).

The rise of emotional AI

Affective computing is a growing multidisciplinary field that draws on computer sciences, engineering, psychology, physiology, philosophy, and even neuroscience. The term was coined by Rosalind Picard in 1995 to mean “computing that relates to, arises from, and deliberately influences emotion” (Calvo et al., 2015: p. 13). As human lives become increasingly assisted and even dominated by technologies and coded spaces within emerging smart cities, there is a need to better understand the role of human affect in human-computer interactions. The term “emotional AI” designates a branch of affective computing that combines artificial intelligence, biosensors, and deep learning algorithms to sense, track, and classify human emotions and affective states (McStay, 2018). Given that affective computing is the foundation for the development of emotional AI, and emotional AI has now been used widely to refer to the commercial applications of affective computing, in this article, we use the two terms interchangeably.

Even though emotion-sensing technologies have not reached technological maturity, their commercial applications have already become pervasive in our daily life, blossoming into a growing global industry worth USD 21.6 billion that is expected to more than double in valuation by 2024 (Crawford, 2021). As a nascent industry, creative applications of the technology are constantly evolving. In the field of education, smartphone apps such as ClassDojo provide teachers with psycho-metric profiles of students, allowing them to score and reward positive behavior while giving a lesson (Williamson, 2021). The smart toy Moxie, meanwhile, assists with a child’s emotional, social and psychological development (Lyles, 2020). In the world of entertainment, Spotify’s emotion-recognition algorithms can suggest playlists by sensing a person’s mood (Ratliff, 2016). Moreover, in response to an increase in elderly driver car accidents in Japan Honda and Softbank co-created a bio-sensing ‘Emotion Engine’ which detects if a driver is drowsy, distracted, or stressed (Dery, 2018).

In the workplace, emotional AI is an emerging layer in “human-centric”, automated management systems and data-driven corporate wellness programs (Elsden et al., 2016; Brassart Olsen, 2020; Cardon et al., 2021). Proponents of emotion-recognition technology claim that, besides inducing higher productivity, emotional AI can reduce turnover rate and produce more harmonious social dynamics. For example, legacy companies such as IBM, Softbank, and Unilever are now using affect tools for recruitment, as well as monitoring the productivity of their workforce. Emotional AI companies such as Empath in Japan market voice-analytic software that allows call center managers to monitor employee moods in real-time. Hazumu Yamazaki, Empath’s chief strategic officer, asserts that their technology dramatically reduces turnover because it allows supervisors to make informed decisions about their employees as they are privy to their emotional states (Yamazaki H., personal communication, 2021). For cash-strapped start-ups interested in low-cost wellness initiatives, Amazon provides a wrist wearable biosensor called “Halo” that tracks a users’ emotional state, detecting depression, anxiety, and even early signs of mental illness (Graziosi, 2020). Similarly, the UK company, Moodbeam markets what they call a “mood awareness” wearable that monitors an employee’s ‘happiness’ level and then shares the daily findings with co-workers (Bearn, 2021). Moreover, in response to spiraling stress-related absenteeism at work caused by COVID, Microsoft has announced plans to use emotion-sensing devices in their worldwide offices to track wellbeing (Spataro, 2020).

On the one hand, proponents of emotional AI argue that algorithmic management can help businesses create meaningful work experiences by allowing individuals to showcase their talents and achievements while simultaneously realizing their organizational goals (McStay, 2018; Gal et al., 2020). They insist that AI-driven performance management can provide objective and constant feedback to employees, something busy human supervisors find difficult to make happen. Researchers also point out that the more a talented employee feels heard and understood, the more his or her engagement scores rise (Baumruk, 2006) On the other hand, a growing body of literature on AI adoption points towards a more oppressive form of digital Taylorism. For example, workers placed under biometric surveillance suffer higher degrees of technostress and burn-out (Mahapatra and Pati, 2018; Khedhaouria and Cucchi, 2019; Dragano and Lunau, 2020), loss of dignity, autonomy, and trust (Jeske and Santuzzi, 2015; Tomczak et al., 2018). Second, emotion-sensing AI, especially those that rely primarily on facial biometrics, has been shown to be widely inaccurate when dealing with gender, ethnicity, and race (Buolamwini and Gebru, 2018). Third, as AI technologies move across national borders, the fact that their algorithmic design and datasets are seldom tweaked for cultural diversity heightens the dangers of misuse, inaccuracy, and bias (Mitchell, 2019).

Added to the above problems is the lack of uniform and consistent international agreements on data collection. On the one hand, many Western countries (apart from the US) are strengthening data privacy legislation to protect citizens. For example, besides Europe’s General Data Protection Rules, as of April 14, 2021, a proposal entitled ‘Regulation on a European approach for artificial intelligence’ has been put forward (Vincent, 2021). The proposal seeks to regulate the collection of non-conscious data for emotion-recognition AI as Article 42(2) requires “any natural person whose personal data is being processed by an emotion-recognition system or a categorization system shall be notified that they are exposed to such a system” (p.47). On the other hand, in other world regions such as the Global South, weak data privacy laws and regulation have created a twenty-first century environment predicated on a rush to harvest data—the new gold. Global market players are descending on countries such as Thailand, India, Malaysia, and Japan to exploit not simply legal but also cultural loopholes to collect the precious and lucrative datasets from unsuspecting populations (Henning, 2019; Tang, 2019). Concurringly, data privacy expert Hiroshi Miyashita observes, “in Japan, the law says almost nothing about the collection of non-conscious or health data of individuals in the context of the workplace. This is due to our cultural norms that dictate respect for authority and the long-standing tradition of trust between employer and employee” (personal communication, 2021). The lack of a strong privacy regulation in the Asian workplace can be explained in a Confucian context, where studies on Confucianism highlight a strong emphasis on harmony, duty, and loyalty to the collective will (Vuong, 2020; Vuong et al., 2018; Chung et al., 2008). As a result, individual rights and privacy trail behind the importance of the collective good (Weatherley, 2002; Whitman, 1985). This may explain the absence or complete absence of pushback of facial recognition technology in many East Asian countries such as China (Mac Síthigh and Siems, 2019).

As outlined above, the rise of affective computing technology poses a myriad of ethical, legal, and cultural challenges, prompting our investigation into the history of the field using bibliometric analysis. As one of the most actively researched topics, the number of scholarly articles on ‘affective computing’ is increasing rapidly, making it virtually impossible to keep up to date with every publication (Tao and Tan, 2005). Thus, we deploy bibliometric analysis on a dataset of 3386 Web-of-Science-indexed academic articles from 1995–2020. As a research method, bibliometric analysis is widely used to provide a broad and quantitative overview of a given field such as AI applications in medicine (Tran et al., 2019), interdisciplinary robotics research (Michalec et al., 2021), global evolution of digital marketing communication (Kim et al., 2019), conversational agents and chatbots (Io and Lee, 2017), etc.

As a research domain with highly practical value, bibliometrics has evolved considerably from library management in the 1950–60s through science policymaking in the 1970s, to research evaluation in the 1980s and onward (Ellegaard and Wallin, 2015; Gingras, 2016). Bibliometrics is related to, though not synonymous with, scientometrics—which studies the various facets of scientific production, such as growth, productivity, socio-organizational structures, interrelationships between science and scholarship, and between science and technology, and governmental policies (Hood and Wilson, 2001). In recent years, the development of bibliometrics and scientometrics has greatly benefited from the declining cost of computation and the open science movement (Light et al., 2014; Vuong, 2020), where code, data, and community are made available to the public (Vuong, 2017). Consequently, open resources allow researchers to collaborate and advance visualization and analysis methods for science mapping and incorporate these into user-friendly, publicly available software packages such as bibliometrix (Aria and Cuccurullo, 2017) or VOSviewer (Van Eck and Waltman, 2010). Given that the purpose of this study is to map the research landscape of affective computing over the past 25 years, i.e., the field’s growth rate, major players, major collaborative networks, and thematic evolution, bibliometric analysis is proven to be a suitable method.

Methods

We searched for the keyword ‘affective computing’ in the Web of Science database for the period from 1995 (i.e., when Rosalind Picard first coined the term) to 2020. The purpose is to look for English-language publications on the topic, whether they are journal articles, books, book chapters, review articles, or editorial materials. The search query, conducted on April 5, 2021, is as follows.

Step

Query

Results

#1

TOPIC: (affective computing) Indexes = SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI Timespan = 1995–2020

3527

#2

TOPIC: (affective computing) Refined by: LANGUAGES: (ENGLISH) Indexes = SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI Timespan = 1995–2020

3469

#3

After manually screening and removing articles published in 2021 and duplicate articles

3448

The results yield a total of 3469 items. Early access materials formally published in 2021 are excluded, resulting in the final list of 3448 items. Bibliometric analysis was conducted using the open-source bibliometrix package in R, developed by Aria and Cuccurullo (2017). The package’s web interface app, biblioshiny, enables users to perform analysis at three levels of analysis, namely sources, authors, and documents, as well as to map the structures of knowledge in conceptual, intellectual, and social terms.

Results

Affective computing at a glance

The overall annual growth rate of scientific production is 11.36%. Figure 1 illustrates the annual scientific production. The two red dash lines serve as markers for each decade to be examined in detail. Specifically, the three periods are: from 1995 to 2005, from 2006 to 2015, and from 2016 to 2020. The decline of total research output in 2020 might be due to the impacts of the COVID-19 pandemic on academia, where many conferences were canceled and researchers in lockdown found it difficult to adjust their work behaviors.

Fig. 1
figure 1

Annual scientific production on ‘affective computing’, 1995–2020 (Source: Web of Science).

Full size image

Table 1 presents the main statistics of the publications on ‘affective computing’ for 25 years, from 1995 to 2020.

Table 1 Descriptive statistics of research on ‘affective computing’ in 1995–2020 (Source: Web of Science).
Full size table

In terms of authorship, the raw data show that the documents were written by a total of 8998 unique authors, with the ratio of documents per author at 0.383. Of these documents, approximately 9.74% were written by a single author, while the remainder were written by multiple authors. A ratio of 2.61 for authors per document means that a paper on ‘affective computing’ was written on average by 2.61 authors. Given that publications in which the number of authors is larger than usual can skew the metrics, the figure for co-authors per document, which is 3.87 here, takes into account the number of times an author appears in the whole set of documents. Both measures suggest that, on average, two to three authors were involved in one publication on ‘affective computing’ during the 1995–2020 period. A collaboration index of 2.81, measured by dividing the total number of authors of multi-authored documents by the total number of multi-authored documents, confirms the aforementioned findings.

Table 2 lists the ten countries with the most publications on ‘affective computing’ in the analyzed period. It compares country output in terms of journal articles and total publications, which include conference proceeding papers. The findings show that the top seven countries—namely the US, China, UK, Germany, Spain, Italy, and the Netherlands—remain unchanged in their ranking, indicating that scholars from these countries are actively engaged in both journal publication and conference participation. Authors from India seem to be similarly prolific in publishing with journals and conferences. Meanwhile, it is noteworthy that when conference proceedings are taken into account, Japan emerges in the top ten prolific countries in the field, suggesting that Japanese scholars, in comparison to Australian scholars, may be more active in conference participation than in journal publication. This discrepancy between conference presentations as opposed to publication output as well as the significant absence of some countries (such as Russia) from our findings, are discussed later in this article.

Table 2 Total scientific output on ‘affective computing’ of the current top ten countries in three periods.
Full size table

Table 3 presents the top ten most-cited countries. The country is determined based on the country affiliation of the corresponding author. The US topped both lists, accounting for 40% of the total research output and 35.58% of the total citations. This speaks volumes to its position as the pioneering destination for research on affective computing. China came second to the US in terms of publications but trailed behind the United Kingdom and Germany in terms of citations.

Table 3 Total citations of the top ten countries in three periods.
Full size table

Meanwhile, Spain, France, and Japan were present in the top ten most productive countries by total publications but were not among the top ten most-cited countries—the two positions occupied by Switzerland and Singapore. Overall, the presence of China, Singapore, Japan, and India as the only four non-Western countries on the lists point to a shift of research destinations away from the formerly dominant Western nations. As shown in Table 2, in the first decade that affective computing research was in its nascent stage, the United States was the major player, accounting for nearly 40% of the total output, whereas China and India were absent in the top ten. However, this situation dramatically changed within the next two periods: China overtook the United Kingdom as the second most productive in 2006–2015 and later also surpassed the US as the most productive in 2016–2020. The number of relevant countries by corresponding authors in the collection also jumped dramatically from just 30 in the first period to 61 and 65 in the last two periods.

Table 3 provides further empirical evidence for the emergence of Switzerland, China, and Singapore in 2006–2015 as three influential research hubs on affective computing. Research articles published by China were increasingly cited worldwide in the recent decade, allowing the country to become the second most cited after the US in the 2016–2020 period.

Collaboration networks

Given that AI researchers seldom work alone and increasingly collaborate outside their home institutions as well as home countries, it is also important to examine the networks of countries involved in international research collaborations.

Table 4 shows that the UK and Germany have the highest rates of multiple-country publications in the top ten most productive countries, at 32.41% and 31.98%, respectively, while India has the lowest, at just 8.49%. This means that British and German authors have many works published with international collaborators, as opposed to Indian authors who tend to publish amongst themselves. China, the Netherlands, Italy, and Spain followed Germany in terms of internationally collaborated publications. By comparison, the US, Japan, and Canada have a relatively lower rate of multiple-country publications, indicating that authors from these countries largely work with their national peers.

Table 4 Publications by single-country and multiple-country collaboration in the top ten most productive countries.
Full size table

Figure 2 provides a visualization of the collaboration network among the top 23 productive countries, in which the bigger the text is, the more numerous are co-authored publications of that country. The networks bring into sharp relief three distinct clusters of affective computing research. Figure 2 measures of importance in network analysis via: (i) betweenness centrality, which measures the number of shortest paths that a node bridges between other nodes, and (ii) closeness centrality, which takes into account the shortest paths between all nodes. Here, the United Kingdom scores the highest in both measures (59.58 and 0.034, respectively), followed closely by the US (48.94 and 0.034).

Fig. 2
figure 2

Networks of co-authored publications by countries.

Full size image

The US and China are the two largest nodes in the purple hub of 11 countries, while the United Kingdom and Germany are the two principal nodes in the red hub of nine countries. What can be seen from Fig. 2 is the rise of scientific output in China, Singapore, and India as shown in Table 2. This is also linked closely to the increased collaboration with authors from the US. The transfer of knowledge is taking place from researchers of more established to those of emerging economies. Meanwhile, the third cluster, with the exception of Canada, consists of all European countries, with Italy and Spain having a higher count of publication output.

Table 5 shows the percentage of a country’s involvement in multiple-country publications, with data from the top five most productive countries. The percentage indicated belongs to the country listed at the beginning of the row. The UK-Germany is the strongest tie, as 84.15% of collaborative papers from the UK involve German authors and 97.18% of Germany’s involve British authors. The US-China tie ranks second. Chinese and American authors collaborate in the majority of their multiple-country collaborative papers: 63.53% of the US’s multiple-country publications (54/85) involve Chinese authors, and the ratio is 52.94% for China (54/102). The US-UK collaboration takes third place (49.41 and 51.22%).

Table 5 Percentage of a country’s involvement in multiple-country publications of the top five most productive countries.
Full size table

Author and article characteristics

Of the 8998 unique authors listed in the corpus, there are certain authors who have published consistently on ‘affective computing’ over the decades. Figure 3 presents the top authors’ research output in the 25-year period.

Fig. 3
figure 3

Top authors’ production over time.

Full size image

While MIT’s Rosalind Picard remains the most active throughout the entire period, the year 2014 marks a dramatic increase in the number of Chinese authors entering the top authors list. The rise of China as an emerging powerhouse in affective computing research is evident in Tables 6 and 7. Table 6 presents the most productive authors by period. Two researchers Bjoern Schuller, from Imperial College London, UK, and Erik Cambria, from Nanyang Technological University, Singapore dominated the period 2006–2020 in terms of productivity. The works of Schuller primarily focus on affect recognition from audiovisual signals in various real-world contexts (Tzirakis et al., 2017): online lectures (Weninger et al., 2013), natural conversations (Schuller et al., 2009), etc. The works of Cambria primarily focus on sentiment analysis and natural language processing (Young et al., 2018; Cambria et al., 2017). It is noteworthy that all of these pioneers of affective computing have monetized their research results—Picard’s Affectiva and Empatica have commercialized a range of emotional AI tools including wearable sensors and analytics for advertising, driver monitoring and health purposes. Schuller co-founded audEERING, which manufactures emotion-sensing devices for audio. Cambria, meanwhile, co-founded SenticNet which applies state-of-the-art sentiment analysis software for marketing.

Table 6 Most productive authors by period.
Full size table
Table 7 Most local cited authors by period.
Full size table

Table 6 also shows that, for a 20-year period only two Chinese authors made the top 10 positions. After 2016, however, there were three. In the top 3 of the period 2016–2020, is Fuji Ren, a Chinese professor working at Tokushima University, Japan. Ren’s work focuses on various applications of linguistic-based affective computing (Ren and Quan, 2012). He has been involved in a corpus for Chinese emotional analysis (Yan et al., 2008), and more recently, algorithm development for behavior understanding through Wi-Fi channel state information (Jia et al., 2020).

Table 7 presents the most locally cited authors, i.e., the authors that have the most citations within the corpus of 3448 articles analyzed in this study. It can be seen that Rosalind Picard has been among the most influential researchers for about 20 years—from 1995 to 2015. Her work primarily focuses on real-time detecting of stress (Sano et al., 2018), frustration (Klein et al., 2002), even suicidal thoughts (Kleiman et al., 2018) captured through wearable devices (see also No.3 in Table 8).

Table 8 Ten most global cited documents.
Full size table

Thematic analysis and evolution

The conceptual analysis function of bibliometrix allows users to screen all topics and keywords, through which the most important topics and trend topics can be identified. In this section, we probe the conceptual structures that have emerged in 25 years of affective computing research. The bibliometrix software package represents terms that appear together in a document, such as words in titles, abstracts, and keywords of an article as a term co-occurrence network. Our analysis focuses on the top 1000 author keywords, with a minimum appearance in 3 articles.

The co-word analysis allows finding subgroups of highly related terms, which can be represented graphically as four typologies of themes, defined by Cahlik (2000) as follows:

  • The Motor themes: the upper-right quadrant, characterized by high centrality and high density, implies that themes in this quadrant are well-developed central and important for the overall research field.

  • The Basic themes: The lower-right quadrant, characterized by high centrality and low density, implies that themes in this quadrant are important for certain research domains, concern topics that transverse to different research areas.

  • The Niche themes: The upper-left quadrant, characterized by low centrality and high density, implies well-developed themes, which however have a rather marginal role in the development of the field.

  • The Emerging or Declining themes: The lower-left quadrant, characterized by low centrality and low density, implies weakly developed and marginal themes.

It is important to note that in network analysis centrality measures the number of times a node lies on the shortest path between other nodes which implies its importance in the network. Density is the measure of how many connections a node has. For more technical explanations and applications of bibliometrix, see Kolaczyk and Csárdi (2014) and Ho et al. (2017).

Figure 4 provides visualizations of the research themes’ evolution in the field of affective computing. Over the course of 25 years, ‘affective computing’ has moved away from the near upper-right to the lower-right quadrant. In other words, in the earliest period, it was characterized somewhere between a core, well-developed theme and a transversal theme (Fig. 4a). By the period 2016–2020, it has moved to the lowest corner of the lower-right quadrant (characterized by high centrality and high density), which means it has firmly become a core, basic theme (Fig. 4c). On the other hand, ‘facial expression’ moved from being a core theme in 1995–2005 to become a transversal theme (upper-right quadrant, characterized by high centrality and low density) in 2016–2020. This trend highlights an increasing research interest from various domains on ‘facial expression.’

Fig. 4
figure 4

Thematic mapping, 1995–2020.

Full size image

There seems to be a declining interest in affective computing applications in the study or detection of mental illnesses over the periods. Evidently, in the period 1995–2005 (Fig. 4a), themes regarding mental problems such as dementia, depression, anxiety, seasonal affective disorder, bipolar disorder, and schizophrenia were present. Keyword clusters highlight how researchers were strongly interested in studying neuroimaging, brain imaging, magnetic resonance imaging, and magnetic resonance spectroscopy, and computed tomography to better understand different brain parts including the temporal lobe, frontal lobe, frontal cortex, hippocampus, cerebellum, and amygdala. Brain morphology, the study of the structural measures of the brain, was integral to research into affective disorders. The study of the neurobiology of bipolar disorder and schizophrenia was particularly strong throughout the late 1980s and the 1990s, with successful attempts at using structural brain imaging to find evidence for brain abnormalities in patients with psychiatric disorders. Then, except for a cluster of themes related to ‘depression,’ many of these topics no longer featured in visualizations for the subsequent 15 years (Fig. 4b, c).

In the period 2006–2015, ‘sentiment analysis’ rose to prominence with the scope of research expanding into a number of topics such as natural language processing (NLP), sentic computing, facial expression analysis, opinion mining, semantic web, emotion awareness, and gamification. The shift towards interest in sentiment analysis reflects both the emergence of Web 2.0 and social media applications such as AI-powered commercial chat-bots.

In the period 2016–2020, the shift away from mental health issues was driven by more pragmatic commercial applications of affective computing, as illustrated by themes such as the ‘feature extraction,’ ‘artificial intelligence,’ or ‘virtual reality,’ which became core themes in this period. We looked at finer details of words that co-occur with ‘eature extraction’ and found keywords such as ‘pattern recognition,’ ‘features selection,’ ‘EEG,’ and ‘physiological signals’ (See Fig. 5). This implies a growing interest among the academic community to further fine-tune and improve computational techniques and datasets for better performance of affective computing algorithms. It also highlights the sharp increase in computing power and availability of data in recent years.

Fig. 5
figure 5

Factorial mapping, 1995–2020.

Full size image

It is worth noting that advancements in brain-computer interface (BCI) research in studying human emotions and communication are also gaining renewed interest and traction in the field of neuro-medicine and video-gaming. As an emerging branch of affective computing, affective BCI permits users to not only perform automatic detection of emotion but also stimulate emotional states or control external devices using brain impulses instead of the brain’s standard output routs of peripheral nerves and muscles (Charles et al., 2020; Steinert and Friedrich, 2020). For example, Moses et al. (2021) used BCI software equipped with natural word processing algorithms to restore the abilities of a patient suffering from anarthria. The team’s experiment involved attaching subdural multi-electrodes to the region of a patient’s sensorimotor cortex responsible for decoding words and controlling communication. Other emerging affective BCI applications are also being used to better understand how emotions and affective states impact the learning process (Xu et al., 2018).

Future directions of affective computing

There are a few notable recent developments in the field of affective computing and sentiment analysis. The latest trend, sentic computing, pioneered by Singapore-based SenticNet, features the ensemble of symbolic AI (knowledge-based methods which are strongly semantic) and sub-symbolic AI (machine learning methods. neural networks, and deep learning algorithms which take advantage of the rise of computing power, availability of data and cloud storage possibilities) (Susanto et al., 2021; Basiri et al., 2021). This hybrid style combines the top-down approach, which leverages symbolic models and knowledge bases (i.e., logic and semantic networks), and the bottom-up approach, which leverages advanced statistical NLP on large, labeled datasets. Cambria et al. (2020) found that this hybrid approach outperforms both symbolic representations and statistical methods alone, and Barisi et al. (2021) found their proposed attention-based deep models for sentiment analysis of short and long texts reach state-of-the-art levels of performance on multiple benchmark datasets. The authors contend the coupling of symbolic and sub-symbolic AI is a step further in the path from mere NLP to natural language understanding (Cambria, 2017; Cambria et al., 2020).

Another important trend is that, rather than the discrete categorization of emotions, continuous modeling based on dimensions (valence, arousal, and dominance) is gaining momentum (Schuller and Schuller, 2018; Cambria, 2017). In this trend, the ensemble method is also utilized for predicting emotional intensity and ambivalence. Akhtar et al. (2019) proposed a stacked ensemble method to solve not only the problem of classification, i.e., the prediction of emotion/sentiment, but also the prediction of intensity in emotion/sentiment otherwise known as the regression problem. There have also been significant developments in ambivalence handling which enhances the capacity of algorithms to classify sentiments into positive versus negative but also recognize four other classes: neutral, mix-positive, mixed, negative, and mixed-neutral (Wang et al., 2020). Wang et al. (2020) proposed an algorithm that includes sentiment scales that can be adjusted for fine-tuned emotion-sensing, and strength-level tuning parameters to consider sentiment intensity and handling ambivalence. For example, words such as ‘extremely’ or ‘super’ are considered the highest enhancer parameter, while ‘minor’or ‘mini’ are considered as reducer parameters. In the field of reading emotions and sentiments from images or videos, researchers also collate different neural networks for classifying different aspects relevant to reading emotions such as objects, scenes, and facial expressions for higher accuracy (Parry and Vuong, 2021; Barros et al., 2020; Do et al., 2020).

Interestingly, notable differences in the algorithmic design of affective computing among different branches can be spotted. Text-based computational processing of emotions, pioneered by the Asia-Pacific cluster with the Singapore-based SenticNet as the focal industrial player, increasingly features the ensemble method which takes advantage of both deep learning (i.e., statistical approach that uses deep neural networks) and symbolic AI systems (i.e., knowledge bases that contain semantic networks using linguistic rules) (Susanto et al., 2020; Cambria, 2017). In contrast, voice-based computational processing, pioneered by the European collaborative network with the Germany-based audEERing as the focal company, is moving towards end-to-end learning with AI systems being designed to learn, extract, and even synthesize emotions by themselves without lesser degrees of human-annotated emotion labels (Schuller and Schuller, 2021). Similarly, vision- and biosignal-based affective computing also features weakly supervised machine learning where the human role is limited to providing emotion labels for images or videos (Barros et al., 2020; Parry and Vuong, 2021; Ngai et al., 2022).

Differences aside, various branches or modalities of affective computing have been shown to collaborate and combine their efforts and techniques. For example, Poria et al. (2017) highlighted the movement of the field from unimodal processing of human affect to multimodal fusion in which computing techniques from texts, audio, and visual signals are combined to generate higher accuracy. Schuller and Schuller (2021) also emphasized the exciting trend of ‘transfer learning,’ in which deep neural networks trained in one modality (for example, to read emotions from images) are then applied to a different modality (for example, speech emotion recognition). Such unifying movement towards multimodal affective computing is epitomized in the Multimodal Sentiment Analysis Challenge—the MuSe challenge, co-organized by top academic and industrial institutions in the field such as Imperial College London (UK), Nanyang Technological University (Singapore), University of Augsburg (Germany), BMW Group, audEEring, SenticNet, etc. (See Fig. 6). This AI challenge aims “to provide a common benchmark test set for multimodal information processing and to bring together the Affective Computing, Sentiment Analysis, and Health Informatics communities, to compare the merits of multimodal fusion for a large amount of modalities under well-defined conditions” (Stappen et al., 2021).

Fig. 6
figure 6

Affective computing’s institutional collaborative networks, 1995–2020.

Full size image

Discussion

Employing bibliometric analysis on a dataset of 3386 Web-of-Science-indexed academic articles during 1995–2020, we have identified major developments in the field over the past 25 years. The annual growth rate of scientific production related to affective computing is 11.36%. This number suggests an exponential growth of scientific publications in this field, as the number of publications output in 2016–2020 alone nearly equals all of the publications from the previous 20-year period (1660 vs. 1726). With mounting commercial and political interest in affective computing applications, we suggest this trend will continue.

Shaky science of emotions

The thematic analysis reveals growing academic interest in the further fine-tuning of computational techniques related to commercial applications of emotional AI and a decline of interest in affective computing to detect and study mental illnesses such as depression and bipolar disorders. In the future, the growing appreciation of the ‘Bayesian brain’perspective in the theoretical study of emotions and affects (Hoemann et al., 2019; Barrett, 2017b) will likely create a new thematic shift in the field of affective computing. In the emerging ‘theory of constructed emotion,’ Barrett (2017a) argues that emotions should be regarded as abstract categories constructed from past experiences to represent bodily sensations and feelings on a cognitive level. For Barrett, the brain constructs such mental representations of affective states as a way to predict the body’s need for external situations so that it can allocate resources to move the body in a cost-efficient manner. Such a constructivist understanding of how the brain generates emotions and affects is radically different from the classical, essentialist view of emotion championed by the psychologist Paul Ekman (Ekman, 1999). It is worth noting that although Ekman’s model of six basic emotions has fallen into disrepute for being oversimplified and contextless (Gendron et al., 2018; Azari et al., 2020; Barrett et al., 2019; Siegel et al., 2018; Heaven, 2020), it is still very influential, especially in vision-based and biosignal-based affective computing (Ivanova and Borzunov, 2020; Krumhuber et al., 2021; Barros et al., 2020). This is perhaps due to the absence of training datasets based on different theoretical models from Ekman.

Within academia, however, recent efforts have been made to move away from the Ekman model, such as by adding more emotions (Cowen et al., 2021; Susanto et al., 2020) or providing contextual information such as cultural or temporal parameters (Barros et al., 2020, Parry and Vuong, 2021, Barisi et al., 2021). The most significant effort is in text-based affective computing. Here, emotional contents from texts are computed based on theoretical models that contain finer granularities of emotions. For example, the Hourglass model has 24 emotion types divided into four dimensions (introspection, temper, attitude, sensitivity), or Ortony, Clore, and Collins’ model—the OCC model has 22 emotion types divided into three hierarchies (consequences of events, actions of agents, aspects of objects) (Susanto et al., 2020). Based on these theoretical models, researchers created lexica and knowledge bases, where words and their affective contents are mathematized, digitized, and stored utilizing logic and linguistic rules. The latest and largest lexicon for sentiment analysis is SenticNet 6, with 200,000 words and multiword expressions, which are mapped into a semantic network using both deep learning and symbolic AI (Cambria et al., 2020). Arguably, in the long term, as the theory of constructed emotion is operationalized and built into computational models to study emotion, it may generate a major thematic shift in affective computing research.

The tension in finding the most appropriate model of emotion has also been found among industry players. The mounting public push back against facial recognition systems is seeing some leading emotional AI companies such as HireVue—a global leader in AI recruitment solutions, moving away from facial coding approaches to gauging emotions and instead of looking towards biosensors that read an individual’s non-conscious data (Nakamura T., personal communication, Jan 15, 2021). Curiously, emotional AI companies such as Affectiva and Realeyes still rely on Ekman’s controversial model while forming strategic partnerships with technology giants such as IBM and NEC. Not only are these efforts directed at expanding their business model but they are also furthering their global reach into regions where data collection and privacy laws are less stringent (Henning, 2019).

A global playing field

In terms of the global playing field, while the US dominates in research output and citation from 1995–2015, China is now leading the field in affective computing research. Publications by corresponding authors from China have overtaken the US as the most productive and surpassed the UK as the second most cited in 2016–2020. A sub-dataset of publications by authors affiliated with Chinese institutions generates Fig. 7 in which China is shown as a prominent figure in the citation network. Here, the size of the China node, which represents its level of connection in the network, is roughly equivalent to those of other major countries such as the US and England. Moreover, it is clear that China forms a citation network with all other countries. Another indicator of the rise of China is the color of its node. The color represents the average year of the cited publications. Here, the average citation year of Chinese-authored papers is in the early 2010s, while the average years of cited American-authored papers and English-authored papers are 2009 and 2002, respectively. This indicates an increased influence of Chinese-authored articles in the recent decade.

Fig. 7
figure 7

Citation network of the top 16 countries that accumulate >1000 citations.

Full size image

Surprisingly, Japan was missing in the top 10 countries measured by citation, even though the country has been widely regarded as a leader in robotics and AI. One reason for this gap can be attributed to language. Japanese researchers must go through the extra steps of English translation, copy-editing, and proofing in order to publish in top international journals. Other reasons were offered in our conversation with Cristian Vlad, the representative of IBM Global Business Services with more than 15 years of working in Japan. Vlad pointed out four main factors influencing the slow adoption and advancement of AI in Japanese society: (1) a lack of advanced training in AI in higher education, (2) a highly bureaucratic system resistant to change, (3) a lack of entrepreneurial culture, (4) a work culture that values loyalty over productivity (Vlad C., personal communication, January 2021).

Similarly, Russia is another global actor not featured in the top countries involved in affective computing. Its total publications recorded in WoS database is only 36 (1.06 %) but nor is the country present in any major collaborative networks (Figs. 2 and 6). Our study indicates that in the increasingly heteropolar world, the US-led race to become the dominant science and technology power is no longer with Russia but rather China. Although still considered a superpower in military terms, Russia’s AI research output in top international journals trails far behind many of its Western counterparts. Evgeny Pashentsev, a leading AI researcher at the Institute of Contemporary International Studies in Ministry of Foreign Affairs, attributes the lack of Russia’s progress in affective computing to a myriad of factors, the first of which is politics. Longstanding geopolitical rivalries and the more recent, anti-Russian sanctions of the US and other NATO countries hamper international collaborations and critically, funding. Second, the militarization of science and sensitive links between AI development and its weaponization. The third factor is bureaucracy. For decades now, serious problems with the country’s economy have resulted in underfunding in many fields of scientific research. Coupled with these obstacles is the lack of a coherent state-led program. Fourthly, language and culture. As with Japan, Pashentseve observes that language remains a significant barrier in publishing but also that Russian researchers often prefer to publish in national journals. The absence of Russian publications in WoS-indexed journals can be traced back to the Soviet era (Pashentsev, personal communication, 2021).

As a result, the above-mentioned barriers have contributed to the low number of Russian publications in affective computing research. However, according to Kolotaev, a Ph.D. candidate at St. Petersburg University, the COVID-19 pandemic has triggered a renewed focus on research on affective computing, and AI. In lieu of government support, large conglomerates such as Sberbank, Yandex, Mail.ru, MTS, or Gazprom have taken the initiative of engaging in emotional AI research for commercial reasons, creating their own research programs for speech emotion recognition, and deep text analytics (Kolotaev, personal communication, 2021). In 2019, these commercial giants founded the “AI Alliance Russian” to fund research and “provide technological leadership for Russia and corporate members of the alliance in the field of artificial intelligence” (AI Alliance Russia, 2021). Private-led initiatives such as these have reinvigorated national interest in the AI industry. This has stimulated an increase in publications in WoS/ Scopus publications, including high-ranking journals. Both Pashentsev and Kolotaev are optimistic about the future of affective computing research in Russia pointing out the nation’s strong grass-root foundations in “high level of education of the population, achievements in mathematics, and recent private industry support” (personal communication, 2021).

Global collaborative networks

Our analysis also identifies two main collaborative networks: the ‘Asia-Pacific cluster’ of the US, China, Singapore, Japan, India; the ‘European’ cluster of Germany, the UK, and the Netherlands. We also find Chinese and American authors collaborate with each other in a majority of their multiple-country publications, which contradicts the ongoing political rhetoric of the AI race and the emerging technological cold war between the two countries. The cross-cultural diversity of the Asia-Pacific cluster might bring about a new wave of studies on the cross-cultural differences in emotionality and their implications for affective computing techniques (Ghotbi et al., 2021, Mantello et al., 2021). Moreover, as technologies move across national and cultural borders, it is likely that researchers will find greater cultural differences and nuances in the way people express and infer emotions (Gendron et al., 2018), which will expose the shortcomings of the current generation of techniques. Here, the academic and commercial interests align and point in the direction of more cross-cultural and multinational research collaborations in the field (Vuong, 2018, 2019).

Perhaps one of the most understudied areas of research in affective computing is emotion acculturation, i.e., how people learn a new mode of emotional expression and inference upon encountering a new culture (De Leersnyder et al., 2011), and how they incorporate new cultural values then adapt emotionally to their transformed set of core values (Vuong et al., 2018; Vuong et al., 2020; Vuong and Napier, 2015;Vuong, 2016). This is likely because of the negative criticism surrounding the basic emotion thesis posited by Paul Ekman (1999). As our understanding of the emotion acculturation process grows at all levels: neuroscientific, individual, interpersonal, and societal, one would expect that the next generations of affective computing research, emerging from of the growing trend of interdisciplinary, multinational collaborations, will produce advanced computational models that simulate the processes of emotional acculturation. This may be a major thematic change in the coming decades.

Conclusion

In summary, with the staggering annual growth rate of 11.36% over the recent 25-year period and huge potential for commercial and political applications, the field of affective computing is expected to continue to expand and become an ever-more dynamic sub-field of AI. This study has highlighted major developments of the field such as the thematic shift away from the detection of mental illness to more commercially viable applications and the unifying movement towards multimodal affect-sensing techniques. Moreover, as our findings suggest, apart from ongoing geopolitical rhetoric and posturing, affective computing research is being driven by international alliances and collaborations in the Asia-Pacific and the European clusters. Certainly, development in the field will accelerate due to the ascension of China and India as well as the return of Russia as a top contender in the AI race. Yet a certain amount of caution needs to be acknowledged. A growing number of scholars believe that the emotion-recognition industry is currently being driven by more hype than science (Buolamwini J and Gebru T, 2018, Barrett et al. 2019; Heaven, 2020; Crawford, 2021). Besides contesting the Eckman model, they argue that sensing, reading, and evaluating the complexity of human emotions cannot be solved simply by creating more powerful algorithms. Rather, they contend that the science community needs to better understand the complex modulators that give rise to affect and emotional states. Hopefully, increasing international collaborations and alliances will correct this situation and assist in the development of more advanced technologies that can better handle such things as gender, race, attitudinal diversity, as well as cross-cultural nuances in emotion.