Emognition dataset: emotion recognition with self-reports, facial expressions, and physiology using wearables

The Emognition dataset is dedicated to testing methods for emotion recognition (ER) from physiological responses and facial expressions. We collected data from 43 participants who watched short film clips eliciting nine discrete emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, and sadness. Three wearables were used to record physiological data: EEG, BVP (2x), HR, EDA, SKT, ACC (3x), and GYRO (2x); in parallel with the upper-body videos. After each film clip, participants completed two types of self-reports: (1) related to nine discrete emotions and (2) three affective dimensions: valence, arousal, and motivation. The obtained data facilitates various ER approaches, e.g., multimodal ER, EEG- vs. cardiovascular-based ER, discrete to dimensional representation transitions. The technical validation indicated that watching film clips elicited the targeted emotions. It also supported signals’ high quality.

was higher in the awe film condition than in other film conditions, all ps < .001, except for liking and enthusiastic conditions ps 23 = 1.00. Disgust was higher in the disgusting film condition than in other film conditions, all ps < .001, except for angry film 24 condition p = .36. Enthusiasm was higher in the enthusiastic film condition than in other film conditions, all ps < .05, except 25 for liking film condition p = 1.00. Fear was higher in the fearful film condition than in other film conditions, all ps < .001, 26 except for angry film condition p = .49. Liking was higher in the liking film condition than in other film conditions, all ps < .01. 27 Sadness was higher in the sad film condition than in other film conditions, all ps < .001. except for angry film condition p = .25. 28 Surprise was higher in the surprising film condition than in other film conditions, all ps < .05, except for disgust film condition 29 p = .06. 30 Furthermore, we found that the liking, enthusiastic, amusing, and awe film clips were more positively valenced than baseline 31 and neutral film clips all ps < .001, indicating the group of positive emotions. The angry, disgusting, fearful, and sad movies 32 were more negatively valenced than baseline and neutral film clips, all ps < .05, indicating the group of negative emotions. In 33 terms of valence, surprise did not differ from the baseline and neutral film clips, both ps = 1.00. Affective clips were arousing 34 relative to baseline levels and to the neutral film clips, all ps < .001. Neutral film clip elicited stronger approach motivation 35 than baseline. Liking, enthusiastic, and awe clips elicited stronger approach motivation than neutral film clip, all ps < .001, 36 indicating the group of approach motivated emotions. The film clips for anger, disgust, fear, and sadness elicited stronger 37 avoidance motivation indicating the group of avoidance motivated emotions. Amusement and surprise did not differ from the 38 neutral film clip in arousal, both ps > .05.

39
Difference Within Film Clips. Self-reported targeted emotions were highest in the corresponding film clip condition 40 (Tab. 2). In amusement film condition, self-reported amusement was higher than self-reported anger, awe, disgust, enthusiasm, 41 fear, liking, sadness, and surprise, all ps < .001. Similarly, disgust was higher than other self-reported emotions in disgust 42 film condition, all ps < .002; sadness was higher than other self-reported emotions in sadness film condition, all ps < .001; 43 and surprise was higher than other self-reported emotions in surprise film conditions, all ps < .001. In some cases, however, 44 more than one emotion took high values within the particular stimulus. In the angry film clip condition, anger was higher than 45 other self-reported positive emotions (amusement, awe, enthusiasm, liking), all ps < .001, but did not differ from other negative 46 emotions (disgust, fear, and sadness), and surprise, all ps > .05. Moreover, distributions of anger and disgust were very similar, with the median higher in the latter emotion. Awe was higher than other self-reported emotions in awe film condition, all ps < 48 .01, except for self-reported liking, p = .90. Enthusiasm was higher than other self-reported emotions in the enthusiasm film 49 condition, all ps < .001, except for self-reported awe, p = .99. Fear was higher than other self-reported emotions in the fear film 50 condition, all ps < .001, except for self-reported surprise, p = .05. Liking was higher than other self-reported emotions in the 51 liking film condition, all ps < .003, except for self-reported awe, p = .05. There were no differences between self-reported 52 emotions after watching the baseline film , all ps > .05.

54
A detailed analysis of the obtained signal-to-noise ratios (SNRs) is provided in this section. 55 We chose to analyze only raw signals provided by the devices, described in Main File Sec. Data Processing and Cleaning.

56
Moreover, we did not analyze signals from accelerometers and gyroscopes, as the experiments were conducted in a sitting 57 position. Implementation of the algorithm used to calculate SNR is provided in the Main File Sec.

72
We have analyzed the Quantum Sense results in relation to the targeted emotion (stimuli annotation). Please note that such 73 analysis validates the Quantum Sense annotations only, not the study procedure nor stimuli effectiveness. The latter is validated 74 in the Main File Sec. Technical Validation.

75
The videos were processed using Quantum Sense software to perform emotion classification with six basic emotions 76 (neutral, anger, disgust, happiness, sadness, surprise). The software classified each video frame if a human face was detected.

77
The annotated frames from each participant were grouped by the stimuli type (within the condition) or by the annotations from 78 the software (between conditions). Then, the number of frames was changed to the percentage of frames classified with a 79 specific emotion within a given stimulus. We noticed that participants had a neutral face over the majority of the experiment.