A Large Finer-grained Affective Computing EEG Dataset

Affective computing based on electroencephalogram (EEG) has gained increasing attention for its objectivity in measuring emotional states. While positive emotions play a crucial role in various real-world applications, such as human-computer interactions, the state-of-the-art EEG datasets have primarily focused on negative emotions, with less consideration given to positive emotions. Meanwhile, these datasets usually have a relatively small sample size, limiting exploration of the important issue of cross-subject affective computing. The proposed Finer-grained Affective Computing EEG Dataset (FACED) aimed to address these issues by recording 32-channel EEG signals from 123 subjects. During the experiment, subjects watched 28 emotion-elicitation video clips covering nine emotion categories (amusement, inspiration, joy, tenderness; anger, fear, disgust, sadness, and neutral emotion), providing a fine-grained and balanced categorization on both the positive and negative sides of emotion. The validation results show that emotion categories can be effectively recognized based on EEG signals at both the intra-subject and the cross-subject levels. The FACED dataset is expected to contribute to developing EEG-based affective computing algorithms for real-world applications.


Background & Summary
Affective computing, aimed at enabling intelligent systems to recognize, interpret, and respond to people's affective states, has drawn enthusiasm in various fields, including artificial intelligence, human-computer interaction, education, etc. 1,2 .In recent years, electroencephalogram (EEG) has gained increasing attention in the field of affective computing 3 .Unlike behavioural modalities such as voice, facial expression, and gestures that might be consciously disguised or restrained, EEG can objectively measure emotional states by recording people's brain signals directly 4 .Compared with other neuroimaging technologies, EEG devices offer advantages such as relatively low cost and high portability, making them promising candidates for practical affective computing applications 5 .However, while research has demonstrated the feasibility of affective state decoding based on EEG signals, efforts are still needed to bridge the research-to-practice gap for EEG-based affective computing techniques towards real-world applications 6 .
First, while the importance of accurate decoding of positive emotions is acknowledged for real-world affective computing applications 7 , existing EEG-based affective computing studies have mainly used classical emotion theories with an oversimplified categorization of positive emotions 6,8 .For example, among Ekman's six basic emotions, only "happiness" can be considered positive 9 .Considering that people usually experience positive emotions more frequently than negative emotions in their daily lives 10 , the relatively limited categorization of positive emotions may fail to effectively describe one's affective states during possible affective-computing application scenarios 7 .Psychologists have called for a more balanced view of both the negative and positive side of emotion 11,12 , and emerging neuroscience studies have provided preliminary support for the decoding of discrete positive emotions.For instance, inspired by recent positive emotion theories 12 , distinct neural representations of positive emotions, such as joy, amusement, tenderness, etc., have been revealed with a video-watching paradigm for emotion elicitation 8,[13][14][15] .However, publicly available EEG datasets have not sufficiently emphasized the positive side of emotion.A finer-grained emotion categorization, preferably with a special focus on positive emotions, is needed for datasets that better fulfill the needs of real-world affective computing applications 6 .
Second, emotion recognition that can be "plug-and-play" is always preferred in practical scenarios due to its time-saving and good user experience 16 .However, individual differences in people's emotional experiences and the correspondingly individualized emotion-related EEG activities have posed challenges to the development of algorithms for cross-subject affective computing 4 .Indeed, substantial drops in the performance from intra-subject to cross-subject emotion recognition have been consistently reported 17,18 , hindering seamless emotion recognition usage.Due to the time and labour cost for EEG data collection, available benchmark datasets usually have a relatively limited sample size (20~60 subjects) [19][20][21][22] .A dataset with a larger sample size, however, may help address the cross-subject affective computing challenges, as the extraction of subject-invariant representation of emotional states could benefit from an increase in the subject number 23 .In particular, the recent rise of deep learning methods has brought new possibilities for cross-subject challenges and also placed higher demands on the sample size 6,24 .With the development of data augmentation techniques 25 , the expected positive effects of the increased sample size could be amplified.
The present Finer-grained Affective Computing EEG Dataset (FACED) aims to address these issues by recording EEG signals from 123 subjects who watched 28 emotion-elicitation video clips covering nine emotion categories (amusement, inspiration, joy, tenderness; anger, fear, disgust, sadness, and neutral emotion).The sample size of over 100 subjects is expected to facilitate the cross-subject affective computing research.The EEG data were recorded using 32 electrodes according to the international 10-20 system.For each video clip, subjective ratings were obtained for all subjects, covering the dimensions of the four negative and four positive emotions, as well as arousal, valence, familiarity, and liking.For validation, we used a classical machine learning algorithm 26 for both intra-subject and cross-subject affective computing and a state-of-the-art algorithm utilizing a contrastive learning framework 4 for cross-subject affective computing.The features of the FACED dataset are summarized in Table 1.The validation supports the effectiveness of nine-category cross-subject affective recognition.The dataset is open-access for research purposes: https://doi.org/10.7303/syn50614194.

Methods
Stimuli and experiment procedure.Twenty-eight video clips were used to elicit nine categories of emotion (four negative emotions: anger, disgust, fear, and sadness; four positive emotions: amusement, inspiration, joy, and tenderness; and the neutral emotion).The selection of emotion labels is based on the following considerations.The four negative emotions were derived from Ekman's six basic emotions 9 , while the selection of the four positive emotions was based on the latest advancements in psychology and neuroscience, as well as specific application requirements: Recent neuroscience studies have identified three positive emotions (inspiration, joy, and tenderness) as being representative 8,13 , and amusement is frequently encountered in application scenarios like human-computer interactions 27,28 .The emotion-evoking video clips were selected from various databases, including the FlimStim database 29 , the database of positive emotional videos 8,13 , the standardized database of Chinese emotional videos 30 , and the database of emotion profile videos 31 .Each negative/positive emotion category had three video clips, while the neutral emotion category had four clips.On average, these video clips lasted about 66 seconds, with duration ranging from 34 to 129 seconds.The details of each video clip are provided in Supplementary Table S1.
Figure 1 demonstrates the experimental procedure.During the experiment, subjects were seated approximately 60 cm away from a 22-inch LCD monitor (Dell, USA).Each trial began with subjects focusing on a fixation cross for 5 seconds, followed by watching a video clip.The sound of video clips was played through stereo speakers (Dell, USA).After each video clip, subjects were required to report their subjective experiences during the video-watching on 12 items, including anger, fear, disgust, sadness, amusement, inspiration, joy, and tenderness, as well as valence, arousal, liking and familiarity.Subjects provided ratings on a continuous scale of 0-7 31,32 for each item and then had at least 30 seconds of rest before starting the subsequent trial.Here, for the valence item, 0 indicated "very negative" and 7 indicated "very positive".For the other items, 0 indicated "not at all" and 7 indicated "very much".The meaning of the 12 items was explained to the subjects before the experiment.
To minimize the possible influence of alternating valence, video clips with the same valence (e.g., positive) were presented successively as a block of four trials.Consequently, there were three positive blocks, three negative blocks, and one neutral block.Between two blocks, subjects completed 20 arithmetic problems to minimize the influence of previous emotional states on the subsequent block 33 .When answering the arithmetic problems, if subjects did not complete a problem in 4 seconds, it would be skipped, and the next problem would be presented.The order of the video clips within each block and the seven blocks was randomized across subjects.Before the experiment, subjects performed one practice trial to become familiar with the procedure.The experimental procedure was programmed with Psychophysics Toolbox 3.0 extensions 34 in MATLAB (The Mathworks, USA).psychiatric disorders.The subjects were de-identified and indexed as S000~S122.The study was approved by the local Ethics Committee of Tsinghua University (THU201906), and informed consent was obtained from all subjects.
Data acquisition.The EEG signals were recorded using a wireless EEG system (NeuSen.W32, Neuracle, China) at a sampling rate of 250 or 1000 Hz.Thirty-two wet electrodes (Ag/AgCl electrodes with conductive gel) were placed according to the international 10-20 system.The impedance was kept below 10 kOhm throughout the experiment.Our experiment was conducted in two cohorts in two distinct time periods, involving two non-overlapping groups of subjects.Cohort 1 included participants sub000 to sub060, while cohort 2 encompassed participants sub061 to sub122.The data in both cohorts was collected with the same experimental procedure.In the experiment, we initially used a sampling rate of 250 Hz, which is comparable with other datasets 19,35 .However, the sampling rate was switched to 1000 Hz 18 later to provide richer information for data analysis.The sampling rate for each subject during recording is provided in Recording_info.csv,along with the dataset.Data in both cohorts were recorded with the reference electrode at CPz and the ground electrode at AFz.The reference and ground electrode were defaulted by the EEG amplifier, which was also used in other emotion-related studies 31,36 .Note that the 32-electrode coverage available in the present dataset allowed multiple re-referencing methods (e.g., the common average or the average of both mastoids) by simple linear computations in subsequent analysis.The spatial placement of the electrodes in the two cohorts is the same, although 6 of them have different names due to the device setting.The electrode information for both cohorts can be found in Supplementary Tables S2, S3.During the experiment, the recorded EEG signals were synchronized to the experimental procedure by sending triggers to the EEG recording system with a serial port when events occurred, which is a common practice in data-task synchronization in EEG-based experiments 37 .The event information during the experiment is listed in Table 2.
Data pre-processing.The dataset was collected in a regular office environment resembling possible practical application scenarios 19,38 .Then, to validate the dataset, we conducted a pre-processing procedure to enable further analysis.The pre-processing process was conducted based on the MNE toolbox 39 , version 1.2.1, with Python 3.10.Codes for data pre-processing were provided together with the dataset.First of all, the unit for the recorded EEG signal was adjusted to μV.Then, the last 30 seconds of each video clip were selected to capture the maximal emotional responses 4,8 based on the timing of events that indicates the end of each video clip (i.e., "Video clip end" in Table 2).Then, the sampling rates of EEG were adjusted to 250 Hz (downsampled when necessary).Next, filtering, interpolation, and independent component analysis (ICA) were conducted to remove possible motion and ocular artifacts, which was similar to the pre-processing pipelines of other datasets like DEAP and SEED.Specifically, a bandpass filter from 0.05 to 47 Hz was applied to the EEG signals with the MNE filter() functions.Following that, samples whose absolute values exceeded three times the median absolute value in each 30-second trial were defined as outliers 4 .In each 30-second EEG trial, if the proportion of outliers for an electrode exceeded 30%, this electrode was defined as a bad electrode and was interpolated, following previous studies 4,40 with the MNE interpolate_bads() function.Then, the ICA method was performed.The independent components (ICs) containing ocular artifacts were automatically defined and rejected by using FP1/2 as the proxy for electro-oculogram with the MNE ica.find_bads_eog() and ica.exclude functions.Next, the cleaned EEG signals were re-referenced to the common average reference.Finally, the order of electrodes in cohort 1 was adjusted to be consistent with cohort 2. Note that all the data pre-processing was conducted offline.The raw EEG data are provided and hereby available to the users.We also provide the pre-processed data to promote more efficient use of the present dataset.We recommend that users read the released pre-processing code before using the pre-processed data to develop a more detailed grasp of the implementation.Nevertheless, users can design their own pre-processing pipeline and apply it to the raw data according to their specific needs (e.g., considering sufficient artifact removal towards electromyography, electrocardiogram, and channel noise).
We also provide commonly-used EEG features, including differential entropy (DE) 41 and power spectral density (PSD) 42 for our dataset.The DE and PSD features were obtained from the pre-processed data within each non-overlapping second at 5 frequency bands (delta band: 1-4 Hz, theta band: 4-8 Hz, alpha band: 8-14 Hz, beta band: 14-30 Hz and gamma band: 30-47 Hz).The formula to calculate DE and PSD followed the practice in the SEED dataset (https://bcmi.sjtu.edu.cn/home/seed/seed-iv.html). 2 2 π σ = Where x is the EEG signal, σ is the variance of the EEG signal.

Data records
The FACED dataset is available in Synapse 43 and stored in the "FACED" repository (Project SynID: syn50614194) at the website https://doi.org/10.7303/syn50614194.As shown in Table 3, the current dataset contains data records from 123 subjects.For each subject, we provide raw EEG data and event data in the ".bdf " file format, self-reported ratings in the MATLAB ".mat" format, pre-processed EEG data in the Python ".pkl" format, DE and PSD features in the Python ".pkl" format.The pre-processed data were obtained after applying the pre-processing pipeline described in the Methods section to the raw EEG data.For each subject, the pre-processed EEG data is presented as a 3-dimensional matrix of VideoNum*ElecNum*(TrialDur*SampRate).The number of video clips is 28.The order of video clips in the pre-processed data was reorganized according to the index of video clips, as reported in Supplementary Table S1.The number of electrodes is 32.The order of electrodes is provided in Supplementary Table S3.The duration of each EEG trial is 30 seconds, and the sampling rate of pre-processed EEG data is 250 Hz.For each subject, the DE and PSD feature is a 4-dimensional matrix of VideoNum*ElecNum*TrialDur*FreqBand.There are 5 frequency bands, corresponding to delta, theta, alpha, beta, and gamma band, respectively.
The data structure of behavioural data is shown below in Table 4.For each subject, the behavioural data includes self-report ratings on 12 items for each video.Additionally, task performances, including accuracy and response time, for the arithmetic problem-solving task during each inter-block interval are also provided.The unit for the response time is in seconds.the present dataset could support a finer-grained emotion recognition.The classification of emotional states was conducted on a 1-second time scale.In the first part, the classical method based on DE features and support vector machine (SVM) 26 was used for intra-subject and cross-subject emotion recognition.Here, anger, disgust, fear, and sadness were labelled negative, while joy, amusement, inspiration, and tenderness were labelled positive.The neutral emotion was excluded due to an imbalanced data amount (4 neutral video clips vs. 12 positive/negative video clips).The recognition was carried out using the pre-processed data with a ten-fold procedure.In the intra-subject emotion recognition, for all positive/negative video clips, 90% of EEG data in each video clip served as the training sets, and the remaining 10% was used as the testing sets for each subject.In the cross-subject emotion recognition, the subjects were divided into 10 folds (12 subjects for the first nine folds, and 15 for the 10 th fold).Then, nine-fold subjects were used as the training sets, and the remaining subjects were used as the testing sets.The procedure was repeated 10 times and the classification performances were obtained by averaging accuracies for 10 folds.The classification accuracies of 78.8 ± 1.0% and 69.3 ± 1.5% (mean ± standard error, the same below for the reported classification accuracies) were obtained for the intra-subject and cross-subject emotion recognition, respectively.Both the intra-subject and cross-subject performances were comparable with previous studies using the same classification methods 26,44 .A drop in performance was also observed in the cross-subject recognition compared with the intra-subject recognition, consistent with findings from previous studies 17,18 .The classification accuracies for each subject were demonstrated in Fig. 4.Both intra-subject and cross-subject classification reveal substantial individual differences, underscoring the value of large-scale datasets in better characterizing population attributes.
In the second part, we performed a classification of the nine-class emotional states to assess if the present dataset could support more fine-grained emotion recognition.The same classification procedure (DE + SVM with a 10-fold cross-validation, as detailed above) was conducted to classify joy, tenderness, inspiration, amusement, anger, disgust, fear, sadness, and neutral emotions.The achieved accuracies were well above the chance level (intra-subject: 51.1 ± 0.9%; cross-subject: 35.2 ± 1.0%), indicating the feasibility of decoding multiple emotional states based on EEG signals.The classification accuracies for each subject were demonstrated in Fig. 5.
Moreover, we also employed one state-of-the-art algorithm named Contrastive Learning for Inter-Subject Alignment (CLISA) 4 for the cross-subject recognition of the nine emotion categories.The objective of CLISA was to reduce inter-subject differences by maximizing the similarity in EEG signal representations among subjects when exposed to the same emotional stimuli, as opposed to different ones.Subsequently, the inter-subject-aligned EEG representations were used to extract features for emotion classification, which are expected to be relatively stable across subjects.Due to its state-of-the-art cross-subject emotion recognition performance on several EEG datasets, we selected the CLISA algorithm to validate the newly proposed FACED dataset.A classification accuracy of 42.4 ± 1.2% was achieved with a ten-fold procedure, demonstrating a 7.2% improvement in the nine-emotion classification cross-subject performance.The accuracy was also comparable with one previous study 4 .The classification accuracies based on CLISA are shown in Fig. 6.The classification results supported the potential to boost the cross-subject performance by integrating the latest advancements in deep learning.

Fig. 3 Fig. 4
Fig.3The topographies of the correlation coefficients between the relative spectral powers at the five frequency bands and the subjects' ratings on the eight emotion items.

Fig. 5 Fig. 6
Fig. 5 The classification accuracies for each subject with DE + SVM in the nine-category classification of (a) intra-subject and (b) cross-subject emotion recognition.The subjects are re-ranked according to their classification accuracies, increasing from left to right.The light gray bars indicate accuracies for each subject, and the white bars indicate averaged accuracies across all subjects.The error bars of white bars indicate the standard error across all subjects.The dotted gray line indicates the chance level of nine-category classification.

Table 2 .
The event information during the experiment.

Table 3 .
Data records in the FACED dataset.Note: The subXXX indicates sub000~sub122.