Background & Summary

Serious games differ from ordinary games, which can be adjusted based on user/patient situations. For example, minimum/maximum angles for joints can be defined while playing a game which is very important for disabled people who need physical exercises. In the meantime, it is possible to get measurement data in serious games like balance, range of motion (ROM) etc. Serious games focus on patients’ required motions with cognitive actions, while ordinal games focus on entertainment.

Serious games are promising tools to improve upper extremity functions in children with neurological disorders. These games are developed as a rehabilitation program option to provide an exercise environment where children with different special needs can do many repetitions and receive immediate feedback during rehabilitation. Serious games have previously been shown to improve cognitive functions and motor skills such as attention, hand-eye coordination, and visual perception and facilitate learning1,2,3,4,5. In addition, games positively improve motivation, attention, processing speed, concentration, and visual discrimination in neurodevelopmental disorders such as dyslexia and mental retardation5,6.

Game therapies are more effective than traditional physiotherapy programs in improving upper extremity functions in children with Obstetric Brachial Plexus7. Furthermore, serious games-based rehabilitation has increased the motivation of children with dyslexia8. Children’s motivation during game therapy is essential, which may lead children to withdraw attention and even drop out of therapy. If a child gets stressed during game therapy, he/she can lose his/her motivation to continue the therapy. Thus, game therapies can be more prosperous if they recognize the children’s emotions/stress and modify the games considering their emotions/stress to increase their motivation and involvement.

Over the last decade, many multimodal emotion recognition databases containing data of adults have been published9,10,11,12,13,14,15,16,17,18,19,20,21,22. Even though many adult databases have been published, few emotion recognition databases including children have been developed. The emotion/stress detection of children is recently attracting considerable interest23,24, especially in children suffering from neurodevelopmental disorders9,25,26,27,28. Facial expression databases of typically developed children have been developed,i.e. CAFE29, NIMH-ChEFS30, Radboud Faces Database31, Dartmouth Database of Children’s Faces32, CEPS33, ChildEFES34. Furthermore, multimodal emotion recognition databases have been published for typically developing children containing audio, video, and physiological signals called EmoReact23 and MMDB35. Note that to our knowledge, no existing datasets are available that contain physiological signals and facial expressions of the children with different special needs (obstetric brachial plexus injury, dyslexia, and mental retardation) to be used for stress/emotion recognition of them.

AKTIVES dataset consists of physiological signals and camera recordings of 25 children with different special needs (obstetric brachial plexus injury, dyslexia, and mental retardation) and typically developed children who played serious games. The physiological signals (blood volume pulse (BVP), electrodermal activity (EDA), and skin temperature (ST)) were collected using the wearable Empatica E4 wristwatch36. Three experts annotated whether the children were stressed or not stressed and whether the children were given a reaction or no reaction through the game therapy. The dataset contains the video files recorded by the camera in Mp4 format. The videos include the child’s face and upper body during the game therapy.

AKTIVES dataset may contribute to the emotion recognition and stress detection models using physiological signals and facial expressions for children with different special needs. In addition, the AKTIVES dataset may serve to tackle the research questions related to (1) the multimodal approach to stress detection using physiological signals, (2) emotion recognition using facial expressions, and (3) the reaction of children to the developed serious games.

Methods

Ethics statement

Ethical approval has been obtained from the Research Ethics Committee of Istanbul Medipol University (NoE-10840098-772.02-6580). Furthermore, the parents of the children have been informed about the experimental procedure for the test and data acquisition setup and signed written consent. An informed consent form is used to inform parents about how the videos of children will be shared.

Participants

A total of 25 children with obstetric brachial plexus injury, dyslexia, intellectual disabilities, and typically developed children participated in the study. In addition, 3 children with obstetric brachial plexus injury, 7 with intellectual disabilities, 4 with dyslexia, and 11 typically developed children participated in the study. The mean age of the children (10 males, 15 females) was 10.2 ± 1.27 years. Inclusion criteria consisted of a diagnosis of obstetric brachial plexus injury, intellectual disability, or dyslexia, aged between 5 to 14 years. Children with other chronic diseases were excluded from the study. Demographic and diagnostic characteristics of the participants were summarized in Table 1. The disability rates provided are 20–39 (individuals with special requirements), 40–49 (individuals with slight special requirements), 60–69 (individuals with advanced special needs), and 70–79 (individuals with very advanced special needs).

Table 1 Demographic and diagnostic characteristics of participants.
Full size table

Games

Becure CatchAPet and Becure LeapBall serious games were used in this study. First, the child tries to touch the rabbit on the screen in the playground with his hand in the virtual environment in the Becure CatchAPet game (Fig. 1). The child earns points for each contact with the rabbit and loses points for each missed rabbit. The child uses his/her wrist flexion/extension movement during the game. The wrist angles are determined by the physical therapists so that each child can virtually touch a rabbit with the correct range of motion. In the Becure LeapBall game, the child tries to drop the ball on the screen into buckets of the same color (Fig. 2). The child tries to hold and release the ball with grasping movements. The child directs the ball in the virtual environment, and if the child throws it into the correct bucket, he/she earns scores. The fail score increases when the child throws in the wrong basket. Various measures are recorded at the end of each game on the Becure E-Therapy web portal.

Fig. 1
figure 1

Becure CatchAPet.

Full size image
Fig. 2
figure 2

Becure LeapBall.

Full size image

Annotation

The children’s videos were recorded while playing Becure CatchAPet and Becure LeapBall. They were watched by three experts and were annotated as “Stress/No Stress” and “Reaction/No Reaction” every ten seconds. The children’s stress situations were assessed using children’s body language and annotated as “Stress/No Stress” and “Reaction/No Reaction”. The main manifestations of anxiety and stress on the human face involve the eyes (gaze distribution, blinking rate, pupil size variation), the mouth (mouth activity, lip deformations), and the cheeks, as well as the behavior of the head as a whole (head movements, head velocity). Additional facial signs of anxiety and stress in children may include a strained face, facial pallor, and eyelid twitching. All three experts are occupational therapists. Meanwhile, experts had at least 2 years of experience as an occupational therapist. Three experts did not see each other’s annotation. The majority of annotation has been kept. For example, if two experts annotated as stressed and one as no, stress has been assigned.

Apparatus

Various studies have been done on emotion recognition and stress detection using Empatica E4 wristbands37,38,39,40,41,42,43. Emotions via Empatica E4 wristband in children with hearing disabilities have previously been investigated44. Further, Empatica E4 has been utilized for emotion recognition in children with atypical or delayed development45. In addition, the sympathetic response of children with autism under stressors has been studied using an E4 wristband46. The Empatica E4 wristband was previously used for children with autism spectrum disorder (ASD)46. In this study, physiological signals blood volume pulse (BVP), electrodermal activity (EDA), and skin temperature (ST) of children with obstetric brachial plexus injury, dyslexia, intellectual disabilities, and typically developed were collected using an Empatica E4 wristband36. The sampling frequency of the BVP signal was 64 Hz, while the EDA and ST signal was 4 Hz36. The E4 wristband was firmly placed on the child’s wrist without cutting off the blood flow. The collected data were synchronized and downloaded via E4 Manager software. The children’s upper body and facial expressions were recorded using the camera. The web camera of the laptop computer (Lenovo G510 with i7 processor) where the Becure Software by INOSENS company runs was used to get videos. The videos were in Mp4 format and had 720p resolution with image frames of 1280×720 pixels. The recording began when the physical therapist started the games and automatically ended when the game ended.

Procedure

The study took place in the Technology Laboratory at Medipol University. The children and their parents were informed about the experimental procedure upon arrival. Then parents signed the written consent. The occupational therapists asked children to play the games approximately 2–3 minutes before the experiment so that children could get familiar with them. The child was asked to avoid unnecessary movements and to cover their faces.Children were instructed to fixate their gaze on a black screen for 30 seconds, serving as the baseline condition. Following the baseline period, the children were engaged in a serious game, either “Becure CatchAPet” or “Becure LeapBall,” for 420 seconds. Subsequently, upon completion of the game, the black screen was again presented for an additional 30 seconds. The procedure was repeated independently for each game.

The experimental procedure is given in Fig. 3. The child’s physiological signal and facial expression were recorded during the session. In this order, each child was asked to do both Becure CatchAPet and Becure LeapBall games. The experimental setup is presented in Fig. 4.

Fig. 3
figure 3

Experimental design.

Full size image
Fig. 4
figure 4

Experimental setup. The parents of the children signed written permission for the child’s picture to be included in this publication.

Full size image

Data processing

The entire experimental setup was collected in a single device to ensure ideal data synchronization during the experiment. The E4 wristband was connected to the E4 Manager, and the device’s clock was synchronized before the experiment. E4 Manager was installed on the device on which the experimental setup was run. The clock update of the device presenting the experiment was checked. E4 was saved as a Unix timestamp when data recording started. The device’s clock was kept up to date, so this timestamp was synchronized with the whole mechanism. During the experiment, synchronous timestamps with stimuli were collected from all devices. The recording start or end (and video duration) timestamp of this video (creation time may not give correct results) were recorded in the metadata of the recorded video for labels to be synchronized with the physiological data. The face video recording was taken from the device on which the experimental setup was operated to ensure synchronization. While the data was kept separately for each client in Plain Text (.txt) format before preprocessing, it was stored in Comma-Separated Values (.csv) format (tabular data) after preprocessing.

A sixth-order Chebyshev II filter with the 18 dB stopband attenuation (Rs) and 0.1 Hz normalized stopband edge frequency (Wn) was used to preprocess the blood volume pulse (BVP) data47. The electrodermal activity (EDA) signal was filtered using a fifth-order Savitzky-Golay filter with a frame length parameter equal to 1139. Note that the two individuals that express the same emotions might have physiological signals with different levels40. Thus, the BVP and EDA signals were normalized between 0 and 100.

The main purpose of this study is to provide a dataset consisting of physiological signals labeled in terms of “Stress/ No Stress” by the experts. However, the face landmarks and facial emotions of children detected with well-known methods were also given in the AKTIVES dataset for researchers to investigate the correlation between gaming motivation and facial emotions for future research. The videos had a 9.7 frame rate, and the videos’ duration depended on the children’s gaming performance. The total number of frames in a video file was nearly 4300. The emotions of the children were also recognized using their faces. Initially, the faces of the children were detected using Multi-Task Cascaded Convolutional Neural Networks (MTCNN)48. Then, the emotions were recognized using an open dataset called The Facial Emotion Recognition (FER 2013)49. FER 2013 dataset included 48 × 48 pixel grayscale images of faces. Each face was labeled with seven categories (0 = Angry, 1 = Disgust, 2 = Fear, 3 = Happy, 4 = Sad, 5 = Surprise, 6 = Neutral). Note that the FER dataset was previously trained on children and adults, and 75.42% accuracy was obtained50.

Data Records

The data recordings (physiological signals, camera recordings, expert annotations) are available at Synapse | Sage Bionetworks51. Figure 5 illustrates the physiological signal and video recording examples. The data is provided in four categories depending on the child’s condition (Obstetric Brachial Plexus Injury, Dyslexia, Intellectual Disabilities, Typically Developed). The child condition folders include the directories given names with each child’s ID (C1, C2, etc.). These child ID folders comprised two subfolders with serious game names Becure CatchAPet, and Becure LeapBall. The annotations of the experts (ExpertLabels.csv) in Comma-Separated Values (.csv) format, physiological (BVP, EDA, ST) and acceleration (ACC) data in a Comma-Separated Values (.csv) files, and camera data available in MPEG-4 (.mp4) format are all included in each serious games subfolders. In serious game subfolders, the automatically detected facial expressions from videos are also given in Comma-Separated Values (.csv) format (facial.csv). The full size of the data included in a repository is 4.94 GB. In total, there are 7 files for a child. These files and their descriptions are summarized in Table 2.

Fig. 5
figure 5

Examples of data available in the AKTIVES Dataset; (a) Physiological signals recorded with Empatica E4: BVP, EDA, ST (b) 3 sample video frames of a child (c) The facial emotions are detected as neural for the first two columns and sad for the third column with FER library. The faces are cropped for better visualization, and the facial landmarks are generated with the Dlib PyPI library. The children’s parents gave written consent to include her image in this publication.

Full size image
Table 2 Data files and descriptions for each participant.
Full size table

Technical Validation

Three experts annotated “Stress/No Stress” and “Reaction/No Reaction” through the execution of the serious games. The percentage of matching annotations to all annotations was calculated to validate the experts’ annotations. The agreement percentage was calculated using Eq. 1. Two experts did the calculations. (Experts 1–2, Experts 1–3, Experts 2–3) and all three experts (Experts 1-2-3).

$$Agreement=\frac{\#same\;annotation}{\#timestamps}$$
(1)

The maximum percentage for the “Stress/No Stress” annotations between two experts was 81.45%, while the minimum was 77.24%. The percentage agreement between the three annotations was 68.18% for the “Stress/No Stress”. The minimum result for the “Reaction/No Reaction” annotations was 69.50% between two experts, and the maximum was 83.24%. The agreement results for all experts were calculated as 63.39%. The results are presented in Table 3.

Table 3 The expert agreement results with respect to “Stress/ No Stress” and “Reaction/ No Reaction”.
Full size table

The signal-to-noise ratio (SNR) of the physiological signals acquired via the Empatica E4 device was calculated to validate the signal quality. The SNR metric was obtained with the help of the autocorrelation function utilizing a second-order polynomial to fit the autocorrelation function curve52. Whole physiological data were examined separately in the case of SNR. Raw physiological signals were utilized for the SNR estimation. Table 4 gave detailed statistical results of the SNR values. The Blood Volume Pulse (BVP) signal mean SNR was 33.44 dB with a 2.48 dB standard deviation, while the median (Q50) was 32.94 dB. The SNRs of the BVP signal change were between 29.56 dB to 39.31 dB. The mean SNR of the Electrodermal Activity (EDA) was 32.56 dB with a 2.38 dB standard deviation. The EDA signal’s median (Q50) SNR was 33.01 dB. The maximum SNR of the EDA signal was 37.62 dB, while the minimum was 25.07 dB. The estimated mean SNR for the Skin Temperature (ST) was 33.22 dB with a 2.05 dB standard deviation. The ST signal’s median (Q50) SNR value was 34.28 dB. There was no value below 26.46 dB or above 34.37 dB for the SNRs of the ST signal. Furthermore, subgroup statistics can be found in Table 4. In agreement with SNR findings, the physiological data acquired from Empatica E4 were good-quality signals.

Table 4 Signal to Noise Ratio (SNR) statistics of the physiological signal acquired via Empatica E4 wristband.
Full size table

Usage Notes

Stress recognition

The most common approach for stress recognition using physiological signals and facial expression includes (1) data collection, (2) signal preprocessing, synchronization, and integration, (3) feature extraction and selection, and (4) machine and deep learning training and validation. A comprehensive overview of all these stages can be found in review papers on stress/emotion detection using wearables53,54,55.

We recommend the following Python libraries for further processing the AKTIVES dataset, which we found useful for preprocessing and feature extraction from physiological data. Numpy Library (https://numpy.org) is utilized for feature extraction from BVP, EDA and ST signals. SciPy Library (https://scipy.org), which provides signal processing algorithms such as signal filtering, is used to filter the BVP and EDA signal. The Pandas Library (https://pandas.pydata.org) is used to resample the psychological signal every 10 seconds. NeuroKit2 Library56, which is a user-friendly package, (https://neuropsychology.github.io/NeuroKit) to extract the features of EDA signals. For the SNR calculation, the GitHub repository52 (https://github.com/psychosensing/popane-2021) was used.

The state-of-the-art face detection and emotion recognition include data collection, feature extraction, and machine and deep learning57,58. For the dataset, face detection and emotion recognition are implemented in Python programming language using the Opencv Library (https://opencv.org/). Emotion detection is performed using the FER Library (https://pypi.org/project/fer/). For facial landmarks, pre-trained facial landmark recognition in the Dlib Library (http://dlib.net/) is used.

Accessing data

The AKTIVES dataset must be used for academic purposes only. Researchers who want to access the AKTIVES dataset must first sign the End User License Agreement (EULA). The EULA is available in the AKTIVES Dataset repository. To access AKTIVES Dataset, a signed EULA must be sent to aktives.project@gmail.com. Only e-mails sent from the academic e-mail address will be considered.