Measuring empathy for human and robot hand pain using electroencephalography

This study provides the first physiological evidence of humans’ ability to empathize with robot pain and highlights the difference in empathy for humans and robots. We performed electroencephalography in 15 healthy adults who observed either human- or robot-hand pictures in painful or non-painful situations such as a finger cut by a knife. We found that the descending phase of the P3 component was larger for the painful stimuli than the non-painful stimuli, regardless of whether the hand belonged to a human or robot. In contrast, the ascending phase of the P3 component at the frontal-central electrodes was increased by painful human stimuli but not painful robot stimuli, though the interaction of ANOVA was not significant, but marginal. These results suggest that we empathize with humanoid robots in late top-down processing similarly to human others. However, the beginning of the top-down process of empathy is weaker for robots than for humans.

Scientific RepoRts | 5:15924 | DOi: 10.1038/srep15924 components induced by painful pictures are influenced by various factors, such as attentional demands 16 , perspective taking 18 , medical training 17 , racial bias 20,21 , duration of stimulus exposure 22 , social context 23 , preceding presentation of another's face picture 24 , and emotional pictures 25 . Thus, ERP is a good tool to investigate various factors of empathy, particularly in temporal aspects.
Robots are becoming more popular and familiar in our everyday lives. Thus, we need to understand how humans socially interact with robots 26 . It is known that humans naturally engage in social communications with computers and virtual characters [27][28][29] . The "media equation" phenomenon may also be applicable to robots [30][31] . If this is the case, we can empathize with both robots and humans. There have been few studies on empathy for pain experienced by non-human others, such as robots and animals. One study 32 reported significantly higher right limbic system activities in subjects observing violent human-human interactions compared to violent human-robot interactions. Their results suggest that we show similar neural activities for empathy directed toward humans and robots, but there are some differences in empathic neural response between empathy for humans and robots. Behavioral studies report that infants have abilities for empathy with non-human agents, such as geometrical objects [33][34][35] . Thus, we focused on the empathy with robots and predicted that human observers could empathize with robots, as well as human others, but with some differences.
The purpose of our study was to investigate neural responses to empathy for pain of a robot hand in comparison with a human hand by measuring ERPs during a well-established empathy-for-pain experimental paradigm 6,8,9,16,18,19,[22][23][24][25][36][37][38] . To our knowledge, there has been no study on empathy for robot pain using EEG. We hypothesized that similar ERPs would be found for both human and robot painful stimuli if we indeed empathize with robots. Moreover, we aimed to investigate different processes underlying responses to human and robot with regard to early bottom-up and late top-down processing.

Results
We measured event-related potentials of human participants who were observing pictures of painful or non-painful situations with human hands or robot hands (Fig. 1). ERP data. We defined the N1, P2, N2, P310, and P3 components based on the grand-averaged values at each channel by visual inspection (Fig. 2). We analyzed P3 with division into the former half (ascending phase of P3) and the latter half (descending phase of P3). The mean amplitude was calculated from each time window at appropriate channels as follows: N1: 90-120 ms at Fz and Cz, P2: 140-190 ms at Fz and Cz, N2: 190-250 ms at Fz and Cz, P310: 300-350 ms at Pz, ascending phase of P3: 350-500 ms at Fz and Cz, descending phase of P3: 500-650 ms at Fz and Cz. The nomenclature of ERP components at 350-650 ms as P3 is based on the previous studies 16,18 on empathy for pain. We used the mean amplitudes for ERPs rather than peak amplitudes because previous studies [16][17][18][19] using similar stimuli and a similar paradigm employed the mean amplitude. Averaged ERPs were calculated from 101-112 trials for each condition after artifact rejection (98.25% of all trials on average). We conducted a two-way repeated-measure analyses of variance (ANOVAs) with the factors of Model (human vs. robot), Pain (painful vs. non-painful), and Electrode (Fz, Cz) for N1, P2, N2, ascending phase of P3, and descending phase of P3, and a one-way repeated-measure ANOVA with the factors of Model (human vs. robot) and Pain (painful vs. non-painful) for P310 as within-subject independent variables. We applied the FDR (false discovery rate, q = 0.05) of Benjamini and Hockberg 39 for post-hoc multiple comparisons of simple main effects.

Right hand
Left hand Non-painful Non-painful Painful P ainful    The simple main effect of Pain was only significant at Fz (F[1,14] = 11.721, p = 0.004, η p 2 = 0.456), indicating that the ascending phase of late P3 recorded at Fz was increased by painful stimuli more than by non-painful stimuli (Fig. 5). The Pain × Model interaction approached significance (F[1,14] = 3.580, , and the other interactions were also not significant. Thus, painful stimuli elicited a larger descending phase of P3 than non-painful stimuli, irrespective of whether the hand belonged to a human or robot (Fig. 6).
Correlation between ERP amplitudes and personal empathic ability. We calculated the mean differential ERP amplitudes between painful and non-painful conditions and performed a correlation analysis between those values and subjects' IRI (Interpersonal Reactivity Index) 40 scores and EQSQ (Empathizing Quotient Systemizing Quotient) 41 scores. We did not find any significant correlation with either Bonferroni correction or FDR.

Discussion
We investigated the effect of the visual appearance of the agent (human and robot) on ERPs induced by empathy for pain. We found common neural responses for empathy directed toward humans and robots in the descending phase of P3 (500-650 ms), and a human-specific response in the ascending phase of P3 (350-500 ms). The descending phase of P3 components was larger for the painful stimuli than for the non-painful stimuli, regardless of whether the hand belonged to a human or robot. In contrast, the ascending phase of the P3 component at the frontal-central electrodes was increased by painful human stimuli but not painful robot stimuli, although the statistical significance was marginal.
Similar to previous studies, our results showed that the P3 amplitude was greater when participants observed human others in painful situations relative to non-painful situations [16][17][18][19][21][22][23][24][25]38 . A important new finding was that the pain effect (differential amplitude between painful and non-painful conditions) shown in the ascending phase of P3 (350-500 ms) was only significant for human stimuli, but in the descending phase of P3 (500-650 ms), it was significant for both human and robot stimuli. These findings suggest that visual cues for humanity (human or robot) partially modulate the top-down controlled processes of empathy for pain, but human observers fundamentally show empathic neural responses to robots, similarly to human others. Li and Han (2010) 18 reported that the ascending phase of the P3 (370-420 ms) response to others in painful situations decreased when participants were told to take the other's perspective while performing a pain judgment task, but the descending phase of P3 response was not influenced by perspective taking 18 . These results were similar to our findings; ERP modulation by visual cues of humanity was found in the ascending phase of P3 (350-500 ms) but not in the descending phase of P3 (500-650 ms). Therefore, we speculate that the difference in empathy between human and robot others was related to perspective taking, which was more difficult for robot others. Because there was no difference in the pain effect between humans and robots at the descending phase of P3 (500-650 ms), it is suggested that emotional regulation, which is reflected in LPP after 600 ms 22,42,43 , functions similarly with respect to robots and humans. Thus, humans can attribute humanity to robots and feel their pain. Because the basic shape of the robot hand in the present study was the same as that of the human hand, the human participants may have been able to empathize with the robot hand. It is thus necessary to test whether a robot hand in very different shape (e.g., a robot hand without fingers) can elicit similar empathic responses in a future study. We did not find any effect of pain conditions on N1 at 90-120 ms or P2 at 140-190 ms. It is controversial whether the N1 (N110) amplitude is influenced by empathy for pain. Recent ERP studies suggested that an early empathic response induced a positive shift of the N1 (N110) component 16,17,19,24 . However, several studies suggested that the N1 (N110) amplitude was not influenced by empathy for others in pain 21,23,37 , which is consistent with the present study.
We found a significant pain effect on N2 at 190-250 ms, which was larger for painful stimuli than for non-painful stimuli, irrespective of whether the hand belonged to a human or robot. This finding conflicts with the previous studies, which show a negative shift of N2 in painful situations.
We found a main effect of model (human and robot) at both P2 and N2, but we did not observe an interaction between model and pain. Thus, differences in hand appearance or visual cues for humanity would modulate very early brain potentials. These results suggest that visual humanity coding occurs in early visual processing.
The P310 (300-350 ms) amplitudes were greater for robot painful stimuli than human painful stimuli, and greater for human non-painful stimuli than human painful stimuli. In other words, P310 was weaker for the human painful stimuli than for the robot painful and the human non-painful stimuli. Fan and Han (2008) 16 reported that the ERP amplitude at 220-300 ms around Pz was larger for non-painful human stimuli than for painful human stimuli. Their result is similar to our findings, although their time window (220-300 ms) was earlier than ours (300-350 ms). The ERP component around 300 ms reflects the process of detecting an infrequent target, usually in the oddball paradigm. It is modulated not only by stimulus frequency, but also by unnaturalness of visual stimuli 44 . The difference between the human painful hand and the robot painful hand may be explained by the unnaturalness of robot hands cut by knives.
It may be argued that the weak response in the ascending phase of P3 with the robot hand may have been caused by differences in stimulus contrast and size between the human and robot hands in pictures. It is reported that pain perception is modulated by the visual size of the body 45 . We carefully controlled stimulus luminance between the painful and non-painful stimuli for both human and robot hands, but did not control the contrast or size between human and robot hands. Then, we conducted a perceptual experiment to measure reaction times to judge pain or lack of pain in pictures with the human-and robot-hand stimuli identical to those of the main experiment. We found a main effect of the human and robot hands in reaction times; the accelerated judgments were faster for the human hands than for the robot hands. Therefore the pain discrimination was easier for the human-hand stimuli than for the robot-hand stimuli, possibly because the robot stimuli had low color contrast and the robot hand was larger than the human hand, although the knives were identical. However, it is not reasonable to consider that the different response of the ERP in the ascending phase of P3 between the human and the robot hands was caused by differing visibility of the stimuli. Because the reality of the stimuli induces a pain effect in N2 (the early component) but not in the late components 16 , the visibility also should affect the early processing and not late processing such as P3. The effect of the visual appearance of the human or robot hand on the pain condition was limited to the ascending phase of P3 and not found in N2 or the descending phase of P3 in the present study. This selective effect cannot be explained solely by stimulus visibility.
Our study showed that humans empathize with both humans and humanoid robots during late top-down processing, as confirmed by identical pain effects in the descending phase of P3. However, there was a difference in the beginning of the top-down-controlled processing of empathy. The pain effect of the ascending phase of P3 was only significant when empathy was directed toward humans, possibly because of the difficulty in taking the perspective of robots. However, the interaction of the human-robot appearance condition and the painful condition by ANOVA was not significant, but just marginal. Thus, the conclusion regarding the ascending phase of P3 should be carefully considered on its limitation.

Methods
Participants. Fifteen undergraduate and graduate students (3 females and 12 males, mean 21.7 years old ± 1.4 standard deviation [SD], 1 left-handed and 14 right-handed) participated in the experiment. All participants provided written informed consent and had normal or corrected-to-normal vision. The methods of the experiment and all experimental protocols were approved by the Committee for Human-Subject Studies at Toyohashi University of Technology, and the experiment was strictly conducted in accordance with the approved guidelines of the committee.
Stimuli and apparatus. Participants were shown 56 color photographs of a human or robot hand in painful or non-painful situations. The images were from a first-person perspective. The painful pictures illustrated accidents that may happen in everyday life, such as a finger cut by a knife, and the corresponding non-painful pictures were made by moving knife-like objects a small distance from the hand (Fig. 1). Seven different situations were used. Left-hand conditions were mirror-reversed pictures of right-hand stimuli. The luminances of painful and non-painful pictures were controlled to be identical (average 14.030 cd/m 2 ). Stimulus presentation was controlled by a computer (Dell Vostro 420, Intel Core2Duo E8500 with 3.16 GHz, 4.0 GB RAM, ATI Radeon HD3400, running Microsoft Windows 7) Scientific RepoRts | 5:15924 | DOi: 10.1038/srep15924 and displayed at the center of a gray (9.018 cd/m 2 ) background of a CRT color monitor (TOTOKU CV921X, 30 × 40 cm, 1024 × 768 pixels, 60-Hz refresh). Each stimulus was 20 × 15 cm, 512 × 384 pixels (11.42 × 8.58 deg at a viewing distance of 100 cm).
Procedure. The ERP recording experiment consisted of eight sessions. Each session included 56 recording trials (56 different pictures: combinations of 7 situations, 2 model conditions (human or robot), 2 pain conditions (painful or non-painful), and left and right hand pictures) and 10 filler trials with pain judgment tasks in a random order. Filler trials were added to sustain participants' attention to pain components in the stimulus and were removed before the ERP analysis. At the beginning of each trial, the stimulus was displayed at the center of the screen (500 ms) following a fixation cross presentation (1000 ms). If it was an ERP recording trial, the next trial continued after a blank screen (randomized at 900-1600 ms). If the trial was the filler, a question mark was presented after stimulus presentation, and participants judged whether the picture was painful or non-painful by pushing one of two buttons held in each hand. The next trial continued after a blank screen was presented (randomized at 1400-2100 ms). Correspondence between buttons and options was counter-balanced among the participants.
Measurements of subjective ratings. After the ERP recording experiment, subjects re-observed all the stimuli without time restrictions. They were asked to provide two ratings for each stimulus: the intensity of pain supposedly felt by the model in the stimuli, and the unpleasantness they felt when they observed the picture. They responded using a mouse to select a value on the Visual Analog Scale on the monitor (approximated as 0-100). The subjects felt greater pain intensity and self-unpleasantness in the painful than in the non-painful conditions, and the difference between painful and non-painful conditions was larger with human-hand stimuli than with robot stimuli. Detailed results and statistical analysis are described in the Supplementary Materials. EEG recording. EEG signals were recorded by a biological amplifier (TEAC Polymate AP1132, 32 channel, 16 bit, up to 2000 Hz sampling) using 32 scalp electrodes (Neuroscan, Quik-Cap) according to the extended 10-20 system with two earlobe electrodes. The electrode at the left earlobe was used as reference. The electrode impedance was kept below 10 Ω . The EEG was amplified (band pass 0.5-30 Hz) and digitized at a sampling rate of 1000 Hz. The ERPs in each condition were averaged separately offline with an epoch beginning at stimulus onset and continuing for 700 ms. Trials containing eye blinks, eye movements, or muscle potentials exceeding ± 50 μ V were excluded from the average for each electrode. Measures of participants' empathy characteristics. Before the ERP recording experiment, participants filled out two questionnaires including the interpersonal reactivity index (IRI) 40 and the Empathizing Quotient Systemizing Quotient (EQSQ) 41 to measure personal empathy ability. The Japanese versions of both questionnaires were used 46,47 . IRI scores of the present participants were similar (difference within 1.0) to those of control participants of Decety et al. (2010) 17 except for the empathic concern subscale (5.93 lower than noted by Decety et al., 2010).