Article | Open

Influence of facial feedback during a cooperative human-robot task in schizophrenia

  • Scientific Reports 7, Article number: 15023 (2017)
  • doi:10.1038/s41598-017-14773-3
  • Download Citation
Received:
Accepted:
Published online:

Abstract

Rapid progress in the area of humanoid robots offers tremendous possibilities for investigating and improving social competences in people with social deficits, but remains yet unexplored in schizophrenia. In this study, we examined the influence of social feedbacks elicited by a humanoid robot on motor coordination during a human-robot interaction. Twenty-two schizophrenia patients and twenty-two matched healthy controls underwent a collaborative motor synchrony task with the iCub humanoid robot. Results revealed that positive social feedback had a facilitatory effect on motor coordination in the control participants compared to non-social positive feedback. This facilitatory effect was not present in schizophrenia patients, whose social-motor coordination was similarly impaired in social and non-social feedback conditions. Furthermore, patients’ cognitive flexibility impairment and antipsychotic dosing were negatively correlated with patients’ ability to synchronize hand movements with iCub. Overall, our findings reveal that patients have marked difficulties to exploit facial social cues elicited by a humanoid robot to modulate their motor coordination during human-robot interaction, partly accounted for by cognitive deficits and medication. This study opens new perspectives for comprehension of social deficits in this mental disorder.

Introduction

Schizophrenia is characterized by symptoms such as delusions, disorganization and hallucinations. Social deficits are as well a core feature of this disorder1,2. In particular, the impoverished ability to process social information and to modulate interpersonal behavior accordingly have severe negative impact on schizophrenia patients’ social life3,4. Engaging and maintaining social interactions relies to a large extent on non-verbal cues. An abundant literature demonstrates that patients with schizophrenia perform poorly when requested to interpret cues conveyed by facial emotions5, hand gestures6, body postures7 and gaze direction8. Among these, facial emotion interpretation has been reported to be highly associated with social competence in schizophrenia9,10. Non-verbal cues, such as facial emotions, provide feedback about intentions and emotional states of others, that influences our behavior11,12,13,14. The ability with which two partners coordinate their movements with one another is often quantified through a measure of synchrony across the two partners’ movements15,16,17. It has been shown that the affiliation between co-actors can lead to spontaneous synchronization of body movements18. Conversely, synchrony during social interaction can lead to a higher affiliation rating and cooperation between individuals19. Recent studies suggest a general impairment in interpersonal synchronization in patients with schizophrenia20,21. However, those studies do not explore the specific influence of social feedback on interpersonal synchronization. This constitutes the core question of our work. In particular, we hypothesize that non-verbal social cues influence our ability to coordinate our movements with that of others, even in very simple motor imitation tasks. We refer to this interplay between social cues and motor synchrony as social motor coordination22. This study is of particular relevance in the context of schizophrenia that affects both synchrony and the interpretation of social cues. To validate our hypothesis, traditional human-human interaction studies of social-motor coordination are limited due to the impossibility to precisely controlling for the social feedback. Therefore, one cannot separate easily the effect of the type (social versus nonsocial) and the frequency of feedbacks on the interaction. To be able to explore quantitatively the link between social feedback and motor synchrony, it is critical to provide comparable and controlled social cues during the interaction.

Socially assistive robotics (SAR) stem from a trend that endows robots with social, emotional and cognitive competences to enhance human-robot interactions. SAR have been used to assess social competences and therapeutic treatment for medical conditions with deficits in social competences, such as dementia23 or autism24,25,26. There is a wealth of applications of SAR as a diagnosis tool to provide a systematic assessment of symptoms related to social deficiencies in children with autism spectrum disorders (ASD)27,28,29. Moreover, SAR can improve engagement and elicit novel social behaviors in ASD individuals, including in subjects who do not interact socially with human therapists30,31. This line of research develops simple social interaction tasks in order to motivate and engage patients to interact socially with the robot. Social skills learned during these human-robot interactive sessions can then be transferred to similar interactions with human partners32.

As schizophrenia shares social symptomatology with ASD, such as a social withdrawal, an impoverished theory of mind and impairments in the interpretation of facial social cues, we posit that the promising results of SAR for ASD could be extended to schizophrenia. Only one study has used pet robots in schizophrenia, aiming at promoting social and emotional functioning in institutionalized patients33. However, to our knowledge, human-robot interaction during a collaborative task has never been exploited to monitor and study social interactions in schizophrenia.

In this work, we propose to use iCub, a humanoid robot able to display controlled social feedback in the form of facial smiles. Our study is hence the first attempt at assessing the potential of humanoid robots to study social cognition in schizophrenia. We selected a simple mirroring task, which consists for the subject in following the robot’s hand motion as accurately as possible. In our approach, the robot is endowed with the ability to adapt the amount and type (i.e., social or nonsocial) of feedback it gives to its human partner based on their synchrony: the more synchrony, the more positive feedbacks. As synchrony can induce an increase in affiliation, this, in turn, leads to an increase in the dyad’s motor coordination (i.e., affiliation-induced synchrony). The difficulty is in solving the causal ambiguity of this interactive loop (i.e., moving from correlation to causation). To help resolve this problem, we introduce a third condition in which the robot offers a neutral face. This serves as a baseline to break one link of the loop. We evaluate the effect of the type of feedback (i.e., social and nonsocial) provided and the frequency at which the feedback is generated, in three conditions:

  1. 1.

    a neutral condition where no feedback is displayed

  2. 2.

    a nonsocial condition, where a tablet mounted on the robot’s head displays a plus sign, and

  3. 3.

    a social condition where the robot displays a smiling facial expression using luminous color light-emitting diodes under the surface of its face, representing the mouth and the eyebrows.

The nonsocial and the social feedbacks are triggered in real time during the interaction by the same algorithm, according to the participant’s performance. In both cases, the feedback is displayed for one second, and is followed by a refractory period of 3 seconds were nothing is displayed (i.e., a neutral face for the social condition, and the tablet without the cross in the nonsocial condition). To obtain congruent feedbacks across conditions, we ensured that the surface covered by the luminous mouth and eyebrows of the iCub robot is the same as the surface covered by the cross on the tablet. Furthermore, the color used to trace the cross is the same as iCub’s facial diodes. Finally, the luminance is equivalent as both feedbacks are displayed by light-emitting sources on a white background.

Furthermore, we explored clinical correlates using clinical standardized evaluation of symptoms severity in schizophrenia. Besides the clinical assessments, all participants were evaluated on the Trail Making Test (TMT-A and B) for cognitive functioning. We also characterized the participants with a measurement of Theory of Mind using the Mind Perception Questionnaire (MPQ) after the end of the trials. The MPQ is designed to evaluate how individuals perceive living and non-living things in terms of experience (e.g. How much is the robot capable of experiencing physical or emotional pleasure?) and agency (e.g. How much is the robot capable of thinking?). This setting is illustrated on Fig. 1. We exploit this setting to validate three hypothesis. First, we hypothesize that the general impairment in synchrony of patients with schizophrenia transfers to the interaction with a humanoid robot. Second, we hypothesize that positive social feedbacks should facilitate interpersonal synchronization in control subjects. However, we expect that this facilitation effect is not present in schizophrenia patients, due to their impairment in dealing with social cues.

Figure 1
Figure 1

Overall results of the statistical analysis. (A) The schizophrenia group, compared to the control group, has a lower measure of synchrony regardless of the existence and the type of the feedback. (B) For the control group, the social feedback has a facilitatory effect on the motor coordination. In contrast, for the schizophrenia group, the social feedback has an impeding effect on the motor coordination. (C) For patients with schizophrenia, the synchrony index during the interaction is associated (i.e., negatively correlated) with cognitive abilities more than with symptomatology. (D) The association (i.e., the negative correlation) between cognitive flexibility and synchrony index is stronger in the presence of the social feedback. Such an observation is not present for the control group.

Results

Participants Group Comparisons

With respect to age and gender, healthy control subjects were matched to patients with schizophrenia; see Table 1. The median age of the patients (Mdn = 29) and the healthy control subjects (Mdn = 28) were statistically comparable (U = 218, z = −0.56, p = 0.57, r = 0.09). The ratio of male to female participants in each group (17/5 for patients, and 15/7 for control group) did not differ significantly (χ2(1) = 0.11, p < 0.73).

Table 1: Demographic characteristics of the participants. Median and range [minimum-Maximum] for non-parametric tests; Education: years of education; NSS: Neurological soft sign scale; TMT: Trail Making test; PANSS: positive and negative syndrome scale.

Synchrony & Social-Motor Coordination

To study the effect of the nature and frequency of feedback on the motor coordination, a multiple linear regression was performed. This model predicts the synchrony index based on the group, the nature of the feedback and its frequency. A significant regression equation was found (F(9, 649) = 23.2, p < 0.001), with R2 = 0.24; see supplementary materials. This model showed that the schizophrenia group has, irrespectively of the nature and the frequency of the feedback, a lower synchrony with the robot during the coordination task compared to the control group (F(1, 649) = 25.06, p < 0.001); see 1A.

Moreover, the linear regression showed the sensitivity of synchrony to the frequency of the feedback in each condition (SMCi). This metric was first exploited to study the effect of the presence of any type of feedback (social and nonsocial), averaged over both control and schizophrenia groups. The results showed a decrease of the SMCi of 42% in presence of feedback compared to the neutral case (F(1, 649) = 5.196, p = 0.023). Secondly, the SMCi was used to contrast the influence of social feedback compared to nonsocial feedback. The social feedback condition improved the SMCi by 85% (F(1, 649) = 3.884, p = 0.049). Furthermore, this effect interacted with the group. The social feedback resulted in a higher SMCi in the control group compared to the schizophrenia group (F(1, 649) = 5.607, p = 0.018). The normalized measure of SMCi across conditions is illustrated in Fig. 1B. As can be seen, with respect to SMCi, the only statistically significant difference across group lied in the social condition.

Examination of Confounding Factors

We conducted a second linear regression, controlling for variables for which the groups were matched; i.e., age and gender. Although no statistically significant effect of age was detected (F(1, 647) = 1.129, p = 0.288), this model showed that female participants, compared to male participants, had a lower measure of synchrony (F(1, 647) = 50.0167, p < 0.001). No statistically significant effect of the different robot hand’s trajectories was detected (F(4, 654) = 0.534, p < 0.711). The results obtained in terms of synchrony and SMCi according to these observations remains consistent with our analysis.

Clinical Correlates

To gain insight into the pathological underpinning of schizophrenia patients’ impairment in this social coordination task, we conducted a correlation analysis between synchrony index and clinical evaluation of symptom severity in the patients group. Our findings showed that patients’ performance regarding synchrony was highly negatively correlated to cognitive flexibility as indexed by the difference between TMT B and TMT A (TMT B-A) performances (R2 = −0.34, p < 0.001). This correlation was statistically stronger not only compared to psychotic symptomatology (z > 5.23, p < 0.001), but also compared to the correlations found in the control group (z > 4.48, p < 0.001). These correlations and their significant pairwise comparisons are illustrated on Fig. 1C.

Furthermore, we explored the correlations between measures of clinical evaluation and the synchrony index across conditions; see Fig. 1D. These correlations suggest that in schizophrenia patients, the synchrony index was negatively correlated to TMT B-A performance more than to the combination of psychotic symptoms, especially in both social and nonsocial feedback conditions (z > 2.55, p < 0.011). Focusing on the correlation between the synchrony index and TMT B-A performance for patients with schizophrenia, an increasing trend was observed across conditions. However, only the difference between neutral and social conditions was statistically marginally significant (z = 1.91, p = 0.055). Finally, the synchrony index was highly correlated with NSS for both patients with schizophrenia (τ = −0.15, p = 0.000) and control participants (τ = −0.26, p = 0.000).

The statistical analysis of the Mind Perception Questionnaire did not show any group (patients or controls)-dependent effect of agency attribution on the performance in terms of synchrony. Detailed statistical analysis and results are given in the Supplementary Materials.

Finally, we explored the correlation between the medication dosage (chlorpromazine equivalents CPZ) and the synchrony index. The synchrony index is negatively correlated to the CPZ in general (τ = −0.10, p = 0.01). Focusing on the different conditions, we observe a statistically significant correlation in the facial condition (τ = −0.13, p = 0.05), but no statistically significant correlation in the tablet condition (τ = −0.07, p = 0.31) and the neutral condition (τ = −0.11, p = 0.14).

Discussion

In this study, we explored the effect of social feedback on motor coordination in schizophrenia during a collaborative task with a humanoid robot. Our objectives were: first, to explore if interpersonal motor coordination impairments already shown during interactions with a human partner20,21 also apply to interactions with a robot partner; second, to investigate social-motor coordination in schizophrenia by quantifying the effect of social feedback on motor coordination; and third, to clarify the factors underlying abnormal behaviors observed during the cooperation task.

First, we confirmed that patients with schizophrenia are impaired in their ability to synchronize with a robot partner in simple motor imitative tasks compared to control subjects. This extends previous findings of synchrony impairments in schizophrenia in human-human interactions20,21 to the interaction with humanoid robots.

More specifically, we investigated how social feedbacks influences the synchrony of the participants during the interaction. We showed that, compared to the neutral condition, the nonsocial feedback deteriorated motor synchrony to the same extent for both control and schizophrenia groups. We speculate that even though the nonsocial feedback is task-relevant (i.e., computed based on the quality of the interaction), its relevance to the task remained unclear to both control and patients with schizophrenia. Therefore, participants were unable to exploit nonsocial feedback as a cue to improve synchrony. Moreover, it may have shifted the participants’ attention away from the robot motions, thus deteriorating the synchrony. This hypothesis is supported by studies showing that visual attention modulates the strength of interpersonal coordination34.

Second, our findings showed that only nonclinical participants benefited from the social feedback elicited by the robot and that this feedback modified their movement accordingly to the proposed coordination goal. To our knowledge, this study is the first to demonstrate that social feedback improves coordination behavior more than nonsocial feedback in control subjects. Comparing the results obtained with social versus nonsocial feedback shows that the emotional content of the feedback facilitated the interaction in control subjects and not the task-related aspects of the feedback.

Importantly, our results revealed that compared to controls, patients with schizophrenia failed to improve their social-motor coordination in the presence of social feedback. This absence of facilitation in schizophrenia may be due to an impairment in the automatic link between perception of social cues (i.e., positive facial emotions such as smiles) and motor coordination. This hypothesis is in line with a body of literature that investigates the impairments in automatic processing in schizophrenia35,36,37,38,39. Alternatively, the lack of a facilitatory effect of social cues in schizophrenia participants may be due to specific cognitive deficits, such as an impaired ability to perceive and interpret facial social cues (i.e., smiles). In a preliminary study, we showed that patients with schizophrenia were able to accurately recognize the valence of facial emotion elicited by the same iCub robot40. This speaks against the concern expressed above.

Finally, our results showed a high negative correlation between patients’ social-motor coordination and their performance in the Trail Making Test (TMT). This correlation was stronger than the correlation with positive, negative symptoms and other clinical symptoms. The TMT B-A score has been shown to be a valid measure of cognitive flexibility, one of the main dimensions of executive functioning41. Cognitive flexibility is the ability to shift from one cognitive operation to another one42 and to interrupt automatic responses to come back to top-down cognitive control43. In non-pathological individuals, the motor-coordination response to social feedback is automatic, and thus does not rely on top-down cognitive control. Our results are thus in line with this observation, as the correlation between synchrony and TMT results was low for control participants.

For schizophrenia participants, a high correlation was obtained, particularly concerning the part B of the TMT; the part B of the TMT is considered as a test of higher level cognitive abilities such as mental flexibility. This finding supports the hypothesis that impairments in the brain circuits related to social processing are compensated by higher cognitive processes such as those involved in cognitive flexibility44,45. In the context of such an impairment, coordination of rhythmic behavior between individuals with schizophrenia engaged in a joint activity is a demanding task as it requires both precision and flexibility46. Our cooperation task is particularly demanding in that the participant has to simultaneously take into account the robot’s facial feedback and coordinate his/her movements with those of the robot while concurrently monitoring the overall integrated ensemble output. Patients with schizophrenia may fail to coordinate their own actions with others’ actions while maintaining effortful control of their own movements. There is abundant evidence that patients with schizophrenia have difficulty using positive feedback to adaptively guide their behavior47,48.

Furthermore, we observed a significant correlation between the synchrony index and medication dosage for the patients. This shows that patients treated with high dosage do not benefit from the facial cues as well as patients with lower dosage. In49, authors analyzed the effect of chlorpromazine equivalents on facial emotion perception. Their results suggest a marginal relationship between higher dosage and greater degree of impairment on tests of facial emotion perception. This observation could explain our result.

Our study has some limitations. First, our sample size is relatively small. Therefore, the results should only be generalized with caution. A second limit concerns the social cues used (i.e. facial feedback). The facial expressions of the robot include only a very narrow aspect of the complexity of a real human facial expression. Future work should explore more realistic and rich social cues, such as gaze, spoken language or haptic communication for the study of physical interactions between humanoids robots and individuals with schizophrenia. Furthermore, the processes by which patients in an interaction start, maintain and end their perceived connection to a robot needs to be further explored. Another important limit of our study concerns the lack of a visual perception measure. Indeed, patients with schizophrenia are known to be impaired at organizing and exploring the visual environment50 which can affect visual organization in space, the processing of low-spatial frequencies, and the pattern of eye movements. Further studies using for example eye-tracking methodology to assess participants’ gaze toward the robot during the task are thus needed.

To our knowledge, this is the first study that examined human-robot interaction in the context of a cooperation task in individuals with schizophrenia. Unlike with robots, one cannot easily manipulate how often and in which manner humans express social cues. This study exploited the fact robots can be used to provide social cues in a controlled way. Specifically, it offered a first systematic assessment of the effect of providing social positive feedback on schizophrenia.

In our study, patients with schizophrenia displayed reduced cooperation ability compared to controls during human-robot interaction through all conditions and, in particular, in the social cue condition. This may be due to the patients’ impoverished ability to process the social cues expressed by the humanoid robot or to their general inability to use social cues to modulate their behavior. In addition, we observed that antipsychotic medication affected the patients’ performance negatively. This suggests that antipsychotic medication reduces patients’ social competences. This result is in line with other studies showing an impeding effect of neuroleptics on emotional facial expressions recognition in schizophrenia spectrum disorders49.

Our study evaluated only one type of social feedback, namely positive feedback conveyed through a smiling face. Social feedbacks are crucial to successful interaction and communication and are conveyed by different modalities. Further studies are needed for evaluating and comparing the effectiveness of other types of feedbacks (e.g. verbal or haptic) in schizophrenia. Social robots, and in particular humanoid robot, may offer a useful tool, in this endeavor, as one can manipulate also their gaze and haptic interactions.

All patients accepted easily to interact with the iCub robot and engaged naturally in the interaction with the robot. This provides positive evidence of the acceptability of humanoid robots for further interaction protocols with schizophrenia patients. The rapid progress in humanoid robotics offers tremendous possibilities for innovation in the study of social interaction deficits. All the above lead us to conclude that robots constitute promising tools for studying social dysfunctions in patients with schizophrenia.

Our study was motivated by the wealth of publications showing the potential of social robots to accompany rehabilitation protocols in ASD, a mental disease bearing similar deficits in social cognition to schizophrenia. However, unlike patients with ASD who are responsive to simple social features emitted by robots, patients with schizophrenia did not benefit from the robot’s social feedback. We cannot exclude that this may result from the simplicity of the social cues generated by the robot. If one had used social robots bearing a stronger resemblance to human faces, such as androids51, the effect may have been different. Further studies using alternative types of social robots may help to confirm or infirm the potential of robots as a tool for therapeutic enhancement of social abilities in schizophrenia. Our study assessed the effect of social cues on a single session with patients who had no previous experience interacting with the robot. As therapeutic effects can only be assessed through repeated sessions, further works should explore the effect of interaction sessions with the robot on the long term.

Methods

Participants

Participants and exclusion criteria

We recruited 44 participants; 22 schizophrenia outpatients, and 22 age and gender-matched healthy participants. Patients were recruited from the University Department of Adult Psychiatry (CHRU Montpellier, France) and fulfilled the Diagnostic and Statistical Manual of Mental Disorders criteria for schizophrenia.

The control participants were recruited in the Montpellier area. They were screened for current psychiatric illness using the Mini-international Neuropsychiatric interview. The control participants did not meet any criteria for current axis I disorder of the DSM-IV-TR.

Exclusion criteria for both the clinical and nonclinical groups were (a) history of head trauma, (b) known neurological disease, (c) an actual ECT treatment, (c) substance abuse and or substance dependence (excluding tobacco and cannabis), and (d) people deprived of their liberty. All participants were native French speakers with a minimal reading level (validated using the fNART test) and were able to understand and perform the social-coordination task described in the following section.

All patients were taking medication, and doses were converted in chlorpromazine equivalents (mean dose = 286 mg, SD = 118, see Table 2). Patients received a neuroleptic treatment, either typical (N = 1) or atypical (N = 21). One patient was also administered with an antiparkinsonian treatment. Six patients were treated with benzodiazepines.

Table 2: Medication of the participants with schizophrenia.

All patients were interviewed by members of the specialized multidisciplinary team of the University Department of Adult Psychiatry, which belongs to a French national network of 10 Schizophrenia Expert Centers (Bordeaux, Clermont-Ferrand, Colombes, Créteil, Grenoble, Lyon, Marseille, Montpellier, Strasbourg, Versailles), set up by a French scientific cooperation foundation, FondaMental Foundation (www.fondation-fondamental.org) and created by the French Ministry of Research.

Except for the SCID, patients were assessed by trained clinical psychologists who rated the PANSS and other clinical scales after a unique clinical interview.

All participants provided written informed consent, prior to the experiment approved by the National Ethics Committee (CPP Sud-Méditerranée-III, Nîmes, France, #2009.07.03ter and ID-RCB-2009-A00513-54) and conforming to the Declaration of Helsinki. Participants were evaluated on the Neurological Soft Signs Scale (NSS)52 to assess subtle abnormalities in sensory-perceptual motor functions directly associated with schizophrenia53,54 or induced by neuroleptic medications55. Patients also completed the Positive and Negative Syndrome Scale (PANSS)56. Cognitive assessment included part A and B of the trail making test (TMT). The difference between the completion time of TMT A and TMT B was used to provide an indicator of cognitive flexibility57. See Table 1 and supplementary materials for further details.

Design

Human-robot collaboration task

We used the iCub58, a 1.20 m tall humanoid robot with 53 degrees of freedom, designed to offer a platform for the study of cognition and for human-robot social interactions. The quality of the collaboration between the robot and the participant was evaluated in the context of an imitation task called the mirror game59, whereby two players mirror each other’s hand motions. The robot provided the user with positive feedback whenever the synchrony between their motions improved. Feedback was either social or nonsocial (see Fig. 2). The interaction was hence composed of two aspects: motor coordination through the imitation game, and social non-verbal communication through the robot’s feedback. This socio-motor coupling enables a collaboration between the robot and the participant.

Figure 2
Figure 2

Schematic of the protocol. (A) Types of feedback displayed by the robot. (B) Human-robot collaboration task. (C) Evaluation of the quality of the interaction based on synchrony. (D) Feedback is displayed when the synchrony index increases. (E) The socio-motor coordination index measures the sensitivity of the synchrony to the frequency of the feedback.

Participants were instructed to follow the hand movements of the robot with their own hand. The robot was programmed to play the mirror game as an assistive leader60. This enabled the robot to adapt to slowdowns in the case that the participant was lagging importantly. This assistive mechanism ensures to maintain the interaction even with poorly synchronized participants. To provide diversity in the robot’s movement, the robot switched across five different reference trajectories. To assure that the robot’s motion was human-like, the trajectories were generated according to a human-movement framework presented in past research61. The behavior of the robot (i.e., control parameters) was fixed throughout the experiment, ensuring that we observe only the effect of manipulated variables, i.e., nonsocial and social feedback.

The participant sat in front of the robot and engaged in the coordination task with all possible combinations between conditions (i.e., neutral, nonsocial, and social feedback) and robotic leading behaviors (i.e., 5 different motion signatures); see Fig. 3. This means that each participant performed 15 randomly-ordered trials, each trial lasting 60 seconds. In order to record the participants’ motions, they were asked to hold a red ball attached to a handle that was tracked by a camera mounted on the ceiling. This led to total of 660 recorded trajectories for the analysis.

Figure 3
Figure 3

The three conditions of the experiment. (A) Neutral face (B) Nonsocial positive feedback (C) Social positive feedback.

Positive social and non-social feedback

The three conditions used are contrasted in Figs 2A and 3: in the social condition, the robot generated a smile using its facial LEDs; in the nonsocial condition, a plus sign was displayed on a tablet fixed on the robot’s head, hiding its face; in the baseline condition, the robot had a neutral face. Feedback was triggered each time the coordination was improved with respect to (1) position error, (2) velocity error, and (3) sum of velocities, compared to the last 5 seconds. Even though no positive feedback was displayed in the baseline condition, we still computed the number of events that could trigger the feedback; see supplementary materials for further details. This served to contrast participants’ synchrony across conditions. Participants took part in all three conditions in randomized order.

Statistical analysis

The experimental design was composed of one independent between-subject variables, a group factor (control and schizophrenia), and one independent within-subject variable, a condition factor (neutral, nonsocial, and social). The number of positive feedbacks during the interaction was considered as a covariate. Demographic characteristics were statistically compared across groups using non-parametric U-Mann-Whitney tests for continuous variables (e.g., age), and Chi-squared tests for binary variables (e.g., gender). Pairwise comparisons between groups were performed using t-tests when necessary.

To evaluate synchrony, we computed the average velocity error between the participants and the robot. This measure (i.e., Synchrony index) is used as the dependent variable in our statistical analysis to study the effect of group and condition. A dummy variable (i.e., Feedback) compares the social and nonsocial conditions with the neutral condition; and a nested dummy variable (i.e., Social) compares the social to the nonsocial condition. To study the sensitivity of the synchrony index to the number of positive feedbacks, the frequency of the feedback was included in the model as a covariate. The estimated slope for this covariate shows how the synchrony index and the frequency of feedback are correlated (i.e., Socio-Motor Coordination index or SMCi). The estimated slope in the neutral case was used as a baseline.

For the clinical correlation analysis, we used a linear regression with the synchrony index as the dependent variable. To compare the correlation coefficients, we used a two-tailed Fisher z-score test.

We explored the correlation between medication dosage and synchrony with a Kendall Tau test.

Data and materials availability

Fully anonymised data is available upon request.

Additional Information

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Couture, S. M., Penn, D. L. & Roberts, D. L. The functional significance of social cognition in schizophrenia: a review. Schizophr. Bull. 32, S44–63 (2006).

  2. 2.

    Penn, D. L., Corrigan, P. W., Bentall, R. P., Racenstein, J. & Newman, L. Social cognition in schizophrenia. Psychol. Bull. 121(1), 114 (1997).

  3. 3.

    Browne, S. et al. Quality of life in schizophrenia: relationship to sociodemographic factors, symptomatology and tardive dyskinesia. Acta Psychiat. Scand. 94(2), 118–124 (1996).

  4. 4.

    Green, M. F., Horan, W. P. & Lee, J. Social cognition in schizophrenia. Nat. Rev. Neurosci. (2015).

  5. 5.

    Kohler, C. G. et al. Facial emotion recognition in schizophrenia: intensity effects and error pattern. Am. J. Psychiat. 160(10), 1768–1774 (2003).

  6. 6.

    Matthews, N., Gold, B. J., Sekuler, R. & Park, S. Gesture imitation in schizophrenia. Schizophr. Bull. 39(1), 94–101 (2013).

  7. 7.

    Thoma, P. et al. Do you see what I feel?–Electrophysiological correlates of emotional face and body perception in schizophrenia. Clin. Neurophysiol. 125(6), 1152–1163 (2014).

  8. 8.

    Rosse, R. B., Kendrick, K., Wyatt, R. J., Isaac, A. & Deutsch, S. I. Gaze discrimination in patients with schizophrenia: preliminary report. Am. J. Psychiat. 151(6), 919–921 (1994).

  9. 9.

    Mueser, K. T. et al. Emotion recognition and social competence in chronic schizophrenia. J. Abnorm. Psychol. 105(2), 271 (1996).

  10. 10.

    Hooker, C. & Park, S. Emotion processing and its relationship to social functioning in schizophrenia patients. Psychiatry Res. 112(1), 41–50 (2002).

  11. 11.

    Knoblich, G. & Jordan, J. S. Action coordination in groups and individuals: learning anticipatory control. J. Exp. Psychol.-Learn. 29(5), 1006 (2003).

  12. 12.

    Sebanz, N., Knoblich, G. & Prinz, W. Representing others’ actions: just like one’s own? Cognition 88(3), B11–B21 (2003).

  13. 13.

    Bhanji, J. P. & Delgado, M. R. The social brain and reward: social information processing in the human striatum. Wiley Interdiscip. Rev. Cogn. Sci. 5(1), 61–73 (2014).

  14. 14.

    Nummenmaa, L. et al. Emotions promote social interaction by synchronizing brain activity across individuals. Proc. Natl. Acad. Sci. 109(24), 9599–9604 (2012).

  15. 15.

    Schmidt, R. C. & O’Brien, B. Evaluating the dynamics of unintended interpersonal coordination. Ecol. Psychol. 9(3), 189–206 (1997).

  16. 16.

    Schmidt, R. C. & Richardson, M. J. Dynamics of interpersonal coordination. In Coordination: Neural, behavioral and social dynamics, Springer Berlin Heidelberg, 281–308 (2008).

  17. 17.

    Delaherche, E. et al. Interpersonal synchrony: A survey of evaluation methods across disciplines. IEEE T. Affect. Comput. 3(3), 349–365 (2012).

  18. 18.

    Chartrand, T. L. & Bargh, J. A. The chameleon effect: the perception–behavior link and social interaction. J. Pers. Soc. Psychol. 76(6), 893 (1999).

  19. 19.

    Hove, M. J. & Risen, J. L. It’s all in the timing: Interpersonal synchrony increases affiliation. Soc. Cog. 27(6), 949–960 (2009).

  20. 20.

    Raffard, S. et al. Social priming enhances interpersonal synchronization and feeling of connectedness towards schizophrenia patients. Sci. Rep. 5, 8156 (2015).

  21. 21.

    Varlet, M. et al. Impairments of social motor coordination in schizophrenia. PLoS One, 7(1), p.e29772 (2012).

  22. 22.

    Schmidt, R. C., Fitzpatrick, P., Caron, R. & Mergeche, J. Understanding social motor coordination. Hum. Mov. Sci. 30(5), 834–845 (2011).

  23. 23.

    Petersen, S., Houston, S., Qin, H., Tague, C. & Studley, J. The Utilization of Robotic Pets in Dementia Care. J. Alzheimers Dis. (Preprint), 1–6 (2016).

  24. 24.

    Jordan, K., King, M., Hellersteth, S., Wiren, A. & Mulligan, H. Feasibility of using a humanoid robot for enhancing attention and social skills in adolescents with autism spectrum disorder. Int. J. Rehabil. Res. 36(3), 221–227 (2013).

  25. 25.

    Scassellati, B. How social robots will help us to diagnose, treat, and understand autism. Int. J. Robot. Res. 552–563 (2007).

  26. 26.

    Billard, A., Robins, B., Dautenhahn, K. & Nadel, J. Building Robota, a Mini-Humanoid Robot for the Rehabilitation of Children with Autism. Assist. Technol. 19(1) (2006).

  27. 27.

    Boucenna, S. et al. Interactive technologies for autistic children: A review. Cog. Comp. 6, 722–740 (2014).

  28. 28.

    Bird, G., Leighton, J., Press, C. & Heyes, C. Intact automatic imitation of human and robot actions in autism spectrum disorders. Proc. R. Soc. B 274(1628), 3027–3031 (2007).

  29. 29.

    Lund, H. H., Pedersen, M. D. & Beck, R. Modular robotic tiles: experiments for children with autism. Artificial Life and Robotics. 13(2), 394–400 (2009).

  30. 30.

    Scassellati, B., Admoni, H. & Mataric, M. Robots for use in autism research. Annu. Rev. Biomed. Eng. 14, 275–294 (2012).

  31. 31.

    Francois, D., Powell, S. & Dautenhahn, K. A long-term study of children with autism playing with a robotic pet: Taking inspirations from non-directive play therapy to encourage children’s proactivity and initiative-taking. Interact. Stud. 10(3), 324–373 (2009).

  32. 32.

    Wainer, J., Robins, B., Amirabdollahian, F. & Dautenhahn, K. Using the humanoid robot KASPAR to autonomously play triadic games and facilitate collaborative play among children with autism. IEEE Trans. Auton. Ment. Dev. 6(3), 183–199 (2014).

  33. 33.

    Narita, S. et al. A pet‐type robot Artificial Intelligence Robot‐assisted therapy for a patient with schizophrenia. Asia Pac. Psychiat. 8(4), 312–3 (2016).

  34. 34.

    Richardson, M. J., Marsh, K. L., Isenhower, R. W., Goodman, J. R. & Schmidt, R. C. Rocking together: Dynamics of intentional and unintentional interpersonal coordination. Hum. Mov. Sci. 26(6), 867–891 (2007).

  35. 35.

    Cho, R. Y., Konecky, R. O. & Carter, C. S. Impairments in frontal cortical γ synchrony and cognitive control in schizophrenia. Proc. Natl. Acad. Sci. 103(52), 19878–19883 (2006).

  36. 36.

    Suslow, T., Droste, T., Roestel, C. & Arolt, V. Automatic processing of facial emotion in schizophrenia with and without affective negative symptoms. Cog. Neuropsychiatry 10(1), 35–56 (2005).

  37. 37.

    Suslow, T., Roestel, C., Droste, T. & Arolt, V. Automatic processing of verbal emotion stimuli in schizophrenia. Psychiat. Res. 120(2), 131–144 (2003).

  38. 38.

    Rauch, A. V. et al. Increased amygdala activation during automatic processing of facial emotion in schizophrenia. Psychiat. Res. Neuroim. 182(3), 200–206 (2010).

  39. 39.

    van’t Wout, M. et al. Exploring the nature of facial affect processing deficits in schizophrenia. Psychiat. Res. 150(3), 227–235 (2007).

  40. 40.

    Raffard, S. et al. Humanoid robots versus humans: How is emotional valence of facial expressions recognized by individuals with schizophrenia? An exploratory study. Schizophr. Res. 176(2), 506–513 (2016).

  41. 41.

    Bowie, C. R. & Harvey, P. D. Administration and interpretation of the Trail Making Test. Nat. Protoc. 1(5), 2277–2281 (2006).

  42. 42.

    Deak, G. O. The development of cognitive flexibility and language abilities. Adv. Child Dev. Behav. 31, 273–328 (2003).

  43. 43.

    Moore, A. & Malinowski, P. Meditation, mindfulness and cognitive flexibility. Conscious. Cog. 18(1), 176–186 (2009).

  44. 44.

    Dyck, M., Loughead, J., Gur, R. C., Schneider, F. & Mathiak, K. Hyperactivation balances sensory processing deficits during mood induction in schizophrenia. Soc. Cog. Affect. Neurosci. 9(2), 167–175 (2014).

  45. 45.

    Suslow, T. et al. Automatic amygdala response to facial expression in schizophrenia: initial hyperresponsivity followed by hyporesponsivity. BMC Neurosci. 14(1), 140 (2013).

  46. 46.

    Keller, P. E., Novembre, G. & Hove, M. J. Rhythm in joint action: psychological and neurophysiological mechanisms for real-time interpersonal coordination. Phil. Trans. R. Soc. B 369(1658), 20130394 (2014).

  47. 47.

    Dowd, E. C., Frank, M. J., Collins, A., Gold, J. M. & Barch, D. M. Probabilistic reinforcement learning in patients with schizophrenia: Relationships to anhedonia and avolition. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 1(5), 460–473 (2016).

  48. 48.

    Farreny, A. et al. Study of positive and negative feedback sensitivity in psychosis using the Wisconsin Card Sorting Test. Compr. Psychiatry 68, 119–28 (2016).

  49. 49.

    Kohler, C. G., Walker, J. B., Martin, E. A., Healey, K. M. & Moberg, P. J. Facial emotion perception in schizophrenia: a meta-analytic review. Schizophr Bull. 36(5), 1009–1019 (2010).

  50. 50.

    Giersch, A., van Assche, M., Huron, C. & Luck, D. Visuo-perceptual organization and working memory in patients with schizophrenia. Neuropsychologia 49(3), 435–443 (2011).

  51. 51.

    Sakamoto, D., Kanda, T., Ono, T., Ishiguro, H. & Hagita, N. March. Android as a telecommunication medium with a human-like presence. In 2nd ACM/IEEE Int. Conf. on HRI, 193–200 (2007).

  52. 52.

    Krebs, M. O., Gut-Fayand, A., Bourdel, M. C., Dischamp, J. & Olié, J. P. Validation and factorial structure of a standardized neurological examination assessing neurological soft signs in schizophrenia. Schizophr. Res. 45(3), 245–260 (2000).

  53. 53.

    Gupta, S. et al. Neurological soft signs in neuroleptic-naive and neuroleptic-treated schizophrenic patients and in normal comparison subjects. Am. J. Psychiatry 152(2), 191–196 (1995).

  54. 54.

    Walther, S. & Strik, W. Motor symptoms and schizophrenia. Neuropsychobiology. 66(2), 77–92 (2012).

  55. 55.

    D’Agati, E., Casarelli, L., Pitzianti, M. & Pasini, A. Neuroleptic treatments and overflow movements in schizophrenia: Are they independent? Psychiatry res. 200(2), 970–976 (2012).

  56. 56.

    Kay, S. R., Fiszbein, A. & Opfer, L. A. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr. Bull. 13(2), 261 (1987).

  57. 57.

    Sanchez-Cubillo, I. et al. Construct validity of the Trail Making Test: role of task-switching, working memory, inhibition/interference control, and visuomotor abilities. J. Int. Neuropsychol. Soc. 15(3), 438 (2009).

  58. 58.

    Metta, G. et al. The iCub humanoid robot: An open-systems platform for research in cognitive development. Neural Netw. 23(8), 1125–1134 (2010).

  59. 59.

    Noy, L., Dekel, E. & Alon, U. The mirror game as a paradigm for studying the dynamics of two people improvising motion together. Proc. Natl Acad. Sci. USA 108(52), 20947–20952 (2011).

  60. 60.

    Zhai, C., Alderisio, F., Słowiński, P., Tsaneva-Atanasova, K. & di Bernardo, M. Design of a virtual player for joint improvisation with humans in the mirror game. PLoS ONE 11(4), e0154361 (2016).

  61. 61.

    Viviani, P. & Flash, T. Minimum-jerk, two-thirds power law, and isochrony: converging approaches to movement planning. J. Exp. Psychol. Hum. Percept. Perform. 21(1), 32 (1995).

Download references

Acknowledgements

This experiment was supported by a grant from the European Union’s Seventh Framework Program (FP7 ICT 2011 Call 9) under grant agreement n FP7-ICT-600610 ALTEREGO. We express our gratitude to the patients and their families for participating in this research and thereby making it possible. We thank Ashwini Shukla for his earlier contributions to the preparation of iCub for the mirror game, and Eric Sauser for his implementation of the kinematic solver for the robot.

Author information

Author notes

  1. Laura Cohen and Mahdi Khoramshahi contributed equally to this work.

Affiliations

  1. Learning Algorithms and Systems Laboratory, School of Engineering, EPFL, Lausanne, Switzerland

    • Laura Cohen
    • , Mahdi Khoramshahi
    •  & Aude Billard
  2. EuroMov, Montpellier University, Montpellier, France

    • Robin N. Salesse
    • , Ludovic Marin
    •  & Benoit G. Bardy
  3. University Department of Adult Psychiatry, CHU, Montpellier, France

    • Catherine Bortolon
    • , Delphine Capdevielle
    •  & Stéphane Raffard
  4. Department of Mathematics, College of Engineering, Mathematics and Physical Sciences, University of Exeter, Exeter, United Kingdom

    • Piotr Słowiński
    •  & Krasimira Tsaneva-Atanasova
  5. Department of Engineering Mathematics, University of Bristol, Bristol, United Kingdom

    • Chao Zhai
    •  & Mario Di Bernardo
  6. Psychology Department, College of the Holy Cross, Worcester, MA, USA

    • Richard C. Schmidt
  7. Institut Universitaire de France, Paris, France

    • Benoit G. Bardy
  8. Laboratory Epsylon, EA 4556, University Montpellier 3 Paul Valery, Montpellier, France

    • Catherine Bortolon
    •  & Stéphane Raffard

Authors

  1. Search for Laura Cohen in:

  2. Search for Mahdi Khoramshahi in:

  3. Search for Robin N. Salesse in:

  4. Search for Catherine Bortolon in:

  5. Search for Piotr Słowiński in:

  6. Search for Chao Zhai in:

  7. Search for Krasimira Tsaneva-Atanasova in:

  8. Search for Mario Di Bernardo in:

  9. Search for Delphine Capdevielle in:

  10. Search for Ludovic Marin in:

  11. Search for Richard C. Schmidt in:

  12. Search for Benoit G. Bardy in:

  13. Search for Aude Billard in:

  14. Search for Stéphane Raffard in:

Contributions

S.R., R.N.S, M.K., L.C., A.B., and B.G.B. contributed to the study design. C.B., D.C., and S.R. recruited and assessed the patients. C.Z., P.S., K.T., and M.dB. developed the cognitive architecture. M.K., L.C., and A.B. developed the robotic aspect, performed the statistical analysis and wrote the first draft. M.K., L.C., A.B., and S.R. prepared the final manuscript. All authors reviewed the manuscript.

Competing Interests

The authors declare that they have no competing interests.

Corresponding author

Correspondence to Stéphane Raffard.

Electronic supplementary material

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Creative Commons BY

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.