Introduction

It is now well admitted that individuals have a tendency to synchronize their behaviors, which constitutes a key form of social exchange throughout the lifetime. Prototypical examples are found in everyday life when people are walking side by side and synchronize unconsciously their steps as well as their attitudes while talking to each other1. A substantial number of studies have shown that the foundation for this laid very early in development, as newborn infants spontaneously imitate a range of simple behaviors of an adult model2.

In addition to mimicry, which is typically defined as the spontaneous, immediate (at the split of a second) imitation of gestures, postures, mannerisms and the dynamics of movement of another person3,4,5, behavioral synchronization denotes the mutual alignment of interaction partners’ behavior on a larger time-scale. Both processes have been found to be critical in human-human interactions since they are suggested to be tied to liking, rapport and empathy6. For example, there is a large amount of evidence that being mimicked promotes rapport7, trust8, altruistic behavior and liking between interacting partners9 and, in turn increases behavioral synchronization and movements in the mimicked partner via a variety of psychological and neural mechanisms10.

In comparison to other types of deliberate, goal-directed, voluntary imitation such as social learning11 or imitation learning12, one crucial aspect of behavioral synchronization is the fact that it occurs without awareness. Indeed, behavioral synchronization is nonconscious, unintentional, effortless, and has the potency to increase prosocial behavior mostly when interacting people are not aware that their behavior is being mimicked. By contrast, overt detection of being mimicked can have deleterious effects on affiliation13 or liking14. In addition, as mimicry of other people’s behavior during social interaction is unconscious, it is thus fundamentally considered as a social behavior modulated by social cues such as eye gaze or social status15.

In the field of psychopathology and neurodevelopmental disorders, mimicry has been particularly studied in autism spectrum disorders. A recent meta-analysis16 indicated that individuals with autism exhibit significant impairments in behavioral synchronization skills on a variety of tasks including body movements, vocalizations, or facial expressions. The same results were found regarding mimicry17,18. Contrary to autism, behavioral synchronization and particularly its relationships with interpersonal functioning has been poorly studied in patients diagnosed with schizophrenia. As a whole, prior studies have repeatedly shown impaired voluntary imitation of face and complex hand gesture movements in schizophrenia patients in comparison to healthy partners19,20,21, especially when the production of a voluntary imitation depends upon working memory22,23. The few studies that focused on behavioral synchronization have produced mixed results. Whereas some studies showed impaired behavioral synchronization of emotional stimuli24,25, a recent study by Simonsen et al.26 failed to find any evidence of such impairment when controlling emotional and cognitive confounding variables in their group of patients diagnosed with schizophrenia.

Importantly, previous research in schizophrenia has mainly targeted nonsocial voluntary imitation or synchronization behavior of either complex or simple movements21 and employed experimental laboratory paradigms measuring specific body movements such as hands22, mouth21, or emotions expression25,27,28. In contrast, mimicry is traditionally studied during naturalistic paradigms that measure the frequency of mimicked movements during social interactions between a participant and a confederate (e.g. conversation).

Recent studies have found that automatic imitation10 or mimicry29 are independent of measures of empathy and social cognition. In line with these findings, reflective processes underlying social cognition (requiring effortful controlled processes) including metalizing, emotion regulation and perception of social stimuli (faces and voices), have been found to be systematically impaired in schizophrenia, while more reflexive aspects of social cognition (i.e. more automatic processes), such as motor resonance or emotion contagion, may be intact30. Therefore, we believe that mimicry (which is a reflexive aspect of social cognition) could constitute a promising tool for social rehabilitation in schizophrenia.

Recent advances in collaborative virtual environments (CVEs) have shown the potential of virtual reality for enhancing ecological validity while maintaining experimental control in social neuroscience research31. Indeed, contrary to a real human partner, virtual agents can be programmed to automatically mimic the participant’s movements after an accurate and fully controlled time delay. Importantly, a large amount of research has shown that individuals react towards virtual agents similarly than towards real people32,33. Accordingly, several studies have used CVEs to explore the positive effects of mimicry on interaction, and provided consistent results showing that mimicking virtual agents are perceived as more persuasive34, likable35, and trustworthy36.

For the reasons mentioned above, this study aimed to investigate 1) whether synchronization of behavior, i.e., the spontaneous initiation of gestures that are congruent with those of an interaction partner, was impaired in individuals with schizophrenia compared with healthy participants; 2) whether manipulating mimicry by a virtual agent would impact (a) synchronization behavior; (b) and the feeling of rapport associated with the virtual agent in two groups of nonclinical individuals and schizophrenia patients.

Results

Synchronization behavior

Results revealed a significant effect of mimicry (F(1,36) = 4.69, p = 0.037, η2p = 0.12). This result displayed on Fig. 1 indicates that participants exhibited a higher synchronization behavior of the virtual agent forearm movements when they were in the Mimicry condition (21.4%) rather than in the No-mimicry condition (16.2%). The analysis failed to reveal any group difference (Tables 1 and 2).

Figure 1
figure 1

Mean percentage of the trials where a forearm imitation motion between virtual agent and participant is detected. Left columns refer to the no-mimicry conditions and right columns refer to the mimicry condition. Indicative information is provided by the colors where white columns refer to the schizophrenia group whereas black columns correspond to the control group. Error bars correspond to the standard deviation around the mean.

Full size image
Table 1 Descriptive statistics of the Mimicry, Amount and Rapport variables.
Full size table
Table 2 ANOVAs for the Mimicry, Range and Rapport variables. Group, Mimicry and Group × Mimicry effects are reported.
Full size table

Amount of non-verbal behavior

The results of the ANOVA Groups (2) × Mimicry (2) performed on the range of motion of the left forearm revealed a significant Mimicry effect (F(1,36) = 7.02, p = 0.012, η2p = 0.16) indicating that both groups exhibited larger arms movement when facing an mimicking virtual agent rather than an virtual agent not mimicking them (respectively 0.083 vs. 0.058 in arbitrary units). In other words, mimicry increased the amount of nonverbal behavior (i.e. movements) in both groups (Tables 1 and 2).

Rapport

Regarding the questions evaluating how much participants felt comfortable during the interaction, results demonstrated a statistically significant effect of mimicry for one of the three questions (F(1,36) = 6.52, p = 0.015, η2p = 0.15) indicating that both groups felt more comfortable with a mimicking virtual agent rather than a virtual agent not mimicking them (respectively +1.55 vs. +1.08). For the question associated with willingness to interact with the virtual agent in the future, no main effect was found statistically significant (Tables 1 and 2). A significant Group × Mimicry interaction was found significant for likeness (F(1,36) = 4.92, p = 0.033, η2p = 0.12), however the post-hoc decomposition of this interaction failed to reveal any significant difference between the conditions. This is due to the between/within design of this interaction and therefore prevents any interpretation of such result. We also collapsed the four items to obtain a rapport total score, that is, rapport, attractiveness, likeness, and willingness to interact. Rapport total score also failed to found statistical differences (Group effect F(1,36) = 0.14, p = 0.71, η2p = 0.004; Mimicry effect F(1,36) = 3.74, p = 0.06, η2p = 0.094; Group × Mimicry interaction F(1,36) = 0.48, p = 0.49, η2p = 0.013), however, we note here a statistical tendency (p = 0.06) with a small to medium effect size (η2p = 0.094) for the mimicry condition.

Mimicry, rapport and psychotic symptomatology

Regarding psychotic dimensions, range of motion in the Mimicry condition negatively correlated with the positive dimension of the five-factor model of the PANSS (r = −0.57, p = 0.034) and the perceived attractiveness (Q2) in the Mimicry condition positively correlates with the CPZ equivalent dose (r = 0.58, p = 0.029). No other significant correlations were found between Mimicry, rapport, psychotic symptomatology, and other clinical and demographic variables. After applying Bonferroni corrections for multiple comparisons with an alpha value of 0.05, no significant correlations remained statistically significant (Table 4).

Table 3 Socio-demographic description of the schizophrenia group and the healthy control group.
Full size table
Table 4 Correlation between motion/rapport variables and sociodemographic/clinical variables.
Full size table

Discussion

Our study investigated synchronization behavior during an interaction with a virtual agent in individuals with schizophrenia.

We programmed a virtual agent to mimic participants’ head and torso movements with a delay varying randomly between 0.5 s to 4 s, compared to a non-mimicry condition. We measured synchronization behavior of forearm movements, responses to being mimicked (i.e. rapport) and amount of non-verbal behaviors, between the two conditions.

First, our findings revealed that schizophrenia patients displayed intact synchronization behavior during interaction with virtual agents compared to the healthy controls group. Indeed, the two groups of participants exhibited similar and a higher synchronization behavior of the virtual agent forearm movements when they were in the mimicry condition rather than in the non-mimicry condition. In addition, both groups exhibited larger arms movement when facing a mimicking virtual agent rather than a virtual agent not mimicking them. On the one hand, this result appears in line with the recent study by Simonsen et al.26 who showed that their schizophrenia group displayed intact automatic imitation compared to matched healthy individuals. On the other hand our study contrasts with the majority of previous findings showing impaired imitation abilities in schizophrenia. First, and as stated by Simonsen et al.26, previous findings assessed voluntary imitation21,22,23 or automatic imitation of emotional stimuli24,25 without controlling for the cognitive, motoric, and/or emotional deficits associated with schizophrenia. Thus it makes difficult to conclude whether imitation deficits were primary or secondary to aforementioned other deficits. Second, our experiment focused on the behavioral synchronization with a virtual agent on a larger time-scale, whereas Simonen et al.26 examined automatic imitation of finger movements presented via short video sequences. Consequently, our findings indicate that the fundamental tendency to synchronize with the behavior of others in social interactions is preserved in individuals with schizophrenia. Additionally, our finding are in line with previous studies in nonclinical individuals34,35 showing that virtual agents have the potential to induce behavioral synchronization and mimicry during interactive paradigms and extends this result to individuals with schizophrenia disorder. Our results also fit well with the results of Riehle & Lincoln37. In their study the authors assessed the amount of smiling and mimicry of smiling via electromyography during a dyadic interaction in participants with schizophrenia and healthy controls. Similarly with our results, they did not find evidence of reduced mimicry of smiling in a face-to-face interaction in the patients group. Their results are particularly relevant with regards to our study. As contrary to our experimental task consisting of an interaction with a virtual agent, they used an ecological task (brief face-to-face conversation) during a human-human interaction. Taken together, these findings suggest that behavioral synchronization during social interaction is preserved in schizophrenia patients”.

Secondly and as hypothesized, our results indicated that behavioral synchronization was associated with better rapport in our two groups. Indeed, both groups felt more comfortable with a virtual agent that mimicked their behavior than the one that did not. In addition and regarding rapport, we found a marginally significant effect in the mimicry condition when we collapsed the four measures of rapport used in our study. This is consistent with previous research using virtual agents and mimicry that showed that mimicking virtual agents are perceived as more likable35 and trustworthy36. In a therapeutic context it may be translated in patients being more willing and feeling more comfortable to share personal information. Previous studies have shown that individuals evaluate therapeutic alliance with a virtual agent in a positive manner and are willing to share their personal information38,39. Associating mimicry and the use of virtual agents controlled by artificial intelligence or algorithm or virtual agents controlled by therapist/clinician may be a promising option to increase access and engagement to psychological services40.

It must be however noted that we did not find specific effects of mimicry on any of our other dependent measures (attractiveness, likeness, willingness to interact) suggesting that these measures of rapport and connectedness could not be adequate in the context of virtual environment. A validated scale of rapport in collaborative virtual environments could be particularly useful for future studies. It is worth mentioning that ratings of liking, and self-other overlap also show inconsistent effects of mimicry in both virtual environment and traditional research settings, leading some researchers to suggest that the positive social effect of being mimicked is perhaps either not always consciously perceived or more fragile than generally reported in the scientific literature41. Finally, it should be noted that the key findings of this study showed remarkable effect sizes, suggesting that they may be relevant beyond mere academic interest.

There is evidence that people imitate the behaviors of those with whom they have ongoing relationships: parents42 and teachers43. There is also evidence that higher coordination of patient’s and therapist’s movement in psychotherapy dyads is associated with better outcome44. However, mimicry occurs spontaneously and unintentionally and it is very difficult to voluntary imitate other’s body posture, hand gestures or emotions during an interaction without being aware that the behavior is being copied45. As a consequence, using virtual agents of well-known person (i.e. referent from a clinical team) could constitute a promising augmentation to virtual reality social skills training intervention or job training interview, via the use of mimicking virtual agents. In addition, series of previous studies have shown that social cues, such as eye-contact17, and emotional facial expressions46, have the potential to modulate synchronization behavior in healthy participants. Further studies are needed to explore if such additional social cues implemented in virtual partners could increase synchronization behavior and by consequences act as a “social” cue in a mental disorder with severe social impairments and rejection.

Conclusion

First, our findings suggest that schizophrenia cannot be considered anymore as a disorder of imitation, particularly as regards behavioral synchronization processes in social interaction contexts. Secondly, our study demonstrated how virtual environments could be used to address important questions in the context of social disorders such as schizophrenia. From a clinical perspective, we believe that our results are very encouraging and have implications for VR training programs. Virtual reality environments have been increasingly used in the context of mental health treatment and within schizophrenia research during the last five years. Virtual reality, as a treatment approach, has demonstrated promise in treating psychotic symptoms particularly hallucination47, but also as an adjunct to social-skills training48 or job interview training49. As such, future virtual reality interventions could use mimicry as a real-time feedback signal to create more meaningful human-virtual agent interactions leading to increase patients’ engagement in VR-based treatment.

Methods

Participants

Nineteen outpatients with schizophrenia and nineteen healthy controls participated in the study. Two additional schizophrenia patients and two healthy participants did not complete the whole experiment. We recruited patients meeting DSM IV-TR criteria for schizophrenia in Montpellier University Hospital. Diagnosis of schizophrenia was established via the Structured Clinical Interview for DSM-IV (SCID-I) by the treating psychiatrist. None were in the acute phase of psychosis. Inclusion criteria were being between 18 and 55 years of age, having a diagnosis of schizophrenia and being able to understand, talk and read French. Exclusion criteria were substance dependency other than cannabis or tobacco, substance abuse other than cannabis or alcohol, and co-morbid neurological disorder. Eighteen patients were medicated at clinically determined dosages. One patient was not taking medication. The mean dose of antipsychotic medication was equivalent to 540 mg/day of chlorpromazine (SD = 334.38)50. Controls were recruited from the general population with no personal lifetime history of any psychosis or affective disorders diagnosis (MINI)51. Controls with a family member with bipolar or schizophrenia disorders were excluded. All participants were native French speakers with a minimal reading level validated using the National Adult Reading Test (f-NART)52. All participants were right-handed as assessed by informal verbal inquiry. All participants provided written informed consent, prior to the experiment approved by the National Ethics Committee (CPP Sud-Méditerranée-III, Nîmes, France, #2009.07.03ter and ID-RCB-2009-A00513-54) and conforming to the Declaration of Helsinki. Accordingly with identifying information policies, written informed consent for publication of identifying information/images was obtained.

Clinical variables

In the schizophrenia sample, we administered the Positive and Negative Syndrome Scale using the Structured Clinical Interview for the PANSS (SCI-PANSS). For this study, five of the analytically derived PANSS factor component scores53 were taken into account: Positive, Negative, Disorganization, Excitement and Emotional Distress (Table 3).

Rapport

No validated measures of rapport such as the Inclusion of the Other in the Self Scale (IOS)54 or the 6-item Willingness to Interact Questionnaire (WIQ)55 have been developed for CVEs. Thus, rapport during the interaction with the virtual agent was evaluated using self-reported measures based on previous studies using CVEs56 and on literature on interpersonal communication. The questionnaire was composed by three questions assessing positive rapport toward the virtual agent (i.e. “I felt comfortable while interacting with this virtual agent”, “I think this virtual agent is attractive”, “I like this agent”) and one question assessing willingness for future interactions adapted from the WIQ (“I want to interact with this virtual agent again in the future”). The questions were answered using a scale from −3 to +3, −3 being equals to “I do not agree al all”, 0 “more or less”, and +3 “I totally agree”. A rapport total score was also provided composed by the sum of the four questions.

Procedure

The experiment was completed in two visits. Visit 1 involved completing informed consent, clinical interviews and questionnaires. Visit 2 occurred approximately 2 days later. It consisted on a virtual mimicry experiment. We designed an interactive protocol comprising two conditions, a mimicry and a no-mimicry condition, where participants stood up in front a photo-realistic 3D virtual agent displayed on a large TV screen (Fig. 2). During the interaction and the two experimental conditions, the virtual agent provided information about some healthy issues: physical activity levels, quality of diet and sleep and how to quit smoking. Texts were vocally delivered by the virtual agent using a computer-generated voice and were synchronized with the virtual agent motion. The motion of the virtual agent was pre-recorded by an actor whose upper-body movements (head, torso, arms and forearms) were tracked with 3D Inertial Measurement Units (IMU). In the mimicry condition, the virtual agent mimics the participants’ head and torso movements with a delay varying randomly between 0.5 s and 4 s, while in the no-mimicry condition, head and torso movements were pre-recorded. Every 5 s to 15 s, the virtual agent was scratching his arm to induce behavioral mimicry by the participant. During the interaction, participants had their movements recorded by six IMUs attached to their arms, forearms, torso and head (Fig. 3). One message was delivered per condition. Each condition lasted 300 s. Conditions were fully counterbalanced across participants. We measured in each condition the range of motion performed by the arms of the participant and the degree of behavioral synchronization between the virtual agent and the participant using the Forearms imitation motion.

Figure 2
figure 2

Picture of the interactive apparatus where participants stood up in front a photo-realistic 3D virtual agent displayed on a large TV screen. Sensors (IMUs) used to capture the motion can be seen on the head, torso, right arm and right forearm of the participant.

Full size image
Figure 3
figure 3

Schematic illustration of the Imitative Virtual Reality Pipeline used in this experiment. 1. Participant motion is tracked using 6 IMUs (red cuboids); 2. Data are wireless transmitted in real-time to the computer; 3. Data are stored and also used for further computation; 4. Data from the different sensors are fused as a function of the mimicry condition, in the no-mimicry condition, only prerecorded motions (blue cuboids) are sent to the rendering process whereas in the mimicry condition, head and torso motions of the participant (red cuboids) are fused with other prerecorded sensors (blue cuboids) and send to the rendering; 5. Virtual environment is generated in realtime using Odysseus Studio and virtual agent’s voice is computer generated using MacVoiceOver. Voice was send to the participant through headphones.

Full size image

Data Analysis

We measured the degree of synchronization behavior between the virtual agent and the participant using the Forearms imitation motion. 2 sets of 6 sensors (inertial measurement units) were used in this experiment. The first set corresponded to the virtual agent motion and the second set to the participant motion. Each sensor data called quaternion was recorded at a sampling frequency of 50 Hz (±1 Hz). Quaternions are simpler and more efficient representations of a rotation in a 3D space than Euler angles. Raw quaternions were normalized to ensure the robustness of further computations57. Normalized quaternions were then interpolated on a constant 50 Hz sampling rate using the Spherical Linear Quaternion Interpolation SLERP method58. We computed the amount of rotation between two samples using the natural metric for the rotation group (induced by the shortest path between its two elements); specifically we used its functional form based on the inner product of unit quaternions, which is most computationally efficient59. The amount of rotation was therefore unwrapped to avoid 2pi jumps in order to allow the next step of the analysis. Cross wavelet transform between the amount of rotation of the forearms of the participant and the virtual agent were computed59 giving rise to 3 time-frequency representations (Left forearm of the virtual agent versus Left forearm of the participant, Left forearm of the virtual agent versus Right forearm of the participant, and Right forearm of the virtual agent versus Right forearm of the participant). Cross wavelet transforms indicates the amount of synchronization between two time series for each sample in time and each period interval. Cross wavelet transforms were computed only in the range of period between 0.5 s to 8 s, corresponding to the range of behavioral mimicry usually described in the literature60. Significant areas (>0.95) of each of these 3 time-frequency representations were extracted and superimposed. Significant areas can be interpreted as synchronization levels significantly higher than random noise. The forearm imitation motion was finally calculated as a percentage of the trial where significant relationships between a movement of the virtual agent forearms and a movement of the participant forearms was detected.

Considering the statistical analysis, group differences and interactions were analysed with ANOVA tests. Post-hoc tests were used when the nature of the effects had to be specified. Size effects on our repeated ANOVA design have been reported using the partial eta squared61 ηp2, and interpreted following Cohen62 where 0.02 corresponds to a small effect, 0.13 to a medium effect and 0.26 to a large effect. When necessary, alpha value of significance was corrected using the Bonferroni procedure.