Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback

Spinal cord injuries disrupt bidirectional communication between the patient’s brain and body. Here, we demonstrate a new approach for reproducing lower limb somatosensory feedback in paraplegics by remapping missing leg/foot tactile sensations onto the skin of patients’ forearms. A portable haptic display was tested in eight patients in a setup where the lower limbs were simulated using immersive virtual reality (VR). For six out of eight patients, the haptic display induced the realistic illusion of walking on three different types of floor surfaces: beach sand, a paved street or grass. Additionally, patients experienced the movements of the virtual legs during the swing phase or the sensation of the foot rolling on the floor while walking. Relying solely on this tactile feedback, patients reported the position of the avatar leg during virtual walking. Crossmodal interference between vision of the virtual legs and tactile feedback revealed that patients assimilated the virtual lower limbs as if they were their own legs. We propose that the addition of tactile feedback to neuroprosthetic devices is essential to restore a full lower limb perceptual experience in spinal cord injury (SCI) patients, and will ultimately, lead to a higher rate of prosthetic acceptance/use and a better level of motor proficiency.


Apparent Movement
The apparent movement of a tactile stimulus is perceived when a number of stimulators are aligned on the user's skin and then activated sequentially. In our setup, the stimulators (i.e. the ERM vibrators of the long sleeves of the tactile shirt) were aligned and spaced on the user's forearm skin surface and activated sequentially according to their position.
Experiments started by optimizing the parameters of tactile feedback (in terms of duration of stimulation per vibrator and time interval between stimulation of two adjacent vibrators) to induce the apparent movement illusion in all our patients.
We tested which combinations of Duration of Stimulation (DoS) and inter-Stimuli Onset Interval (ISOI) would induce the most salient apparent movement sensation by varying the overall duration of skin stimulation in all eight patients.
While wearing the tactile shirt, subjects were presented with apparent movement durations (DoApM) ranging from 400 ms to 1800 ms. Using three vibrators, the DoApM was equal to 2xISOI + DoS. Following this linear relation, all possible combinations of ISOI and DoS were derived for each presented DoApM. During each trial, the subjects were able to modify the stimulation by pressing a key that incremented the value of DoS by 50 ms (or 25 ms for ISOI).
The subjects reported orally which stimulation yielded the strongest impression of continuous movement. DoApM values of 400, 600, 1000, 1400 and 1800 ms were tested for a sequence of vibration triggered from both Proximal to Distal (PtD) and Distal to Proximal (DtP).
For all eight subjects, we computed a linear regression on the DoS values and observed a linear relation between DoApM (going from 400 to 1800 [ms]) and DoS (or ISOI) in terms of inducing the apparent movement sensation (coefficient of determination R 2 > 0.95 for all subjects). Two analyses of variance (ANOVA) were performed with the slope and the offset of the linear fit as dependent variables for each ANOVA to analyze the effect of direction of stimulation (PtD versus DtP). The independent variable was the direction of the apparent movement (2 levels: PtD and DtP). No effect was found for either the slope or the offset (both p > 0.05). Means and standard errors of DoS over all patients are shown for the different DoApM ( Fig. S6A-B). The low standard errors suggest that the mean can be used as a good estimator of the DoS (therefore ISOI) for any stimulation duration, for all patients. A linear regression was then computed on the mean DoS (R 2 > 0.95). Thus, we found that the same linear regression can be used as an estimator of DoS (and ISOI) to generate realistic apparent movements for any duration of stimulation in the studied range for all subjects.

Principal parameters to define the floor texture
In the simulation of floor texture experiment, we searched for the parameters upon which patients relied the most to choose each ground texture.
For this we ran a k nearest neighbor classification algorithm considering each one of the four tactile parameters (amplitude of proximal, middle and distal vibrator and stimulation timing). We considered all trials over all sessions at once (N = 120 x 15 = 1800). Data was bootstrapped 100 times in training and testing set (ratio 5 to 1). Mean per parameter and session is shown in Figure   S7A.

Responses similarity probability
We calculated the probability of serving similarities among patients' responses clusters. Considering the following partition of the PV/ST referential: top left and right area, lower left and right area and the central area, there are 5 3 = 125 possible configurations for placement of the centroid of the three textures.
Considering all configurations had the same probability p (1/125) to be chosen, the probability of having at least m times a configuration over n trials can be calculated using the probability mass function of the binomial distribution B(n,p): In the experiment we observed that in 9 out of 15 sessions, subjects chose the same configuration. The probability of this sequence is thus:

Parameter separability for floor texture simulation test
To analyze how patients chose the factors that defined a given virtual ground surface in the presence or absence of a virtual ground, we calculated for each subject the average difference of chosen factors between the repetitions of trials for each condition. More specifically, we derived the vector Tij representing the distribution of Euclidian distances between Ti and Tj, where Ti,j are vectors containing the 40 sets of four tactile parameters obtained in the 40 trials where surfaces ti,j were presented. For example, the distribution T 12 of distances between the ground types t 1 and t 2 is given by the Euclidian distance of the four factors between all combinations of trials m of T 1 with surface type t 1 and all trials n of T 2 with surface type t 2 . Notice that the maximum distance is given by the diagonal of the 4D hypercube with edges of size 10 (there are 10 levels per factor), thus √4 × 10 = 20. The number of repetitions per floor type was 40, so that any vector T ij had 40 × 40 = 1600 elements. We also calculated the distribution of distances within a surface type, i.e. Tii or Tjj. T 11 is the distribution of distances between all combinations of two trials , , ≠ for surface t 1 . T 11 has 1600 − 40 = 1560 elements. To analyze separability between surface t 1 and t 2 we ran a multiple comparison test between T 12 , T 11 and T 22. We also checked if the mean distance between surfaces 1 and 2 was larger than the mean distance between elements in surface 1 and those of surface 2.

Tactile Shirt hardware
The haptic display was based on a latching system featuring circular The tactile shirt provided tactile/proprioceptive feedback to patients both during training in virtual reality and training with a mechanical prosthetic device.
In the first case, the walking phase was generated by the virtual reality controller itself. Communication with the tactile shirt was performed through RS232 serial communication. In the second case, force and distance sensors where interfaced with the tactile shirt to detect walking phases (stance and swing) of the mechanical prosthetic device. Corresponding tactile feedback was displayed through the tactile shirt.   Their arms were resting on a table and a veil placed over the arms was used to eliminate visual cues. Only the proximal and the distal vibrators on both arms were activated but the patients were told that any of the six vibrators could be activated. Prior to the experiment all the vibrators were activated sequentially to check whether the patients were able to detect each vibrator and to familiarize them with the numbering of the vibrators (1 to 6). During the test, the vibrators were activated randomly for 50 ms, each stimulus repeated at least three times.
The four bars in the stacked bar plots correspond to the four activated vibrators, divided into two groups for the left and right arm. The stacks are the percentage of answers for each perceived vibration: proximal, middle and distal. The percentage of middle vibrators corresponds to the percentage of error in localizing the activated vibrator as the middle vibrator was never activated.
Patients were able to localize the vibration successfully with percentages of 88% and 87% for proximal and distal vibrator localization. There were no discrimination errors between distal and proximal vibrators. Subject's answer to Q2 (floor of type X I saw in the head mounted display was visually realistic) vs. rating of floor realism (only considering the realism of the virtual environment). No correlation was found between the responses to the two questions. (C) Principal tactile parameters considered by patients to discriminate the different floor texture is calculated using knn classifier on the 120 repetitions during exploratory phase (Chance level = 0.33). Classifier accuracy is given for all 15 sessions (row) and all parameters (columns).
Parameters PV and ST are found to be best ones for all patients. 13 randomly sized and placed cubes were presented in the head mounted display. Patients had to move their head and find the cubes and describe their relative position to each other. All patients successfully passed the test. The second test evaluated patients' susceptibility to motion sickness. They had to use the keyboard and move their head to respectively translate and rotate the view in a simulated house (simulation of the Tuscany house https://share.oculus.com/app/oculus-tuscany-demo).