Introduction

Spinal cord injury (SCI)-related complete tetraplegia and paraplegia are some of the most severe and disabling conditions within human pathology. For persons with complete tetraplegia, owing to the far greater limitations on daily activity1 and the almost complete dependence on caretakers, the reduction in post-SCI quality of life (QOL) is considerably higher.2 The possible availability of BCI and related mechatronic/robotic devices for daily use, subject to engineering improvements and extensive clinical testing, has the potential to improve motor complete tetraplegics’ QOL, mainly in terms of autonomy and self-esteem, as a consequence of regaining even a limited capacity to voluntarily control some common actions, which help fulfill basic needs, within activities of daily living (ADL). It is still an open research question as to which of the various BCI designs tested by now has the more advantageous mix of safety, cost and performance. The current study addresses the potential impact/usefulness of noninvasive EEG–BCI for the accomplishment of an ADL such as bringing to the mouth a glass in order to drink.

A reliable decoding method of the cerebral motor commands has been achieved in nonhuman primates using microelectrode implants, initially involving cortical microwires introduced during single recording sessions:3 using visual feedback, nonhuman primates could learn how to control a robotic arm without attempting to move their own.4 In nonhuman primates, correlates and predictors of hand movement were found and 1D, 2D and 3D cursor control was achieved. Further, as recently showed in macaques, by associating multielectrode recordings with multijoint arm movement capture, spiking patterns of local primary motor cortex allow the expression of many multidimensional, ‘naturalistic’ movements in the entire upper limb, including the possibility to achieve high-dimensional reach and grasp actions;5 however, the control of reaching and grasping specific targets within defined ADL—such as bringing a glass toward the mouth in order to drink—has not yet been clearly replicated in humans. After having a BCI system called ‘BrainGate’ implanted in the primary motor cortex, a chronic tetraplegic patient succeeded in controlling a cursor with which a displayed ‘simulated e-mail’ could be opened, as well as operate a multi-jointed robotic arm device (only ‘rudimentary actions’), by cortical voluntary commands.6 Although the information transmission rate achievable by invasive BCI is of a higher order of magnitude than that of noninvasive BCI, the duration and success rate of invasive BCI use in humans is yet unclear because of small test population sizes and confounding effects of the surgery itself.7 Very recent studies on two tetraplegic patients implanted since several years report that intracortical neural interface systems based on microelectrode arrays can provide long-lasting, accurate control of a 2-D computer cursor.8, 9

With regard to noninvasive technologies, EEG is, in the authors’ opinion, the only realistically practical noninvasive BCI method at present. Alternative imaging modalities such as functional magnetic resonance imagery, magneto-encephalography and positron emission tomography are quite expensive,10 technically demanding and not portable in terms of electrical energy usage and (except for near-infrared spectroscopy) size. Therefore, these modalities are rather inadequate for BCI systems meant for daily functional assistance, especially in the home. Near-infrared spectroscopy is cheaper than the aforementioned alternatives and rather portable. Yet, as in the case of functional magnetic resonance imagery, which is also based on measurement of the blood oxygen level-dependent signal,11 the related hemodynamic response's (naturally) long delay makes it rather slow; however, the same very recent study points to the potential of near-infrared spectroscopy to moderately complement EEG information in hybrid BCI systems.12 Although robotic assistive devices also carry a significant cost in purchase and maintenance, the basic underlying technology is ever evolving, and it may be hoped that significant research, development and innovation efforts in service and ambient-assisted living robotics, already underway, will lead to an affordable, reliable and capable enough solution in the near future.13

The main restrictions to noninvasive, EEG–BCI, quotidian use are limited information transmission rate and difficulty of daily recording setup, which is mainly a consequence of traditional gel-based EEG montage. The latter requires experience, time and hair washing before and after use. To address this potential shortcoming, the Brain2Robot team from Fraunhofer Institute-FIRST, which formed part of the current collaboration, has developed a new recording technology, which, using only six dry electrodes placed on the scalp approximately above the motor cortex, in a matter of minutes matches sensory-motor rhythm (SMR) EEG–BCI performance in healthy subjects.14 In SMR, which is the motor-imagery-driven BCI paradigm chosen for this study, EEG event-related desynchronization in the alpha and beta frequency bands discriminate between single motor imaginations of the left and right hand, as well as foot and tongue movements.15 Other EEG BCI designs require selective attention to cued sensory stimuli (evoked potentials); they may offer slight advantages, but information transmission rates for all noninvasive BCI designs remain roughly of the order of one binary decision every 2–10 s,16 owing to high inter-trial variability and spatial mixing, whereby cortical activity sources as far as 5 cm apart may effectively superimpose onto scalp potentials14, 17 and confound pattern-recognition algorithms. SMR was chosen as we considered it important that a QOL-rewarding BCI system for motor action in tetraplegics focus on motor imagination/intention rather than on visual or auditory attention, as the main basis of control. How well tetraplegics rate in terms of ability to use SMR BCI compared with healthy subjects (commonly studied in EEG–BCI research) is an open question the study aims to address, as it is central to overall feasibility of daily EEG–BCI use. About 20% of healthy subjects achieve accuracy rates of more than 80%18 for SMR–BCI, whereas performance shows relatively minor improvements with longer-term practice;19 evoked potentials of P-300-type-based paradigms seem slightly better from this point of view.20 However, provided that instructions are well understood and correctly followed, the maximal potential of EEG–BCI performance (that is, proficiency) for a given subject and a given paradigm seems to vary greatly among subjects. The proficiency level required for BCI-mediated robot control can be even more demanding than the more commonly studied BCI cursor control,21 as a safely controlled robot does not react in cases of no motor imagery or movement imaginations/actions, which do not correspond to an agreed-upon mental command (that is, idle-state control is required).

Although over time most clinical tests of BCI focused on patients whose severity of paralysis (almost totally paralyzed/locked-in22 and, meta-analyzed more recently, totally paralyzed/with complete locked-in state,23 resulting in a rather disappointing conclusion for this latter kind of patients) made it too difficult to communicate by other means, limited surveys have been undertaken also with non-locked-in paralyzed users, implying slow brain rhythm modulation as the BCI modality—a paradigm requiring lengthy training (on order of weeks).

Hence, some patients succeeded thereby in achieving 1D and 2D cursor control (for instance, Wolpaw and McFarland, using an adaptive slow brain rhythm algorithm on four subjects—two healthy and two paralyzed after cervical and thoracic SCI, respectively—reported that cerebral activity provided 2-D cursor control)24 and in operating a mental switch for appropriately opening and closing an electrical-driven hand orthosis25 or to command muscle functional electrical stimulation-driven orthoses.26 SMR (or motor imagery)-based paradigms—which favor discrete-valued BCI outputs and which require mere hours of training—were later applied, for instance, to the switching between distinct grasp phases of the lateral grasp provided by an implanted neuroprosthesis, Freehand functional electrical stimulation System, in a post-SCI complete tetraplegic27. As briefly mentioned above, a recent study has shown slight improvements in EEG–BCI control (for instance, the error rate of the EEG classification improved by only 12%) in one muscular dystrophy chronic tetraplegic patient, after more than 5 months of virtual reality (VR)-based BCI training.19

The Brain2Robot team had progressively developed a system consisting of a 7-degrees-of-freedom robot able to grasp objects using SMR–BCI and 3D gaze tracking (Figure 1) and tested it on healthy volunteers;21 the current study was undertaken in the hope that if such technology could be safely used by SCI tetraplegics as well, it might eventually lead to practical everyday use of noninvasive BCI/brain–machine interface (BMI) technology for assistive/rehabilitative aims. For this purpose, a mixed team of researchers, from Fraunhofer FIRST Institute, in Berlin and from TEHBA—mainly from the Physical and (neuromuscular) Rehabilitation Medicine P(nm)RM Clinic Division—in Bucharest, completed a trial on the largest lot, until now, to our knowledge, of chronic post SCI tetraplegics. The study design entailed, on one hand, a proof-of-concept survey and related functional outcomes assessment in terms of BCI/BMI performance, consisting of specific EEG–BCI-training sessions and the combined use, in some subjects, of noninvasive BCI and an assistive robotic arm device, all these in order to evaluate the feasibility of this kind of BCI/BMI to be used by chronic tetraplegics as assistive technology for basic ADL, such as drinking, and thus to improve their autonomy and related QOL. On the other hand, telephone interviews and mail/e-mail correspondence were collected, based on an own, simple, custom questionnaire, within a long-term clinical post-trial follow-up, continued up to 14 months after initial experiments, in order to detect delayed (side) effects and also the patients’ perception of their EEG–BCI control capacity, if present.

Figure 1
figure 1

Left: schematic of coordinate systems and position variables. The base coordinate system (x0, y0, z0) is centered at the base of the robot arm's shoulder joint. The head tracking cameras provide a current estimate of the head-centered coordinates (x0, y0, z0), by means of markers mounted on snugly mounted glasses equipped with two pupil tracking cameras, which are used to obtain the gaze azimuth and elevation angles (φ and θ) and the gaze distance r, in head coordinates. Right: typical set up (pictured). The robot is shown in the middle of a ‘grab’ sequence. The head tracking cameras seen in the schematic are mounted on the facing wall (off-frame)21 (FRAUNHOFER-FIRST, Berlin/TEHBA, Bucharest; 2008).

Full size image

Open questions central to clinical and assistive/rehabilitative EEG–BCI/BMI feasibility, which we aimed to address, included the distribution of SMR–EEG–BCI performance among tetraplegics, what could be done in terms of test paradigm to maximize this performance, how a case-by-case evaluation of EEG–BCI/BMI feasibility could eventually be implemented in a clinical setting and, importantly, what the patients’ informed qualitative opinion of EEG–BCI/BMI potential is.

Materials and methods

To efficiently use an EEG–BCI/BMI system, including an assistive robotic arm as in our study, a novel subject must previously undergo BCI training; the proficiency he/she achieves is critical to overall success, and therefore we present the following:

Subject selection

EEG-based BCI testing/training sessions were undertaken on a group of volunteers recruited from among inpatients of the P(nm)RM Clinic Division of TEHBA. We enrolled nine chronic tetraplegics (eight men and one woman), aged between 23 and 51 years (average 33.11, median 34), according to a custom and mutually agreed medical protocol, which covered the necessary appropriate safety and confidentiality precautions28, 29 for the trial, including written informed consent from each subject and ethical review board commissions’ approvals by the appropriate local affiliations (TEHBA and UMPCD).

Each subject was clinically and functionally assessed, including assessment of the AIS (American Spinal Injury Association Impairment Scale) score30, 31 (http://www.asia-spinalinjury.org/publications/59544_sc_Exam_Sheet_r4.pdf), by TEHBA's physicians who contributed to this study. The enrolled tetraplegics’ gender, age, neurological level of SCI, time elapsed since the SCI, AIS (in figures converted/expressed) Frankel impairment degree, AIS scores, percentage of EEG–BCI performance (calibration accuracy) and the preferentially collected most discriminative EEG frequency bands (represented by min/average/max, see ‘EEG–BCI analysis and control’ below) are shown in tabular form (Table 1 ).

Table 1 Patient data, including: initials (changed by convention), gender, age, neurological level of SCI, time elapsed since the injury, AIS (in figures converted/ expressed: A=1; B=2; C=3) Frankel impairment degree, AIS scores, percentages of EEG–BCI performance (calibration accuracy) and preferentially collected most discriminative frequency bands (represented by the statistical afferent functions min/average/max)
Full size table

Inclusion criteria

  • Chronic (that is, longer than 6 months) post SCI patients with tetraplegia, motor complete or severe incomplete.

Exclusion criteria

  • History of associated moderate or severe traumatic brain injury at the time of SCI and/or other brain current/previous condition(s), including the need for specific medication

  • Current or previous psychiatric-associated condition(s), including the need for specific medication

  • Excessive spasticity—>3 on Modified Ashworth Scale32

  • Active period of severe postural hypotension, of autonomous dysreflexia (AD) and/or other cardiovascular conditions

  • Pressure sores

  • Fever

  • Active urinary tract infection (UTI)

  • Dyspepsia.

Safety and comfort measures

As not all of the participants in the study were able to sit in a wheel chair during the BCI sessions, which lasted up to 3 h, we tried to maximize the patients’ comfort on a case-by-case basis, especially the choice of upright or supine position (see further).

Our initial (pre/intra/post training) survey (that is, the clinical–functional aforementioned examinations and medical assistance, if needed, and the training sessions of BCI/BMI, using the cap with electrodes for EEG recording, appropriate software for data collection, computer tuning and robot control sessions) was carried out during 10 days at the premises of TEHBA.

Within this period, of the initial (pre/intra/post training) survey, each of the tetraplegic participants was assisted and medically supervised by a physician from the P(nm)RM Clinic Division of TEHBA in case of any autonomous dysreflexia or other problems that might have appeared (although all patients were stable, we considered it preventively necessary). Subjects were instructed to solicit the termination of the session if they felt any discomfort.

Clinical questionnaire

Telephone interviews and mail/e-mail correspondence were carried out based on a custom-designed, simple questionnaire (such that participants could respond comfortably given their difficulties in assisted/unassisted writing; see Table 3). Hence, referring to their health-related state during training and post training, within the 10 days of the experiments and at about 6 and 12 months after the initial trial, we asked them whether they felt any discomfort/trouble (boredom, fatigue, sleepiness, insomnia, headache, dizziness, nausea, anxiety, concentration and/or memory matters, or any other relevant discomfort and a description thereof) during, immediately after and/or a long time after the experiments. We also enquired about their personal qualitative assessment of whether they felt they were in control of the BCI (the cursor) and/or the BCI/BMI (the robotic arm) system.

Technical approach

A 64-channel DC amplifier setup (BrainAmp128DC, Munich, Germany) was used, containing 54 EEG channels, two Electromyography (EMG–that is, one bipolar) channels for left hand muscles, two EMG channels for right hand, two EMG channels for foot muscles—only for the training session—and four electrooculogram channels. The 54 gel-prepared EEG channels were evenly distributed over a standard 128-channel cap, in an extended 10–20 system. Visual cues and feedback required by the paradigm were displayed on a computer screen. The offline BCI analysis, as well as the online cue presentation and feedback cursor control, was performed using Matlab (Mathworks, USA) and VTK (www.vtk.org).

The study entailed three consecutive phases: calibration (alternatively, training), feedback and robot control. In this text, by ‘tuning’ we refer to learning by computer and by ‘learning’ we refer to subjects, whereas the procedure of obtaining calibration data and for the subject to practice motor imagination was denoted ‘training’. The term ‘cues’ refers to audio-visual signals that were used in the training phase (for example, a voice command that says ‘left’ and a video of a left hand movement, and the corresponding initial command letter is displayed on the center of the computer screen). The types of mental tasks chosen as BCI commands are referred to as ‘classes’. Both training and feedback were presented in two variants, ‘normal’—that is, according to protocols standard at the time of the study and ones for which we had encouraging preliminary results— ‘video’/VR, which was different for each subject, in order to explore the means of increasing the percentage of proficient subjects for whom BCI/robot use would be ultimately successful.

Training

In the first part of the experiment (‘training’), 1–2 sessions of 140 sequences each (see Table 2 ), consisting of random sequences of 35 trials of four classes (mental tasks), randomly distributed among ‘left’, ‘right’, ‘foot’ and ‘relax’ cues were presented visually by means of a letter, which appeared in the middle of the computer screen (‘S’, ‘D’, ‘P’, corresponding to beginning letters for each cue/class name in the subjects’ native language, that is, Romanian, and ‘X’ for ‘relax’). The instructions were to imagine arrhythmic movements of the fingers of the right hand (‘D’, ‘right’) or left hand (‘S’, ‘left’), of (both) ankles (‘P’, ‘foot’) and no movement (‘X’, ‘relax’). The subject was asked to imagine the cued class while moving neither the limbs nor the eyes.

Table 2 Results of BCI training and performance
Full size table

There were two types of training cue sequences: ‘letter’, meaning only the letter is shown, and ‘audio-visual’, in which a video of the movement to be imagined (arrhythmic fluttering of fingers or ankle flexion/extension) was presented along with the letter. The timing of the imagination was cued by four sounds (beeps), equally spaced at 1 s, the subject being instructed to begin the movement imagination at the fourth beep. Apart from offline checks, surface electromyograms (bipolar EMGs placed on flexor carpi radialis or the most distal arm muscles, for which contraction could be previously observed or felt) and electrooculograms (vertical/horizontal—bipolar for left eye) were monitored. The duration of each training session was about 30 min compared with a maximal duration of the complete experiment of 3 h, including preliminary preparations.

After the training session, subject-specific frequency bands and spatial filters (Common Spatial Patterns or CSPs, see below for description) were chosen using cross-validated tuning of parameters, whereas trials and channels judged to be outliers due to electrode malfunction, excessive muscle activity and/or movement were removed by computational heuristics15 and visual inspection. The most discriminable pair of classes was identified using a 10-fold cross-validated classification error on calibration trials; see Table 2.

Feedback

In the second part of the experiment, subjects were asked to horizontally move a cursor, represented by a cross displayed on the screen to a target represented by a bar on either the right or left side of the screen by imagining the corresponding pair of classes (‘1D feedback trials’/‘normal trials’), which was previously calibrated: this class pair was different for each patient and was aimed at maximizing classification accuracy among possible class pairs (see Table 2; ‘Class used’). The cross movement speed was proportional to the output of the CSP classifier in one of two directions (see equation 1 below, last line), each corresponding to one mental imagination task, thus providing continuous performance feedback to the subjects. After the training phase, 1 to 3 sessions (see Table 2), each consisting of several (56 in average) normal, synchronous left/right target feedback trials, were performed using best class pair combinations determined from the training data. The patients who were judged to be good performers (>70%) on calibration sessions and continued to the feedback phase were split into two groups: those using VR training (‘feedback session, video’, ‘FV’; see Table 2) and those who did not (‘feedback session, normal’, ‘FN’; see Table 2). This phase lasted longer in the case of VR feedback use or in the case that a second pair of classes was tested to improve performance.

The subjects then performed multiple idle-to-active feedback trial attempts. The cues were given to imagine one of the motor imaginations or to relax by means of large targets on either side of the screen: the cursor movement started from the middle of the screen as soon as the cue was given.

For reasons of patient comfort, two of the patients maintained a supine position and received cues from a computer-generated image, projected on the ceiling (Figure 2). Two participants performed additional feedback sessions using a 3-D image of hands moving on the display (Figure 3 and, respectively, ‘feedback VR trials’; see Table 2), and all robot testings were conducted with the subjects sitting in a wheel chair placed in front of a standard table (Figure 1).

Figure 2
figure 2

Training session for the tetraplegics: the patient was lying on a bed and was receiving cues from a screen displayed by means of a projector on the ceiling above him/her (FRAUNHOFER-FIRST, Berlin/TEHBA, Bucharest; 2008).

Full size image
Figure 3
figure 3

Training session with a tetraplegic subject in wheel chair and in front of a standard table (on the display there can be seen a virtual representation of a pair of upper limbs: their movement can be controlled by the subject through EEG sensory motor rhythms, EEG–SMR; FRAUNHOFER-FIRST, Berlin/TEHBA, Bucharest; 2008).

Full size image

Eye tracking and robot control

Eye tracking was used to provide motion end-point information to the robotic arm by inferring the location of the object to be grasped from the gaze focus point concurrent with motor imagination. In contrast to common applications, such as tracking the user's gaze trajectory over a computer screen or an image, which requires 2-D gaze tracking, identifying the target position in 3D space requires 3D (stereo) gaze tracking. For this purpose, we used a pair of head-mounted cameras that track both the left and right pupil (View Point Eye Tracker, Arrington Research, Scottsdale, AZ, USA). The three spatial coordinates of the gaze point were parameterized in spherical coordinates as azimuth, elevation and distance, with respect to the origin of a hypothetical reference frame fixed to the head. A schema is shown in Figure 1. The position and rotation of the head reference frame, as well as the gaze tracking calibration target, were located and tracked in 6 degrees of freedom using an OptiTrack system with the Rigid Body Software (Natural Point, Inc. Cornvallis, OR, USA).

The 3D gaze-tracking problem consists of mapping pupil coordinates for left and right eye to a 3D point into head centered coordinates. To learn this mapping, we trained a linear basis function model u = wT f(x), where the four-dimensional input vector x comprises the pupil coordinates [xL,xR,yL,yR ], u is any of the outputs φ, θ or r, and the basis functions f(x) are polynomials up to second order, that is, f(x) ∈ { 1,xL,xL2,xLyL,...,yR2}. To prevent over-fitting, we assumed a Gaussian distribution of the weight vector and learned its parameters (by Bayesian linear regression), instead of inferring only its maximum. When predicting gaze, we integrate over this distribution analytically.33

The subjects manipulated an object (standard drinking glass, 6 cm diameter, randomly placed on the table) using a robot arm (Assistive Robotic Manipulator, Exact Dynamics, Didam, Netherland) mounted on a standard table (Figure 1). The ‘grab’ command was given to the robot when the robot was in ‘home’ position, the head position and gaze vector were stable (within 2° for over 2 s), the gaze distance was less than 1 m, the target point was within the robot's workspace and an activation threshold was reached by the BCI command. The target point was calculated as the intersection of the gaze vector with the table, z0 = 0 (Figure 1). For further details see Danóczy M et al.21 The instructions given to the subjects were to wait for the experimenter to place the glass on the table, and then at their own volition and pace to activate the BCI system by performing the agreed-to ‘grab’ class (the first of the feedback class pair) and by focusing steadily on it. After the robot grab action sequence was completed, they were instructed to perform the imagination again, as to place the glass back, and the entire sequence could commence again.

The ‘grab’ action consisted of grabbing a plastic glass placed on the table by moving the robot with an open gripper 10 cm closer to the subject in the z0 direction than the estimated target point (glass position), and then moving forward 20 cm while closing the gripper. The glass was brought by the robot to ‘drinking position’ (where the top of the cup was ∼15 cm in front of the subject's chin) and then placed back on the table (‘place’) upon another BCI activation, after which the robot returned to ‘home’ position and a new sequence could be started. The sequence was repeated 5–6 times for each subject, which completed the robot control phase (see Table 2), with success (glass grabbing) observed in at least three trials in each subject.

EEG–BCI analysis and control

The basic pattern-recognition process relied on the discrimination between two brain states (corresponding to two classes), using a heuristic search for the most discriminating frequency band for a particular subject's training data15 and a widely used algorithm (in EEG–BCI) known as CSP. The latter was used to obtain suitable spatial filters for performing EEG–BCI classification in a two-class paradigm (for example, left/right hand movement imagination).34 It has benefited from many enhancements over the past decade, some of which are described in the context of the Berlin Brain–Computer Interface.15

CSP assumes that the signal measured by EEG sensors is a linear spatial mixture of (unknown) original sources. In the equation below, the EEG measurements are denoted by v(t), the matrix of 2k filters W is the CSP filter matrix and h(t) is the impulse response of the frequency band filter (5th order Butterworth with low- and high-pass parameters as in Table 2, sampling frequency 200 Hz). The frequency band is calculated from the R2 statistic of the difference among spectral density averaged over Laplace-filtered central–parietal electrodes at 2 Hz resolution, by iteratively and symmetrically expanding an interval from the frequencies with the two highest R2 values, until the total R2 score is greater than 85% of the total score. The same procedure (85% of total R2), using the root-mean-square power after band filtering instead of spectral power, is used to obtain a poststimulus time interval for integration. The value z(t) is the band-filtered EEG, and c(t) is the classifier output after scaling by the linear discriminant analysis classifier vector a and additional bias b, whereas the cursor position p(t) is a time-integrated version thereof.

The rows of the unknown mixing matrix W are called patterns, whereas the columns of the de-mixing matrix W−1, which can be interpreted as possible solutions of the so-called inverse problem, are called filters. The goal of CSP is to find spatial projections in sensor space that optimally de-mix the measured signal by maximizing the variance in one class while minimizing the variance in the other class, thereby achieving optimal discrimination for subsequent classification. The filters are obtained by solving a generalized eigenvalue problem while simultaneously diagonalizing the covariances of both classes. The cursor integration speed r varied among feedback sessions (min 0.5 max 4 mean 1.16), and was chosen such that the subject could reach a threshold value of ±0.85 between 2 and 4 s.

Statistical analysis

The main aim of the statistical analysis was to identify possible predictors of EEG–BCI performance (denoted EEG_BCI in Figure 5) before any actual EEG–BCI testing and calibration, multiple linear regression (MLR) being chosen for the purpose. To see whether the predictors were themselves related or redundant, we used cluster analysis according to Ward's method (which groups individual variables by taking into account their correlations). MLR and clustering were performed using the SPSS package (http://www.spss.com/). Apart from EEG–BCI calibration performance (the predicted variable), the following variables were considered as possible predictors: age of the subject (denoted ‘age’ in Figure 5), AIS motor score (denoted ‘motor’ in Figure 5), AIS sensory score (denoted ‘sensitive’ in Figure 5), time elapsed since SCI (denoted ‘months_since_SCI in Figure 5), training feedback performance (denoted ‘feedback_perf’ in Figure 5) and the individual spectral power components of each subject's typical EEG spectrum (see also Tables 1 and 2; denoted ‘theta’, ‘beta’, ‘gamma’ and ‘alpha’ in Figure 5), and were included in the clustering linear regression-based analysis.

Results

During the initial clinical (pre/intra/post training) survey and, respectively, the long-term post-trial follow-up period, no significant adverse reaction occurred, except for headache in one patient, 1 day after the experiment, discomfort/fatigue felt by three subjects (see Table 3 ), including one patient who fell asleep during one of the training sessions, and mild transient postural hypotension in one patient who could therefore not perform the feedback phase of the study.

Table 3 Post-trial (including long-term) follow-up: synthesis of the personal statements resulting from the correspondence (by telephone/mail/e-mail) done with each patient participant to our study
Full size table

In synthesis, one subject felt discomfort after the calibration session and therefore did not perform feedback trials, two subjects felt discomfort after the feedback trials, three other subjects did not reach 70% accuracy, and two of them, see Table 3, verbally reported being not certain of mentally controlling the cursor, and under mutual agreement further sessions were not attempted. Three subjects completed all phases including robot control.

All subjects completed the calibration phase, seven out of nine completed the feedback phase and three out of seven completed the robot control phase as well.

Considering the aforementioned minor symptoms encountered during the initial (pre/intra/post training) experimental phase of the research (10 days), we considered appropriate to continue the study by adding a second phase: a post-trial follow-up survey consisting of telephone interviews and mail/e-mail correspondence with the enrolled participants after they were discharged (being at their homes and, obviously, without the continued possibility of using the BCI/BMI apparatus). This was done to check, on one hand, for any delayed side effects and, on the other hand, their own perception on their capacity to control, by BCI, the cursor and/or (BCI/BMI) the robotic arm device (see Table 3).

The post-trial follow-up was carried out in a period starting at 6 months (in order to be certain that any possible reported events could cover a long-term follow-up), which lasted up to 14 months after the end of the initial (pre/intra/post training) survey. Considering also that most of the tetraplegics either cannot write at all or do so with great difficulty, some written answers from the enrolled subjects justifiably arrived with significant delays (up to 14 months after the initial experiments). In the questionnaire responses, seven (77.7%) of the nine subjects reported having felt control of the cursor and three (33.3%) of them over the robot as well (Table 3). These self-assessments are consistent with the numerical results of feedback/training accuracy except for subjects CU and DZ who reported having controlled the cursor despite a peak accuracy of only around 52% in feedback sessions.

EEG–BCI performance classification trial/EEG–BCI performance accuracy averaged 81.0%, with median 79.2%, whereas feedback training performance/feedback training session accuracy (in which subjects were asked to reach virtual targets within a maximum of 10 s left/right on the display) reached an average of 70.5% with median 68.8%. For the seven subjects who completed at least one feedback training session, individual best-session target achievement accuracies (‘Feedback Best’; see Table 2) were as follows: XE: 94.3%, XE(2): 71.4%, DZ: 52.3%, XY: 87.5%, QM: 72.1%, CU: 52.0%, HD: 68.5%, YX: 69.9%. There was a significant (P<0.05) Pearson's correlation coefficient between training trial accuracy and feedback trial accuracy of 0.902. There was no statistically significant difference (as indicated by the bilateral t-test) of average BCI performance for patients performing calibration in a supine position (86%) compared with patients who did it in sitting position (79%). There was no significant difference (t-test) between sessions using video or normal training cues (P<0.33). Similarly, there was no such difference among VR and normal feedback sessions (P<0.57, two subjects performed VR feedback, and two others used left/right hand imagination suitable for VR).

In designing our subject assessment procedure, we were forced to make rational compromises between maximizing the chance of success of EEG–BCI (and subsequently of robot use) on one hand and of subject comfort and potential performance-reducing effects of boredom and fatigue on the other hand. The calibration phase was uniformly undertaken by all subjects, being just long enough to be able to ascertain the potential of any combination of recorded mental imaginations to provide a useful level of classification accuracy. As the ultimate reliable test of EEG–BCI proficiency in robot use is feedback (that is, cursor) performance, varying numbers of sessions, featuring different mental imagination combinations (among promising pairs from calibration data analysis), were necessary and, respectively, different types of feedback were performed, with a goal of 70% accuracy as prerequisite to robot use.

MLR was performed on spectral power components of each subject's typical EEG spectrum (Figure 4) in order, as already mentioned, to identify probable predictors of BCI training accuracy from potential prerecorded clinical data. Using the theta power band (4–8 Hz) as a normalization factor, it resulted that beta (13–30 Hz) spectral power density was positively correlated (coefficient 0.432, standardized coefficient 0.745, P=0.025), whereas gamma power band (31–40 Hz) was negatively correlated (coefficient −0.500, standardized coefficient −4.014 P=0.203) with BCI performance defined as training accuracy see Tables 1 and 2.

Figure 4
figure 4

Spectral shapes and class-related variation at electrode ‘C4’. The spectra (dB) are averaged from 1 Hz resolution FFT of 500 ms windows with Hamming windowing and 250 ms overlap, and staggered (for readability) according to BCI performance with best performers at the top. Width of each line is proportional to mean class difference divided by pooled s.d. Dotted lines represent stagger in the y axis for each patient and 1/f2 spectral roll-off.

Full size image
Figure 5
figure 5

The variables considered in the study represented in the main components plane, obtained by principal components analysis.

Full size image

Taking into account only the beta component of the EEG spectrum influence, in order to reduce the number of predictors considered, relative to the small number of cases, the additional influence of clinical variables/factors on EEG–BCI performance was tested in a second linear regression model. The results showed that the sensory AIS score (range: 0 min to 224 max, the level of deficit being 224; coefficient −0.177, standardized coefficient −0.512, P=0.089) was also a potential influence. The rest of the assessed independent variables in the model (including subject age: P=0.31, AIS motor score, and time elapsed since SCI) had negligible regression coefficients.

Cluster analysis (Ward's method where the distance was determined by the Pearson's correlation coefficients) revealed that EEG–BCI performance is grouped near to the beta spectral density and to feedback performance (‘feedback_ perf’), and is far from the variables theta, age, motor and months since SCI, which compose another cluster. The interrelation among these variables can be confirmed by looking at Figure 5 and considering its caption/ explanatory text below it.

We did not observe differences in performance among subjects and trials with and without video/audio-enhanced training or VR-perspective feedback control. It was hoped that including VR perspectives and audio/video-based imitation cues would increase final (feedback) performance. Although this has not happen, these did help, in our opinion, to make the paradigm more easily understandable and interesting to our subjects.

Discussion

When comparing BCI modalities, it is important to estimate the percentage of patients who can effectively use it. Quantitative measurements of the feasibility of BCI decoding among a population are often based on calibration accuracy (that is, analysis that can be done offline) and depend on an priori-determined threshold that separates proficient vs ineffectual users; various researchers may use different values based on experience. Also pertinent is the subjects’ own perception: seven of nine patients who reported a feeling of intentional control is encouraging, but what we can quantitatively conclude is that three of the nine patients (33%) showed accuracies of more than 70% in both offline/calibration and online/feedback training and eventually controlled the robot arm device as well. In the latter, the state of the action is shown to the subjects in real time, yielding an error value correlated with but slightly higher than calibration error. The performance distribution reported herein is consistent with the literature on SMR–EEG–BCI performance in healthy subjects.18 Previous comparative studies provide rather inconsistent results concerning differences in BCI control performance/accuracy in post-SCI patients and included rather small numbers of subjects/ patients;18, 24 any future comparative studies will have to overcome complexities of factors matching among patient and healthy populations.

Perhaps more important is the fact that both calibration and feedback training errors were below 30% for three subjects; thus, they were able to control the robot without intervening discomfort and complications. Admittedly, three of nine is not extremely encouraging, and the success rate of invasive BCI, assuming that invasive recording stability and safety issues will be conclusively resolved, may turn out to be higher. Given the current inherent safety and cost advantages of EEG–BCI, a simple calculation of the expected target tetraplegic population can shed further light on its ultimate usefulness. TEHBA researchers have initiated29 a system online of specific data acquisition/registration (including with dynamic clustering for self-updating of each logged-in post-SCI patient's personal identification profile), which is pending regulatory approval. According to a related national database query, the Romanian incidence rate of SCI was, for instance, 0.0185% in 2009.35 Combining these results with the internationally determined tetraplegia to paraplegia ratio (https://www.nscisc.uab.edu/public_content/pdf/Facts%202011%20Feb%20Final.pdf) and with the total estimated population of the European Union (EU) of around 500 million people (http://epp.eurostat.ec.europa.eu), and also with a related estimation of the Parliamentary Assembly of the Council of Europe (http://assembly.coe.int/Documents/AdoptedText/ta02/EREC1560.htm), we can (largely) infer that the total number of tetraplegics in Europe is about 150 000–250 000 persons. Further, it is to be considered that, at present, complete tetraplegia–the main target of the assistive technology we have tested in this survey–is 16.3%, according to very recent epidemiological data (https://www.nscisc.uab.edu/public_content/pdf/Facts%202011%20Feb%20Final.pdf). Hence, there should be approximately 35 000 persons in Europe alone who may benefit from a specialized market of EEG–BCI/BMI-based assistive technology.

For those patients who are not SMR–EEG–BCI proficient, either other residual muscle control modalities or other types of BCI and/or other—possibly including combined—kinds of assistive related devices might possibly be used. Further practice could increase the number of SMR–EEG–BCI proficient subjects, albeit slightly. Among other possible factors underlying inter-subject EEG–BCI proficiency are wide variations in innate ability/proficiency, personal determination and motivation—factors that are very difficult to assess individually.

The influence of AIS sensory score would imply that a higher sensory deficit could favor a slightly increased ability to perform motor-imagery-based BCI. One reason for this effect might be the task itself: motor imagery. For a healthy subject this is a difficult action to perform, as he/she must monitor sensory feedback to ensure that no movement is elicited. Our tetraplegic subjects never reported difficulty in performing it, as the elicited hand movement imagination involved highly affected somatic areas.

Previous studies using EEG recordings showed some influences of the body posture (standing, sitting or supine) on cortical bioelectric activity, mainly regarding the activation of the dominant hemisphere frontal cortex. The adaptive reactions related to body posture involve the baro-receptors and the locus coeruleus noradrenergic integrative systems, which have an important role in both stress-related and psycho-stimulant-induced arousal, modulation of perception and other behavioral processes, such as attention and cognition.36, 37, 38, 39 Owing to the existence of multiple noradrenergic systems, postural differences induce a complex influence upon the locus coeruleus modulator activity. It was emphasized that the supine position was associated with decreased motivation and aggressiveness40, 41 and increased ability and speed in solving anagrams.42, 43 The relationship between posture, bioelectric patterns of cortical activity and related modulation of neuropsychological aspects (motivated perception and implementation of goal-directed behavioral responses) is complex and not completely understood. Although the sympathetic–adrenergic system in chronic tetraplegics is generally re-adapted to an upright sitting position from the supine one, which dominates the acute and subacute post-SCI stages, for two of the enrolled subjects the risk of postural/orthostatic hypotension was judged to be most safely minimized in the supine position. Under these circumstances, one of the two patients who did train in a supine position succeeded to control the robot too, and did so from a seated position.

Therefore, it seems that much of the BCI training can be done from a supine position. In addition, combined physical and psychological factors influence the level of endurance of post-SCI tetraplegic patients, especially if they sit upright. The assistive use of BCI/BMI could be beneficial, including as a motivating endeavor. In the survey performed, three (Table 3) of the patients reported fatigue, mainly from the concentration required: this can be seen as a disadvantage (motor imagination does require concentration, even in the case of paralysis) and an advantage (it encourages the patient to play an active part in attaining concrete motor action goals using motor imagination).

Besides BCI there are three other technologies needed for the implementation of the Brain2Robot system: eye tracking, head tracking and a robot. The basic technology in all three cases is evolving rapidly: head tracking is now part of commonly available gaming consoles such as the Microsoft Xbox/Kinekt, and eye tracking is now incorporated in multiple open-source software based on low-cost webcams (http://thirtysixthspan.com/openEyes/). Many of these platforms appeared in a short time since the experiments were conducted; robustness and cost depend on market size, and the standardization of the requisite supply chain of basic parts and software components has also evolved positively. The system tested in this study is merely a proof of concept, prerequisite and hopefully encouraging for entities willing to test and commercialize such systems; the market size, estimated above, certainly allows for opportunities to do so.

In addition, we can also hope that in the near future EEG–BCI caps (specifically dry electrode based ones), coupled with mechatronic robotic arms and/or lower body exoskeletons, could contribute to assist not only basic ADL—requiring the use of upper limbs tetraplegic patients, as in our trial—but the orthostatic posture and locomotion, in patients with complete paraplegia, as well.44 In this respect, the near future might bring further revolutionary achievements, including in the field of advanced assistive/rehabilitative technologies; some of the TEHBA researchers, including the corresponding author, have already related, recognized results (the Gold Medal at ‘Salon International des Inventions’, Genève 2008, for ‘Dispositif orthetique mechatronique’), and also anticipations regarding the possible combination between EEG-BCI/BMI using dry cap electrodes and a mechatronic exoskeleton, activated including by the user's cerebral voluntary commands, that may contribute to wireless power a ‘robotic suit’ engine for bionic standing and gait assistance, in paraplegics.44 This state-of-the-art concept and trend gathers, at present, valuable international research forces such as the ‘Walk Again’ project (http://www.walkagainproject.org/?page_id=3).

Conclusions

The potential of motor imagery EEG–BCI/BMI to be useful for daily self assistance in chronic tetraplegics relates to their BCI training accuracy performance, which, in light of our results, can be predicted by relative beta spectral power density, positively (increasing therewith), and AIS sensory score, negatively (higher sensory deficit may mean a slightly increased ability to perform motor imagery based BCI), although data from more subjects are necessary to validate this latter finding.

Hence, EEG–BCI/BMI could be a valuable method to compensate for some important limitations in chronic tetraplegics and possibly other severely disabled patients, useful in daily self assistance for basic ADL, such as using/manipulating a drinking glass, and thus has a potential to improve their QOL, given further improvements in miniaturization, user friendliness and cost of the underlying technology.

The current study promises the development of a rational and effective procedure of screening and training post-SCI tetraplegics for EEG–BCI/BMI use, given the reality that only a moderate fraction of these may be able to ultimately benefit.

Data Archiving

There were no data to deposit.