Introduction

Functional electrical stimulation (FES) elicits muscle contraction using electrical impulses and is used as a motor neuroprosthesis to facilitate movement after spinal cord injury (SCI). The term ‘neuroprosthesis’ will be used in this study to refer to a motor neuroprosthesis.

Neuroprostheses are controlled using switches, linear variable resistors, joysticks, position sensors, electromyographic signals and speech.1 A more intuitive method for controlling neuroprosthetic devices would be to use brain activity. Brain–machine interfaces (BMIs) translate brain signals into control commands for electronic devices.

Electrical signals reflecting brain activity have been used most extensively for developing BMI systems. Non-invasive techniques for recording the electrical activity of the brain include EEG (electroencephalography). Invasive techniques allow the recording of local field potentials (LFPs) reflecting the activity of a group of neighbouring neurons. LFPs can be recorded using macroelectrodes placed on the surface of the brain, resulting in electrocorticographic (ECoG) signals, and using microelectrodes placed intracortically. These microelectrodes can also record the activity of individual neurons.

Operating an EEG-based BMI requires the user to change brain activities voluntarily. To do this, the user is often trained for up to several months. Therefore, an important challenge in the development of BMIs is minimizing the required training. For example, using ECoG signals, the training time can be reduced to the order of minutes.2

Spectral and temporal changes in brain activity elicited by voluntary movements have been used to reduce the training time of a BMI user. These changes have allowed the identification of specific movements and the detection and estimation of kinematic parameters of the motion performed.3

Single cell recordings in monkeys and in humans have yielded important results in the detection and estimation of kinematic parameters from brain activity making it possible to control computer cursors and robotic arms using the activity from a group of neurons.4, 5 However, there are concerns regarding the reliability and long-term stability of single neuronal recordings.6

Local field potentials, including ECoG recordings, offer alternatives to single neuron recordings. The changes in power of frequency components of LFPs reflect kinematic information of arm movements,7, 8, 9, 10 and recently it was possible to predict the hand position from ECoG signals.6

The convergence of the fields of neuroprosthetics and BMI seems to be the natural next step in the development of these two fields. Using BMI technology, a neuroprosthesis could detect the intention to perform a specific movement and deliver the electrical stimulation to produce that exact movement. The benefit of such a system would be enhanced further if the user required little or no training to use this device.

Upper limb neuroprostheses11, 12, 13, 14 have been controlled by individuals with quadriplegia using single cell recordings15 and EEG signals after training that lasted between 3 days and 4 months.12 To generate control commands, the EEG-based BMI users used motor imagery,11 closing and opening of the eyes13 and self-regulation of power in specific frequency bands.12 The accuracies reported for these systems range from 7613 to 94.2%.12

The purpose of this study was to test the feasibility of using ECoG signals as a control strategy for a neuroprosthesis for grasping. A neuroprosthetic device was created and controlled using ECoG signals acquired earlier. In the following section, it will be shown that an ECoG-driven neuroprosthesis can be implemented using a 4-channel ECoG electrode and minimal training time to achieve 94.5% accuracy.

Materials and methods

To test the integration of a BMI system and a neuroprosthesis for grasping, it is necessary either to (1) implant an individual with SCI with an ECoG electrode or (2) use ECoG signals recorded from another individual, already implanted with a subdural electrode, to control a neuroprosthesis instrumented on an individual with SCI. As we could not justify the implantation of the ECoG electrode solely for testing the system feasibility, we used the second approach.

Subjects

Two subjects participated in this study. Subject 1 was a 67-year-old woman implanted with subdural (ECoG) electrodes for the treatment of essential tremor (Figure 1a). She was recruited from the movement disorder clinic at the Toronto Western Hospital and gave informed consent to participate. The study was approved by the University Health Network Research Ethics Board.

Figure 1
figure 1

Participants. (a) X-ray image of subject 1 with subdural electrodes implanted on the motor cortex over the representation of the upper limb. (b) Subject 2. (c and d) Surface stimulation electrodes used to elicit palmar and lateral grasps.

Full size image

Subject 2 was a 35-year-old man with a complete cervical SCI (C6 level/ASIA B) and had received 4 weeks of FES therapy treatment for restoring grasping function, as part of another study.16 He was able to use his arms and wrists but had no hand movement. He gave written and informed consent to participate in this study as required and approved by the Toronto Rehabilitation Institute Research Ethics Board. Figure 1b shows subject 2 wearing the neuroprosthesis for grasping.

We certify that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed-up during the course of this research.

Electrocorticographic and motion recordings

Electrocorticographic signals and arm movements were recorded simultaneously from subject 1 during a single 1-h session 3 days after the initial implantation of the subdural electrode. The subdural electrode had a single row of four platinum–iridium disc contacts (4 mm diameter and 10 mm centre-to-centre distance, RESUME 3586, Medtronic, Minneapolis, MN, USA). The electrode was implanted over the arm representation of the left motor cortex, which was confirmed intra-operatively using electrical stimulation delivered directly to the brain and observing contractions of the right upper limb.

Monopolar ECoG signals were band limited (0.5–500 Hz) and recorded (sampling rate=2 kHz, SynAmps2, Compumedics, Charlotte, NC, USA). The ECoG signals were downsampled to 200 samples per second and subtracted from each other to create differential signals using the recordings from non-adjacent electrodes (for example, contacts 3 and 1).17

Using only the right upper limb, subject 1 performed wrist flexion, reaching to the right and reaching to the left movements after an auditory cue and held the final position of the movement until a second auditory cue. A motion sensor was placed over the dorsal aspect of the third metacarpal of the right hand to record the movement using a six-dimensional (X, Y, Z, roll, yaw and pitch) motion-capture system (Fastrak, Polhemus Inc., Colchester, VT, USA). Only position recordings (X, Y and Z) were used for this study.17 Each movement was repeated at least 35 times, and each trial was visually inspected to identify mistrials, defined as: (1) a trial in which the individual had performed a movement different from what had been instructed; (2) the participant had started to move before the auditory cue; or (3) the movement was not completed. Table 1 and Figure 2 show the movements performed along with the number of trials used to carry out this study.

Table 1 Movements performed by subject 1 along with duration of movement, available trials and movement ranges
Full size table
Figure 2
figure 2

Movements performed by subject 1. These included (a) wrist flexion (WF), (b) reaching to the right (RTR) and (c) reaching to the left (RTL).

Full size image

We created a nearest neighbour classifier (Matlab, Mathworks, Natick, MA, USA) using five trials of each motor task to identify the performed arm movements by analyzing the ECoG signals. The remaining trials were used to test the neuroprosthetic system.

To classify the subdural signals, we identified ECoG spectral components correlated with the kinematic components of the arm movement (Pearson correlation coefficient >0.1; statistics degrees of freedom=600). Time-resolved spectra were obtained using a spectrogram (128-sample Hamming window, 128-point fast Fourier transform and 127-point overlap). The 20 frequency components with the highest correlation coefficients were grouped using a histogram with bins representing frequency bands of 10 Hz. Details of this process can be found in Chin et al.17

Neuroprosthesis for grasping

The right hand of subject 2 was fitted with a neuroprosthesis for generating palmar and lateral (key pinch) grasps.18 The neuroprosthesis was designed and created specifically for this study using a Compex Motion four-channel transcutaneous electrical stimulator (Compex S.A., Lausanne, VD, Switzerland).

The grasping movements were achieved by stimulating: (1) flexor digitorum superficialis and flexor digitorum profundus using two electrodes connected in parallel with channel 1 (20 mA) for generating finger flexion; (2) flexor pollicis brevis using channel 2 (14 mA) for generating thumb opposition; and (3) extensor digitorum communis using channel 3 (22 mA) for generating hand opening. The Palmar grasp was obtained by stimulating channels 1 and 2, simultaneously. The lateral grasp was achieved by stimulating channel 1 followed by channel 2, 500 ms later. Stimulation of channel 3 generated hand opening. In all cases, the stimulation frequency was set at 40 Hz and the pulse duration was 300 μs. Figure 3 depicts the stimulation profiles to elicit the grasping synergies.

Figure 3
figure 3

Stimulation sequences for (a) palmar and (b) lateral grasps. Both stimulation sequences used three stimulation channels. Each channel stimulated a different muscle or nerve at different times for generating synergistic movements. (I) Pressing button 3 was used to turn the stimulator on through a random selection and classification of electrocorticographic (ECoG) signals recorded while subject 2 was reaching to the left (RTL). (II) Pressing buttons 1 or 2 caused the system to classify randomly selected ECoG signals recorded while subject 2 was performing wrist flexion (WF) or reaching to the right (RTR), respectively. The result of the classification triggered a specific stimulation sequence to elicit palmar or lateral grasps. (III) Grasping was sustained until either button 1 or 2 was pressed a second time, which resulted in a change in the stimulation delivered to facilitate hand opening. (IV) After 3 s, the stimulation stopped and the neuroprosthesis returned to an idle state. (V) Button 3 could also be used to turn the stimulator off through the classification of a randomly selected ECoG signal recorded when subject 2 was reaching to the left. This figure was adapted from Popovic and Keller.18

Full size image

The neuroprosthesis had three accessible buttons (Buddy Button 57000 Series, AbleNet Inc., Roseville, MN, USA) that could be activated by subject 2 using the dorsal aspect of his left hand. Pressing buttons 1 or 2 elicited palmar grasp or lateral grasp, respectively. A second activation of either of these switches generated hand opening. Button 3 was used to turn the neuroprosthesis on and off. A 2-h training period allowed the user to become comfortable using the neuroprosthesis commanded with the three buttons.

Thereafter, the accessibility buttons were disconnected from the stimulator and reconnected to the ECoG classifier. With this modification, pressing buttons 1, 2 and 3 resulted in the random selection and classification of an ECoG signal recorded when subject 1 had performed wrist flexion, reaching to the right and reaching to the left movements, respectively. The classified trials excluded the recordings used to create the classifiers. After the classifier determined which ECoG signal had been extracted, it commanded the neuroprosthesis to perform the desired hand function or to turn the stimulator on/off. The correct classification of the ECoG signals resulted in the correct stimulation sequence delivered by the neuroprosthesis. A diagram depicting the complete implemented neuroprosthetic system is shown in Figure 4.

Figure 4
figure 4

Complete experimental setup. The user pressed one of three buttons to control the neuroprosthesis. Each button was associated with a dataset of ECoG signals recorded earlier. The system randomly extracted a single trial of the corresponding dataset, which was classified using a nearest neighbour classifier. The result of the classification process was then used to trigger a stimulation sequence. When the classification was successful, the correct stimulation sequence was delivered by the neuroprosthesis. Conversely, an incorrect classification would result in an incorrect action taken by the neuroprosthesis. ECoG, electrocorticographic; WF, wrist flexion; RTR, reaching to the right; RTL, reaching to the left.

Full size image

Subject 2 picked up 18 objects,19 shown in Table 2, which required palmar and lateral grasps. Each object was grasped and lifted from a table using the neuroprosthesis. The participant held the objects in pronation and supination and then released them. To test all of the available classes (ECoG signals), the user was asked to turn the neuroprosthesis off and on after grasping two objects consecutively.

Table 2 Objects used to test the neuroprosthesis
Full size table

Classification tests

The accuracy of classification (as the number of trials in which the system performed the action required by the user divided by the total number of activations of the neuroprosthesis) was measured. The time between the activation of a user switch and the issuing of the classification result was also recorded.

Results and discussion

We confirmed the presence of ECoG spectral components correlated with each one of the kinematic dimensions of the movements performed by subject 1 (Table 3). Figure 5 shows histograms created by grouping 20 spectral components identified as the most strongly correlated with kinematic recordings. The distribution of correlated frequencies was found to depend on the type of movement performed by subject 1 (P<0.001, χ2-statistic), suggesting that the recorded LFP activity is distinct for each arm movement performed.

Table 3 Correlation coefficients values between ECoG spectral components and kinematic recordings
Full size table
Figure 5
figure 5

Histograms resulting from grouping the 20 spectral components identified as the most strongly correlated with kinematic recordings. Each bin in the histogram corresponds to a frequency range of 10 Hz. Values are expressed as probabilities after dividing the magnitude of each column by the total number of spectral components used for each plot.20 The amplitude of each column represents the probability that a spectral component within the frequency range defined by the bin is correlated with movement. Each histogram was different and unique for each one of the movements performed by subject 1.

Full size image

Subject 2 was able to use the neuroprosthesis with an accuracy of 94.5%. Most of the incorrect classifications occurred when the system attempted to classify trials corresponding to wrist flexion, as shown by the confusion matrix provided in Table 4. A closer inspection of the kinematic recordings revealed that the wrist flexion motion was less consistent than the reaching movements. This was likely the cause of the misclassification. The average time elapsed between the ECoG classification and the activation of the neuroprosthesis was 1870±109 ms.

Table 4 Confusion matrix for the implemented classifier
Full size table

There are seven grasping styles and dozens of combinations, which can be generated using the current FES technology. However, providing a user interface to control these functions independently remains a challenge, regardless of the user's motor abilities. This is an unsolved problem in the FES field. A BMI capable of identifying multiple movements has the potential to command multiple grasps using a single interface.

In this work, we presented a BMI system that used ECoG signals to control a neuroprosthesis for grasping. Activation of the neuroprosthesis triggered an off-line classification process of a single ECoG trial. The result of this classification triggered specific electrical stimulation sequences to perform palmar and lateral grasps, as well as for turning the electrical stimulator on and off. We believe that the short time required to create the system (<60 min, including both the neuroprosthesis and BMI design) along with the small number of trials used to create the classifier (that is , five) using activity from only four contacts to identify three different movements performed with the same limb make this system unique.

The technology and procedures used in this study have a good record of stability and reliability in clinical applications resulting in an increased interest in the development of BMIs using ECoG signals. However, it is still necessary to verify the long-term performance of subdural electrodes in BMI applications.

This work allowed us to explore our ideas of the integration of BMI systems and FES, and run a true end-to-end system test on the use of ECoG signals to control a neuroprosthesis for grasping. Although several reports describe the control of robotic or virtual systems with brain activities, we selected this application because of its clinical prevalence; FES is the technology currently used to facilitate movement in persons with SCI. However, to conduct our tests, we could not justify the implantation of subdural electrodes or the instrumentation of an arm with a neuroprosthesis when these interventions were not required for medical reasons. Owing to these ethical challenges, the only possible solution was to use two subjects, which is why the ECoG signals used to trigger the neuroprosthesis were not recorded from the same individual instrumented with the neuroprosthetic system.

Although the movements identified from subject 1 were different from those produced by the neuroprosthesis in subject 2, we believe that this work brings us closer to a situation in which individuals will be able to elicit a movement in their paralyzed limbs by attempting or imagining that same movement. We feel confident that in future implementations, this discrepancy can be overcome and that the neuroprosthesis will be able to produce the exact movement identified from ECoG recordings. By doing this, the level of transparency of interaction between the user and a neuroprosthetic device will increase dramatically.

Voluntary movement-related changes in power in the β-band appear to show temporal differences20 in patients with ET. These differences may affect the correlation values on which the classification method is based. However, we have tested successfully the presented method in individuals with Parkinson's disease and ET, and we are confident that the method used will work in different patient populations.

Although our system operates on ECoG signals recorded while actual arm movements were performed, our immediate work will focus on developing a classifier that will be able to classify imagined and/or intended movements for controlling the neuroprosthesis. We also plan to develop a system capable of identifying specific arm movements from ECoG recordings in real time.