Abstract
Study design:
Proof of concept study to control a neuroprosthesis for grasping using identification of arm movements from ECoG signals.
Objective:
To test the feasibility of using electrocorticographic (ECoG) signals as a control method for a neuroprosthesis for grasping.
Setting:
Acute care hospital, Toronto Western Hospital and spinal cord injury (SCI) rehabilitation centre, Toronto Rehabilitation Institute, Lyndhurst Centre. Both hospitals are located in Toronto, Canada.
Methods:
Two subjects participated in this study. The first subject had subdural electrodes implanted on the motor cortex for the treatment of essential tremor (ET). ECoG signals were recorded while the subject performed specific arm movements. The second subject had a complete SCI at C6 level (ASIA B score) and was fitted with a neuroprosthesis, capable of identifying arm movements from ECoG signals off-line, for grasping. To operate the neuroprosthesis, subject 2 issued a command that would trigger the release of a randomly selected ECoG signal recorded from subject 1, associated with a particular arm movement. The neuroprosthesis identified which arm movement was performed at the time of recording and used that information to trigger the stimulation sequence. A correct ECoG classification resulted in the neuroprosthesis producing the correct hand function (that is grasp and release).
Results:
The neuroprosthesis classified ECoG signals correctly delivering the correct stimulation strategy with 94.5% accuracy.
Conclusions:
The feasibility of using ECoG signals as a control strategy for a neuroprosthesis for grasping was shown.
Similar content being viewed by others
Introduction
Functional electrical stimulation (FES) elicits muscle contraction using electrical impulses and is used as a motor neuroprosthesis to facilitate movement after spinal cord injury (SCI). The term ‘neuroprosthesis’ will be used in this study to refer to a motor neuroprosthesis.
Neuroprostheses are controlled using switches, linear variable resistors, joysticks, position sensors, electromyographic signals and speech.1 A more intuitive method for controlling neuroprosthetic devices would be to use brain activity. Brain–machine interfaces (BMIs) translate brain signals into control commands for electronic devices.
Electrical signals reflecting brain activity have been used most extensively for developing BMI systems. Non-invasive techniques for recording the electrical activity of the brain include EEG (electroencephalography). Invasive techniques allow the recording of local field potentials (LFPs) reflecting the activity of a group of neighbouring neurons. LFPs can be recorded using macroelectrodes placed on the surface of the brain, resulting in electrocorticographic (ECoG) signals, and using microelectrodes placed intracortically. These microelectrodes can also record the activity of individual neurons.
Operating an EEG-based BMI requires the user to change brain activities voluntarily. To do this, the user is often trained for up to several months. Therefore, an important challenge in the development of BMIs is minimizing the required training. For example, using ECoG signals, the training time can be reduced to the order of minutes.2
Spectral and temporal changes in brain activity elicited by voluntary movements have been used to reduce the training time of a BMI user. These changes have allowed the identification of specific movements and the detection and estimation of kinematic parameters of the motion performed.3
Single cell recordings in monkeys and in humans have yielded important results in the detection and estimation of kinematic parameters from brain activity making it possible to control computer cursors and robotic arms using the activity from a group of neurons.4, 5 However, there are concerns regarding the reliability and long-term stability of single neuronal recordings.6
Local field potentials, including ECoG recordings, offer alternatives to single neuron recordings. The changes in power of frequency components of LFPs reflect kinematic information of arm movements,7, 8, 9, 10 and recently it was possible to predict the hand position from ECoG signals.6
The convergence of the fields of neuroprosthetics and BMI seems to be the natural next step in the development of these two fields. Using BMI technology, a neuroprosthesis could detect the intention to perform a specific movement and deliver the electrical stimulation to produce that exact movement. The benefit of such a system would be enhanced further if the user required little or no training to use this device.
Upper limb neuroprostheses11, 12, 13, 14 have been controlled by individuals with quadriplegia using single cell recordings15 and EEG signals after training that lasted between 3 days and 4 months.12 To generate control commands, the EEG-based BMI users used motor imagery,11 closing and opening of the eyes13 and self-regulation of power in specific frequency bands.12 The accuracies reported for these systems range from 7613 to 94.2%.12
The purpose of this study was to test the feasibility of using ECoG signals as a control strategy for a neuroprosthesis for grasping. A neuroprosthetic device was created and controlled using ECoG signals acquired earlier. In the following section, it will be shown that an ECoG-driven neuroprosthesis can be implemented using a 4-channel ECoG electrode and minimal training time to achieve 94.5% accuracy.
Materials and methods
To test the integration of a BMI system and a neuroprosthesis for grasping, it is necessary either to (1) implant an individual with SCI with an ECoG electrode or (2) use ECoG signals recorded from another individual, already implanted with a subdural electrode, to control a neuroprosthesis instrumented on an individual with SCI. As we could not justify the implantation of the ECoG electrode solely for testing the system feasibility, we used the second approach.
Subjects
Two subjects participated in this study. Subject 1 was a 67-year-old woman implanted with subdural (ECoG) electrodes for the treatment of essential tremor (Figure 1a). She was recruited from the movement disorder clinic at the Toronto Western Hospital and gave informed consent to participate. The study was approved by the University Health Network Research Ethics Board.
Subject 2 was a 35-year-old man with a complete cervical SCI (C6 level/ASIA B) and had received 4 weeks of FES therapy treatment for restoring grasping function, as part of another study.16 He was able to use his arms and wrists but had no hand movement. He gave written and informed consent to participate in this study as required and approved by the Toronto Rehabilitation Institute Research Ethics Board. Figure 1b shows subject 2 wearing the neuroprosthesis for grasping.
We certify that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed-up during the course of this research.
Electrocorticographic and motion recordings
Electrocorticographic signals and arm movements were recorded simultaneously from subject 1 during a single 1-h session 3 days after the initial implantation of the subdural electrode. The subdural electrode had a single row of four platinum–iridium disc contacts (4 mm diameter and 10 mm centre-to-centre distance, RESUME 3586, Medtronic, Minneapolis, MN, USA). The electrode was implanted over the arm representation of the left motor cortex, which was confirmed intra-operatively using electrical stimulation delivered directly to the brain and observing contractions of the right upper limb.
Monopolar ECoG signals were band limited (0.5–500 Hz) and recorded (sampling rate=2 kHz, SynAmps2, Compumedics, Charlotte, NC, USA). The ECoG signals were downsampled to 200 samples per second and subtracted from each other to create differential signals using the recordings from non-adjacent electrodes (for example, contacts 3 and 1).17
Using only the right upper limb, subject 1 performed wrist flexion, reaching to the right and reaching to the left movements after an auditory cue and held the final position of the movement until a second auditory cue. A motion sensor was placed over the dorsal aspect of the third metacarpal of the right hand to record the movement using a six-dimensional (X, Y, Z, roll, yaw and pitch) motion-capture system (Fastrak, Polhemus Inc., Colchester, VT, USA). Only position recordings (X, Y and Z) were used for this study.17 Each movement was repeated at least 35 times, and each trial was visually inspected to identify mistrials, defined as: (1) a trial in which the individual had performed a movement different from what had been instructed; (2) the participant had started to move before the auditory cue; or (3) the movement was not completed. Table 1 and Figure 2 show the movements performed along with the number of trials used to carry out this study.
We created a nearest neighbour classifier (Matlab, Mathworks, Natick, MA, USA) using five trials of each motor task to identify the performed arm movements by analyzing the ECoG signals. The remaining trials were used to test the neuroprosthetic system.
To classify the subdural signals, we identified ECoG spectral components correlated with the kinematic components of the arm movement (Pearson correlation coefficient >0.1; statistics degrees of freedom=600). Time-resolved spectra were obtained using a spectrogram (128-sample Hamming window, 128-point fast Fourier transform and 127-point overlap). The 20 frequency components with the highest correlation coefficients were grouped using a histogram with bins representing frequency bands of 10 Hz. Details of this process can be found in Chin et al.17
Neuroprosthesis for grasping
The right hand of subject 2 was fitted with a neuroprosthesis for generating palmar and lateral (key pinch) grasps.18 The neuroprosthesis was designed and created specifically for this study using a Compex Motion four-channel transcutaneous electrical stimulator (Compex S.A., Lausanne, VD, Switzerland).
The grasping movements were achieved by stimulating: (1) flexor digitorum superficialis and flexor digitorum profundus using two electrodes connected in parallel with channel 1 (20 mA) for generating finger flexion; (2) flexor pollicis brevis using channel 2 (14 mA) for generating thumb opposition; and (3) extensor digitorum communis using channel 3 (22 mA) for generating hand opening. The Palmar grasp was obtained by stimulating channels 1 and 2, simultaneously. The lateral grasp was achieved by stimulating channel 1 followed by channel 2, 500 ms later. Stimulation of channel 3 generated hand opening. In all cases, the stimulation frequency was set at 40 Hz and the pulse duration was 300 μs. Figure 3 depicts the stimulation profiles to elicit the grasping synergies.
The neuroprosthesis had three accessible buttons (Buddy Button 57000 Series, AbleNet Inc., Roseville, MN, USA) that could be activated by subject 2 using the dorsal aspect of his left hand. Pressing buttons 1 or 2 elicited palmar grasp or lateral grasp, respectively. A second activation of either of these switches generated hand opening. Button 3 was used to turn the neuroprosthesis on and off. A 2-h training period allowed the user to become comfortable using the neuroprosthesis commanded with the three buttons.
Thereafter, the accessibility buttons were disconnected from the stimulator and reconnected to the ECoG classifier. With this modification, pressing buttons 1, 2 and 3 resulted in the random selection and classification of an ECoG signal recorded when subject 1 had performed wrist flexion, reaching to the right and reaching to the left movements, respectively. The classified trials excluded the recordings used to create the classifiers. After the classifier determined which ECoG signal had been extracted, it commanded the neuroprosthesis to perform the desired hand function or to turn the stimulator on/off. The correct classification of the ECoG signals resulted in the correct stimulation sequence delivered by the neuroprosthesis. A diagram depicting the complete implemented neuroprosthetic system is shown in Figure 4.
Subject 2 picked up 18 objects,19 shown in Table 2, which required palmar and lateral grasps. Each object was grasped and lifted from a table using the neuroprosthesis. The participant held the objects in pronation and supination and then released them. To test all of the available classes (ECoG signals), the user was asked to turn the neuroprosthesis off and on after grasping two objects consecutively.
Classification tests
The accuracy of classification (as the number of trials in which the system performed the action required by the user divided by the total number of activations of the neuroprosthesis) was measured. The time between the activation of a user switch and the issuing of the classification result was also recorded.
Results and discussion
We confirmed the presence of ECoG spectral components correlated with each one of the kinematic dimensions of the movements performed by subject 1 (Table 3). Figure 5 shows histograms created by grouping 20 spectral components identified as the most strongly correlated with kinematic recordings. The distribution of correlated frequencies was found to depend on the type of movement performed by subject 1 (P<0.001, χ2-statistic), suggesting that the recorded LFP activity is distinct for each arm movement performed.
Subject 2 was able to use the neuroprosthesis with an accuracy of 94.5%. Most of the incorrect classifications occurred when the system attempted to classify trials corresponding to wrist flexion, as shown by the confusion matrix provided in Table 4. A closer inspection of the kinematic recordings revealed that the wrist flexion motion was less consistent than the reaching movements. This was likely the cause of the misclassification. The average time elapsed between the ECoG classification and the activation of the neuroprosthesis was 1870±109 ms.
There are seven grasping styles and dozens of combinations, which can be generated using the current FES technology. However, providing a user interface to control these functions independently remains a challenge, regardless of the user's motor abilities. This is an unsolved problem in the FES field. A BMI capable of identifying multiple movements has the potential to command multiple grasps using a single interface.
In this work, we presented a BMI system that used ECoG signals to control a neuroprosthesis for grasping. Activation of the neuroprosthesis triggered an off-line classification process of a single ECoG trial. The result of this classification triggered specific electrical stimulation sequences to perform palmar and lateral grasps, as well as for turning the electrical stimulator on and off. We believe that the short time required to create the system (<60 min, including both the neuroprosthesis and BMI design) along with the small number of trials used to create the classifier (that is , five) using activity from only four contacts to identify three different movements performed with the same limb make this system unique.
The technology and procedures used in this study have a good record of stability and reliability in clinical applications resulting in an increased interest in the development of BMIs using ECoG signals. However, it is still necessary to verify the long-term performance of subdural electrodes in BMI applications.
This work allowed us to explore our ideas of the integration of BMI systems and FES, and run a true end-to-end system test on the use of ECoG signals to control a neuroprosthesis for grasping. Although several reports describe the control of robotic or virtual systems with brain activities, we selected this application because of its clinical prevalence; FES is the technology currently used to facilitate movement in persons with SCI. However, to conduct our tests, we could not justify the implantation of subdural electrodes or the instrumentation of an arm with a neuroprosthesis when these interventions were not required for medical reasons. Owing to these ethical challenges, the only possible solution was to use two subjects, which is why the ECoG signals used to trigger the neuroprosthesis were not recorded from the same individual instrumented with the neuroprosthetic system.
Although the movements identified from subject 1 were different from those produced by the neuroprosthesis in subject 2, we believe that this work brings us closer to a situation in which individuals will be able to elicit a movement in their paralyzed limbs by attempting or imagining that same movement. We feel confident that in future implementations, this discrepancy can be overcome and that the neuroprosthesis will be able to produce the exact movement identified from ECoG recordings. By doing this, the level of transparency of interaction between the user and a neuroprosthetic device will increase dramatically.
Voluntary movement-related changes in power in the β-band appear to show temporal differences20 in patients with ET. These differences may affect the correlation values on which the classification method is based. However, we have tested successfully the presented method in individuals with Parkinson's disease and ET, and we are confident that the method used will work in different patient populations.
Although our system operates on ECoG signals recorded while actual arm movements were performed, our immediate work will focus on developing a classifier that will be able to classify imagined and/or intended movements for controlling the neuroprosthesis. We also plan to develop a system capable of identifying specific arm movements from ECoG recordings in real time.
References
Popovic MR, Keller T, Pappas IP, Dietz V, Morari M . Surface-stimulation technology for grasping and walking neuroprosthesis. IEEE Eng Med Biol Mag 2001; 20: 82–93.
Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW . A brain-computer interface using electrocorticographic signals in humans. J Neural Eng 2004; 1: 63–71.
Pfurtscheller G, Neuper C, Birbaumer N . Human brain-computer interface. In: Riehle A, Vaadia E (eds). Motor Cortex in Voluntary Movements: A Distributed System for Distributed Functions. CRC Press: Boca Raton, 2005, pp 367–401.
Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 2006; 442: 164–171.
Carmena JM, Nicolelis MA . Advances in brain-machine interfaces. In: Riehle A, Vaadia E (eds). Motor Cortex in Voluntary Movements: A Distributed System for Distributed Functions. CRC Press: Boca Raton, 2005, pp 349–366.
Schalk G, Kubanek J, Miller KJ, Anderson NR, Leuthardt EC, Ojemann JG et al. Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J Neural Eng 2007; 4: 264–275.
Mehring C, Nawrot MP, de Oliveira SC, Vaadia E, Schulze-Bonhage A, Aertsen A et al. Comparing information about arm movement direction in single channels of local and epicortical field potentials from monkey and human motor cortex. J Physiol Paris 2004; 98: 498–506.
Rickert J, Oliveira SCd, Vaadia E, Aertsen A, Rotter S, Mehring C . Encoding of movement direction in different frequency ranges of motor cortical local field potentials. J Neurosci 2005; 25: 8815–8824.
Mehring C, Rickert J, Vaadia E, Oliveira SCd, Aertsen A, Rotter S . Inference of hand movements from local field potentials in monkey motor cortex. Nat Neurosci 2003; 6: 1253–1254.
Heldman DA, Wang W, Chan SS, Moran DW . Local field potential spectral tuning in motor cortex during reaching. IEEE Trans Neural Syst Rehabil Eng 2006; 14: 180–183.
Pfurtscheller G, Müller-Putz GR, Schlögl A, Scherer BGaR, Leeb R, Brunner C et al. 15 years of BCI research at Graz University of Technology: current projects. IEEE Trans Neural Syst Rehabil Eng 2006; 14: 205–210.
Lauer RT, Peckham PH, Kilgore KL . EEG-based control of a hand grasp neuroprosthesis. Neuroreport 1999; 10: 1767–1771.
Heasman JM, Scott TR, Kirkup L, Flynn RY, Vare VA, Gschwind CR . Control of a hand grasp neuroprosthesis using an electroencephalogram-triggered switch: demonstration of improvements in performance using wavepacket analysis. Med Biol Eng Comput 2002; 40: 588–593.
Pfurtscheller G, Neuper C . Chapter 28 Future prospects of ERD/ERS in the context of brain-computer interface (BCI) developments. Prog Brain Res 2006; 159: 433–437.
Pohlmeyer EA, Perreault EJ, Slutzky MW, Kilgore KL, Kirsch RF, Taylor DM et al. Use of intracortical recordings to control a hand neuroprosthesis. Third International IEEE EMBS Conference on Neural Engineering, Kohala Coast, Hawaii, USA, IEEE; 2007, 418–420.
Popovic MR, Thrasher TA, Adams ME, Takes V, Zivanovic V, Tonack MI . Functional electrical therapy: retraining grasping in spinal cord injury. Spinal Cord 2006; 44: 143–151.
Chin CM, Popovic MR, Thrasher A, Cameron T, Lozano A, Chen R . Identification of arm movements using correlation of electrocorticographic spectral components and kinematic recordings. J Neural Eng 2007; 4: 146–158.
Popovic MR, Keller T . Modular transcutaneous functional electrical stimulation system. Med Eng Phys 2005; 27: 81–92.
Popovic MR, Contway C . Rehabilitation engineering laboratory hand function test for functional electrical stimulation assisted grasping. Proceedings of the 8th International Functional Electrical Stimulation Society Conference 2003; 231–234.
Tamas G, Palvolgyi L, Takáts A, Szirmai I, Kamondi A . Delayed beta synchronization after movement of the more affected hand in essential tremor. Neurosci Lett 2006; 405: 246–251.
Acknowledgements
We would like to thank Ms Carolyn Gunraj and Dr Kei Masani for their assistance. This study was financially supported by the Toronto Rehab Student Scholarship Fund, Natural Sciences and Engineering Research Council of Canada (#480588), Canadian Fund for Innovation (#7313), Ontario Innovation Trust (#7313), Ontario Ministry of Health and Long-Term Care and the Canadian Institutes of Health Research.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Márquez-Chin, C., Popovic, M., Cameron, T. et al. Control of a neuroprosthesis for grasping using off-line classification of electrocorticographic signals: case study. Spinal Cord 47, 802–808 (2009). https://doi.org/10.1038/sc.2009.41
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/sc.2009.41
Keywords
This article is cited by
-
Decoding hand and wrist movement intention from chronic stroke survivors with hemiparesis using a user-friendly, wearable EMG-based neural interface
Journal of NeuroEngineering and Rehabilitation (2024)
-
Why brain-controlled neuroprosthetics matter: mechanisms underlying electrical stimulation of muscles and nerves in rehabilitation
BioMedical Engineering OnLine (2020)
-
Functional electrical stimulation therapy for restoration of motor function after spinal cord injury and stroke: a review
BioMedical Engineering OnLine (2020)