The intuitive control of upper-limb prostheses requires a man/machine interface that directly exploits biological signals. Here, we define and experimentally test an offline man/machine interface that takes advantage of the discharge timings of spinal motor neurons. The motor-neuron behaviour is identified by deconvolution of the electrical activity of muscles reinnervated by nerves of a missing limb in patients with amputation at the shoulder or humeral level. We mapped the series of motor-neuron discharges into control commands across multiple degrees of freedom via the offline application of direct proportional control, pattern recognition and musculoskeletal modelling. A series of experiments performed on six patients reveal that the man/machine interface has superior offline performance compared with conventional direct electromyographic control applied after targeted muscle innervation. The combination of surgical procedures, decoding and mapping into effective commands constitutes an interface with the output layers of the spinal cord circuitry that allows for the intuitive control of multiple degrees of freedom.
Natural and intuitive control of upper-limb prostheses requires the establishment of a man/machine interface that explores the perception–action cycle directly based on biological signals
Muscle interfacing is generally possible only when relevant remnant muscle tissue is available following the amputation. The higher the amputation level, the greater the need for control signals, with fewer muscles available to interface for intuitive control. Nonetheless, muscles can be denervated and reinnervated by nerves that used to carry the neural code to the missing limb 14 . This procedure is known as targeted muscle reinnervation (TMR) 2,15,16 and it constituted a breakthrough in prosthetics 17 . In TMR, muscles serve as biological amplifiers of nerve activity because of the association between the action potentials discharged by the efferent nerve fibres (axons of motor neurons) and the action potentials of the innervated muscle fibres. Each nerve action potential is transduced into a compound muscle fibre action potential that carries the same neural information, coded by its timing of occurrence.
TMR and electromyographic (EMG) recordings from reinnervated muscles determine a man/machine interface that, in principle, allows the indirect detection (from muscle signals) of the ensemble efferent activity of any nerve once it is directed to an accessible target muscle. However, the classic use of TMR and EMG for man/machine interfacing does not aim at decoding the underlying neural information (timings of occurrence of action potentials) sent to muscles by motor neurons. Rather, this interface uses the EMG as an interferent signal (coloured noise) from which few global features are extracted 18 , for example, amplitude or spectral moments, for either direct control or for identifying sets of predefined movements 19 . This global approach is common to previous TMR investigations, even when more advanced (multi-channel) EMG systems have been employed 19 . This procedure limits the decoding quality of the neural interface because the global EMG is a spatiotemporal summation of action potentials that creates correlations among multiple channels and therefore determines an ill-posed inverse problem for decoding 20 . Similarly, control methods based on more selective intramuscular EMG recordings have been so far based on global EMG analysis without decoding the contributions of individual motor neurons 21,22 or on a very small number of decoded motor neurons (for example, two experimentally decoded motor neurons from intramuscular EMG signals 23 ). Decoupling the neural information contained in the EMG signals, which exactly correspond to the timings of discharge of the efferent nerve fibres, from the shapes of the muscle fibre action potentials would determine a direct interface with the spinal motor neurons.
Here, we describe a neural interface that, following TMR 24,25 , extracts the sources of neural information—the discharge timings of motor neurons—through EMG deconvolution. The decoded neural information is then mapped into effective commands for intuitive prosthetic control. The effectiveness of the decoded neural information for potential prosthesis control is demonstrated offline by associating the activity of motor neurons innervating the missing limb of amputees with the kinematics of intended motor tasks. We experimentally validate this association for multiple degrees of freedom in six patients following TMR. Therefore, here we present and substantiate the concept of a man/machine interface that extracts neural information sent from the output spinal cord circuitries by a combination of surgical procedures, advanced neural decoding and mapping into multiple degrees of freedom.
The approach is based on recording multi-channel (>50 channels) EMG signals from reinnervation sites following TMR. The EMG signals are decoded using a blind source separation method that separates the timings of activation of each motor unit from the waveforms representing the muscle fibre action potentials. The decoding thus provides multiple series of discharge timings of motor neurons reinnervating the target muscles. The discharge timings are projected into degrees of freedom by various mapping methods. Figure 1 schematically presents the processing steps common to all of the analyses performed. Three main experimental tests were conducted on TMR patients (Table 1) to present and validate the decoding and mapping approach. The aim of each experiment was to highlight different characteristics of the proposed approach. Experiment 1 presents the classification of the motor-neuron activity into discrete classes. Experiment 2 shows proportional control based on motor-neuron discharges. Finally, experiment 3 presents simultaneous and proportional mapping over multiple degrees of freedom.
Experiment 1 (classification)
This experiment was performed on patients T1, T2 and T3 (Table 1) and was designed to compare the use of motor-neuron discharge timings with respect to classic global EMG features for the classification of intended tasks into a predefined set of classes (motor tasks). For this purpose, the EMG signals recorded from the reinnervation sites (six reinnervation sites for patient T1, four for T2 and five for T3; Table 1) were decomposed (decoded) into the contributions of individual motor units, separating the discharges of the innervating motor neurons from the waveforms of the muscle fibre action potentials. The discharge timings were then pooled to obtain series of ensemble discharge timings for the muscle regions where the recording EMG electrode grids were located (the area covered by each recording EMG grid was divided into four muscle regions). These series of ensemble discharges represented the neural drive sent to the corresponding muscle regions and were used as features for offline classification into 9, 7 and 11 classes (movements) for the three patients investigated.
The average number of motor units identified by decomposition of the multi-channel EMG signals in each reinnervation site during the executed tasks was 23.1 ± 11.2 (mean ± s.d., average over the three patients and all reinnervation sites and tasks).
There were no significant differences in the number of decoded motor units or in the accuracy of decoding between reinnervation sites, an observation that was valid for all experiments and all patients. This result is in agreement with several previous studies on EMG decoding in able-bodied individuals across a variety of muscles and contraction types 26 . The use of motor-neuron discharge timings for motion classification provided an almost perfect classification accuracy of >97% as an average over the three patients (Fig. 2). In the current experimental conditions, this performance was superior to both the use of the EMG root mean square (r.m.s.) (average accuracy of 71% across all patients) and the use of r.m.s. together with time domain features (average accuracy of 85%). The difference was particularly evident for patient T3, who could execute the greatest number of tasks (11 classes) and for whom the classification accuracy was only ~70% when using the classic EMG features and >95% when using the decoded motor-neuron discharge timings (Fig. 2). Figure 2 presents the confusion matrices for the classification.
Figure 3 highlights one of the reasons for the effective discrimination of motor tasks based on motor-neuron discharge timings with respect to global EMG features. The action potentials of three motor units are representatively shown as detected over the surface of the pectoralis muscle in subject T1 during hand closing and wrist supination. The spatial mapping of the global EMG amplitude and the action potentials of the motor units are presented. The global EMG amplitudes, which are shown as an amplitude map (Fig. 3a), are similar for the two tasks, and are therefore difficult to discriminate using this feature. The spatial distribution of amplitude is indeed localized in the same area, represented by the upper left corner of the spatial amplitude maps in Fig. 3a. Therefore, the two tasks are executed by a neural drive reaching similar regions of the reinnervated muscles. A direct control of these two tasks in a physiological way (that is, with natural muscle contractions corresponding to the two tasks) would not be possible because of this overlap. Classification of these two classes with global EMG features is, however, possible to some extent since classification uses features that may differ between the two tasks. Nonetheless, the similarity of the two EMG amplitude maps and the variability in execution over multiple trials reduces the classification performance when using global features. The spatiotemporal structure of the active motor units during the tasks is conversely very different and can be used for more robust discrimination.
This experiment showed that the extraction of the timings of discharge of motor neurons may be beneficial for discriminating tasks of the missing limb with respect to global EMG feature classification. In the proposed approach, contrary to the global EMG that represents an average activity over the skin surface, each motor neuron may provide discriminating information from its series of discharges as well as from the location of the innervated muscle fibres.
Experiment 2 (direct control).
This experiment (patients T4, T5 and T6) was designed to test the use of motor-neuron discharge timings for proportional control. This has implications in methods for direct control following TMR. The tests were done on a single degree of freedom which was mapped proportionally from global EMG amplitude and from motor-neuron discharge timings during slow-varying force contractions, in the full range of muscle activation. In this experiment, the average number of motor units identified by decomposition of the multi-channel EMG signals at the observed reinnervation site across all three patients was 11.8 ± 3.8. Figure 4 presents an example of slow linear increase and decrease in intensity of activation by patient T4. Decomposition of the generated signals qualitatively revealed the two physiological mechanisms for increasing the level of activation: recruitment of additional motor neurons and modulation of the discharge frequency of the active motor neurons (Fig. 4c). It was also evident that the information in the population activity of motor neurons was highly correlated to the intensity of muscle activity and this was repeatable over different trials of the same task (Fig. 4e).
The representative results shown in Fig. 4 were confirmed in the three tested patients. Figure 5 shows representative estimates of contraction intensity for one patient as well as the individual results for each patient. The motor-neuron discharge timings have been pooled to obtain the ensemble of discharge timings of all motor neurons, as an estimate of the neural drive. This estimate was compared with the global EMG amplitude. The estimates using motor-neuron discharge timings outperformed those obtained with surface EMG for a large range of processing intervals and for all patients (Fig. 5).
This experiment showed that it is possible to extract direct proportional commands from motor-neuron discharge timings and these commands are more accurate than those obtained with classic EMG amplitude.
Experiment 3 (control of multiple degrees of freedom)
Finally, a direct mapping between decoded motor-neuron discharge rates and kinematics was performed in experiment 3, for patient T6 only. The mapping into mechanical function was performed with two methods, either signal- or model-based. The signal-based approach applied a dimensionality reduction to the motor-neuron discharge timing series without requiring kinematic labelling. The model-based approach projected the series of discharge timings into degrees of freedom by a forward biomechanical estimation of joint moments.
The signal-based approach was applied to 244 unsorted extended sources (over all reinnervation sites) during a recording of 30 s. The sources were extended by the EMG decomposition approach (see Methods) and all extensions were maintained for this estimation approach, contrary to all other methods presented, for which the extended sources were not used. The extensions of the sources represent redundant information, but redundancy is exploited to define appropriate subspace projections with this approach. Due to the inclusion of extended sources for this approach, the number of sources used (244) was greater than for all other methods.
Figure 6 shows the analysis made on three trials of the same composite movement with three concurrently active degrees of freedom. The third trial was used for test purposes, after calibration in the first two trials. The estimate on the train and test trials in this representative example corresponded to an average R 2 in the estimation of kinematics of 0.87 and 0.73, respectively. It has to be noted that it is impossible to guarantee that the mirror movements were identical for the two sides and therefore the obtained R 2 is an underestimate of the actual performance. The performance obtained with three concurrently active degrees of freedom was similar to those obtained for two active degrees of freedom (with an average R 2 of 0.65 and 0.72, respectively) and for one degree of freedom (0.81 and 0.76). Figure 6 shows the quality of the matching.
The second mapping approach in this experiment was based on biomechanical modelling. Neural data-driven musculoskeletal modelling was used to demonstrate the possibility of reconstructing the neuro-mechanical function of the patient’s phantom limb. This approach enabled translating motor-neuron discharge timings, decoded from reinnervated residual muscles, into the forces simultaneously produced by 12 musculo-tendon units acting on a mechanical model of the amputee’s missing limb (Fig. 7). The forces of the muscle-tendon units were concurrently projected to three degrees of freedom of the missing limb (elbow flexion/extension, forearm pronation/supination, wrist flexion/extension). The similarity between joint moments in the intact (as reference) and missing limb was quantified using both R 2 and the root mean squared difference (r.m.s.d.). Figure 7 shows a representative example, which involved elbow flexion/extension (R 2 = 0.82, r.m.s.d. = 0.91 Nm), forearm rotation (R 2 = 0.77, r.m.s.d. = 0.04 Nm), and wrist flexion/extension (R 2 = 0.60, r.m.s.d. = 0.02 Nm), simultaneously. When decoding mechanical function for two degrees of freedom simultaneously—involving wrist rotation and flexion/extension—the performance coresponded to R 2 = 0.72, r.m.s.d. = 0.04 Nm and R 2 = 0.73, r.m.s.d. = 0.05 Nm, respectively. When only a single degree of freedom was estimated (elbow flexion/extension), the performance corresponded to R 2 = 0.79, r.m.s.d. = 1.24 Nm.
These results illustrate the possibility, in principle, of estimating the biomechanics of a missing limb by decoding the neural drive to the muscle units and therefore the ultimate neural code representing the motor tasks. With this approach, predicted limb biomechanics can be translated into mechatronic functions of external devices that can be as complex as an intact biological limb.
We have presented a new neural interface based on the decoding of the discharge timings of spinal motor neurons that provides the neural drive to muscles of missing limbs in amputees, to generate intuitive commands for potential prosthesis control. The interface is realized by the combination of TMR, which connects the axons of the target motor neurons to available muscle fibres, and the decoupling, by deconvolution, of the electrical activity of the muscle fibres innervated by each axon from multi-channel EMG recordings. We have shown that this interface enables the generation of meaningful and accurate control signals for potential prosthetic control, both in the context of pattern recognition with a large number of classes and within the paradigm of simultaneous and proportional control of multiple degrees of freedom.
The proposed interface allowed the detection of a large number of motor neurons (on average >10) for each reinnervation site. The series of discharges of these motor neurons could be accurately identified, so that the neural information sent from the pools of motor neurons previously innervating the missing muscles in the patients could be assessed. At a theoretical level, the decoded motor-neuron behaviour from EMG recordings may be considered a new EMG feature for use in myocontrol; however, we consider it as conceptually different from any previous attempt for myocontrol. The main breakthrough is the change of analysis scale from macroscopic (EMG) to microscopic (timing of motor-neuron discharges). Because the EMG is a signal generated by the spatiotemporal convolution of thousands of motor-neuron discharges, any set of mathematically derived EMG features will reflect the temporal and spatial resolution of the EMG as a random process. In contrast, motor-neuron discharge timings have a precise physiological meaning and represent the way in which movement is naturally coded at the spinal level. It is the same set of features that direct intrafascicular nerve interfacing attempts to extract 5 . Therefore, our results are on par with the identification of discharges directly from axonal interfacing with implanted intrafascicular electrodes. Nonetheless, with respect to implanted nerve recodings, the proposed system identifies a greater number of motor neurons and extracts the complete series of discharges of these motor neurons (see for comparison, ref. 27 where the error rate is approximately 30% with at most 5 or 6 decoded efferent fibres from nerve interfacing). In this view, the proposed approach constitutes the ultimate exploitation of TMR as a means for neural interfacing using muscles as biological amplifiers. The results presented show that direct access to the neural code sent by peripheral nerves to muscles is possible and practical following TMR. Moreover, this study also shows that motor-neuron spike trains can be effectively used as a relevant source of information for synthesizing complex prosthetic commands. In comparison, direct neural recordings have been usually applied to map commands using global features of the interference nerve signals without spike sorting (for example, ref. 28 ).
The accuracy of the proposed interface has been proven with various control approaches and experimental designs on patients with amputations at different levels. First, we applied methods for signal classification, which are dominant in the scientific literature on myocontrol 29 , to the decoded series of discharge timings. The classification results indicated an almost perfect discrimination (on average >97%) of up to 11 classes. The results obtained from global EMG analysis in this study are partly in agreement with those reported in previous studies (for example, refs 15,19,30 ). However, given the relatively small number of available TMR patients in the current and previous studies and differences in the experimental and analysis conditions, comparisons between studies are difficult. For example, using classic EMG features, a classification accuracy of 98% and 93% was previously reported for two patients and eight classes 30 , which is a similar result as obtained in the present study for patients T1 and T2 for 7 and 9 classes, respectively. In our experiment, however, the performance with classic EMG features degraded for patient T3 when classifying 11 classes and was lower than that previously reported in four patients for 16 classes 19 . However, three of the four patients in this previous study were transhumeral amputees with physiological innervation of the lateral heads of biceps and triceps and the window duration used for the analysis (256 ms) was longer than in our study (100 ms). In the experimental conditions of our study and with the choices we have made for the processing parameters, the proposed approach based on decoding motor neurons led to greater accuracy than the classic pattern recognition approach. Given the relatively small number of patients in this and previous studies, however, this result cannot be generalized.
Furthermore, we tested the possibility for the patients to proportionally control one degree of freedom through decoded motor-neuron activity (direct control). The decoded neural information allowed a finer separation of intensity levels than surface EMG amplitude. Surface EMG has indeed an associated estimation variance that is limited by the bandwidth of the signal 31 , while the extracted neural information can in principle exactly predict force, because force is generated as a direct linear transformation of the decoded neural activity 32 . Accordingly, the variability in force control shown in Fig. 5 was lower when using the motor-neuron discharge timings than with EMG amplitude in almost all conditions and patients tested. These results indicate the possibility of superior precision in proportional control using the discharge patterns of motor neurons with respect to EMG amplitude.
The final experiment was related to the simultaneous and proportional control of multiple degrees of freedom, shown for two approaches. The signal-based approach identifies the control signals without labelling of the kinematics during training, whereas the model-based approach is based on training and relies on the subject’s anatomy. Both methods provided an estimate of the mechanics of multiple degrees of freedom. The TMR procedure allows the full reconstruction of the neural signals sent to the muscles of the missing limb, despite the absence of these muscles, so that the biomechanics of the missing limb can be estimated, in principle, accurately. This experiment was performed on only one of the patients, due to limited availability of the other patients. However, the patient who participated in this test did not show EMG signals of better quality than the other patients and, importantly, the decomposition was of the same quality in this and other patients. Moreover, the patients who participated in the second experiment, including the one who took part in the third experiment, had very similar performance in their motor neuron proportional control (Fig. 5). Therefore, the patient selected for the third experiment on simultaneous and proportional control was similar to the others for the type and quality of neural information extracted and for the control he could achieve using this neural information in a previous experiment. There is no reason to believe that the results of the third experiment are specific to the patient selected.
The superior performance of the use of motor-neuron discharge timings with respect to global EMG in some of the conditions analysed has several biophysical reasons. The global EMG features vary with changes in the waveforms of muscle fibre action potentials. This factor of variability is removed by separating the neural information from the muscle fibre action potentials. Moreover, the decoding method does not use clustering based on the shape of the action potentials but rather identifies the discharge timings by sparseness constraints (see Methods). Therefore, changes in the action potential shapes do not influence the decomposition. The separation of action potentials from the neural drive also eliminates the effect of EMG amplitude cancellation 31 , which poses an intrinsic limit to the accuracy of EMG amplitude estimates. This partly explains, for example, the observations of the second experiment. In the specific clinical case of TMR patients, there are additional reasons for the effectiveness of the proposed approach with respect to global EMG analysis. The discharge patterns of reinnervated motor units are probably easier to discriminate than for physiological innervation due to a reduced complexity of the interference signal. Moreover, the reinnervation causes a loss of the natural relation between muscle unit size and recruitment order so that the amplitude of the EMG signal is probably a poorer indicator of muscle activation in these patients than in physiological conditions, as suggested in ref. 23 . Finally, the territories of the muscle units of different motor-neuron pools tends to overlap during the reinnervation process and their activities are therefore difficult to differentiate from the global EMG analysis.
The proposed interface can presumably be applied to any nerve that can be redirected to accessory muscles. Even in the absence of target muscle tissue, this may be obtained either with transplants of small muscle portions 33 or, in the future, by growing muscle tissue around the terminal portion of the nerve 34 . Therefore, the proposed system should be seen as a general neural interface that accurately decodes the efferent activity of nerves without inserting electrodes into nerves. Moreover, although the current system was tested with non-invasive EMG recordings, the same concepts and algorithms can be directly translated to implanted multi-site muscle electrodes, such as the intramuscular arrays we have recently proposed 35 or multi-channel extensions of epimysial devices already tested in patients 36 . These muscle implants could be surgically realized at the same time as the TMR procedure.
Although obtained under several experimental conditions and on patients with different characteristics, the presented results are currently limited to an offline analysis. The training and use of the proposed interface needs to be performed online and should explore the perception–action cycle of the user in a closed-loop scenario to fine tune motor control to achieve the desired movements. Although these future steps are needed for a full appreciation of the potential of the approach, the current work provides proof of the feasibility of decoding motor-neuron behaviour in amputees and of using motor-neuron behaviour for man/machine interfacing.
The online implementation of the proposed concepts for the control of prostheses in daily-life activities requires the solution of practical challenges. The first of these challenges is the requirement for an online EMG deconvolution that is computationally complex. Nonetheless, this operation is now feasible due to the speed of current microprocessors 37 . The performance with the online user’s control may differ from the offline results 38 . However, because motor neurons acting on a degree of freedom receive predominantly common input 39 , the identification of different sets of motor neurons in different sessions would not influence the mapping and control. Despite these promising features, the performance of the system over different conditions without re-training during online use has not been tested and constitutes a challenge for the clinical translation. Moreover, the clinical efficacy of any of the control schemes proposed will depend on the design of the mechatronics of a given prosthetic limb. The current study focuses on an extensive offline experimental validation on patients that provides a proof-of-concept for an innovation in myocontrol whose clinical translation needs further efforts.
In conclusion, we have proven the possibility of decoding the behaviour of virtually all the pools of motor neurons that physiologically innervated the muscles responsible for the movement of a missing limb and that are reinnervated to other muscle tissues by TMR. This decoding was demonstrated in six patients with different amputation levels and TMR procedures, and was shown to provide information that can be used for intuitive and effective commands over multiple degrees of freedom. The approach is the ultimate exploitation of the TMR concept that allows spinal interfacing using muscles as natural amplifiers of nerve activity. A full clinical translation of this new concept requires online implementation of the proposed algorithms and testing the long-term adaptation with the user in the loop.
Measurements were performed on six TMR patients whose characteristics relevant for this study are reported in Table 1. The patients were referred to O.C.A. for prosthetic fitting and underwent TMR surgery. The characteristics of the patients reported in Table 1 refer to the time when each patient was included in the experimental measurements. The TMR procedures for all patients were performed at the Medical University of Vienna, Austria. In patients T1, T2 and T3, all major nerves of the brachial plexus were redirected into muscles of the chest region (Table 1). Patients T4, T5 and T6 were transhumeral amputees for whom the TMR procedure resulted in the medianus, ulnaris and radialis nerves being reinnervated into the brachialis, caput breve bicipitis and caput laterale tricipitis muscles, respectively.
The experimental protocols as well as the informed consent forms for the experiments were approved by the ethics committee ‘Ethikkommission der Medizinischen Universität Wien’ (approval numbers 1279/2014, for patients T1–T3, and 1234/2015, for patients T4–T6).
Three experiments were performed with the aim of showing different control strategies for active prostheses; all were based on the decoded activity of the motor neurons reinnervating muscles above the amputation. The patient selection for each experiment was not randomized since the measurements were performed over the course of two years and designed with the purpose of presenting progressively more advanced features of the proposed interface. The six patients participated in the different experiments depending on their availability at the time when the methods applied in the three experiments were developed. Therefore, the experimental work represents a case series of several experimental sessions on a total of six patients with the purpose of presenting the concept of spinal interfacing via TMR and motor-neuron activity decoding. Figure 1 shows the general concept underlying the methods applied in all experiments. Grids of 64 electrodes for surface EMG recordings were located over each reinnervation point. The multi-channel EMG signals were processed to extract the series of discharges of the innervating motor neurons 24,40 . The identified spike trains were then used to define control commands. The mapping between neural information and motion of degrees of freedom varied in the three experiments, as described in the following, with the aim of showing different control solutions.
EMG recording and processing
In all of the experiments, surface EMG signals were recorded with high-density and flexible electrode grids. The placement of the grids differed across individuals and corresponded to the locations of reinnervation following surgery. The EMG electrode grids incorporate copper tracks on a kapton support and each grid comprises 64 sensors (8 × 8 electrodes, 1 mm diameter; Spes Medica) with an inter-electrode distance of 10 mm in both directions. The grids were applied on the skin surface by a 1-mm-thick double adhesive foam with holes corresponding to the electrode locations. The skin–electrode contact was facilitated by the use of conductive paste. The multi-channel signals were amplified by a multi-channel amplifier (EMGUSB2, OTBioelettronica; cutoff frequencies 3–900 Hz), sampled at 2,048 Hz and A/D converted with 12-bit precision. Although a surface recording system was chosen for this study, the same processing methods and experimental tests can be also applied with implanted muscle electrodes 40 . The only fundamental characteristics of the EMG recording for the proposed approach is the availability of several (>30) channels per reinnervated site, as it can be achieved with invasive technology by, for example, our recently developed thin-film electrodes 35 .
The EMG signals were decomposed by a blind source separation algorithm 40 . The decomposition provided the discharges of activation of the innervating motor neurons. The decomposition algorithm is described in detail in ref. 40 , but we also provide a brief description of its basic working principles here.
Multi-channel EMG signals are observations that contain the convolutive mixtures of motor-neuron spike trains. For each observation (EMG channel), unknown finite impulse response filters (motor unit action potentials) act on the sources (series of motor-neuron discharge timings). This convolutive mixture of sources can be converted into a linear instantaneous mixture by extending the sources to include the n original sources as well as their delayed versions, with delays from 1 to the filter length L (ref. 40 ). The m original observations (EMG channels) are also extended, by a factor R, to maintain a greater number of observations than sources.
For each time sample (k), indicating the original sources, observations and noise, respectively, with (k) = [s 1(k),s 2(k) ,…, s n (k)] T , (k) = [x 1(k), x 2(k), …, x m (k)] T and (k) =(n 1(k), n 2(k), …, n n (k)) T , where T is the transpose operator, the extended model is as follows: with the extended sources, observations and noise as (k) = [ (k), (k) ,…, (k)] T , (k) = [ (k), (k), …, (k)] T and (k) =[ (k), (k), …, (k)] T respectively, with:
and the equivalent scalar mixing matrix representing the filters is:
where D r is the duration of the recordings and h ij the action potential of the jth motor unit recorded at the channel i. The noise in model (1) represents electronic noise as well as the activity of motor units represented at the skin surface by low-energy action potentials that are not separated.
The linear instantaneous model of equation (1) is inverted to recover the matrix of the extended sources. In this study, the inversion is performed by spatial whitening followed by the fixed point optimization procedure with a cost function that maximizes the sparseness of the estimated sources 40,41 . The method extracts sources (series of motor-neuron discharge timings) associated with individual motor neurons, as proven by the unique representation of the associated multi-channel surface action potentials 42 . The estimated sources are trains of delta functions centred at the instant of motor-neuron activation, with an amplitude that may vary due to the estimation process. To extract the discharge timings information only, a local peak detector was applied to the estimated sources, by comparing each candidate peak with surrounding peaks and considering a refractory period of 10 ms. It is relevant to note that the decomposition method extracts the original sources as well as their delayed replicas, as defined by the extended source model of equation (3). For all analyses performed in this study, the delayed replicas were excluded and only the estimated original sources were further processed for the mapping procedures. The only exception is the signal-based mapping for simultaneous and proportional control of multiple degrees of freedom based on principal component analysis (PCA) (experiment 3), for which we used the full extended estimated sources (as in equation (3)) as input. For this reason, the number of time series of discharge timings used as input with that approach is much greater than for the others (L times greater).
Applying the above decomposition procedure, Fig. 1 shows an example of sets of discharge times of populations of motor neurons during tasks of wrist flexion and extension by a patient following TMR. These discharge timings are not abstract mathematical features, as in classic myocontrol, but instead correspond to the natural neural code sent from the spinal cord to the muscles.
After EMG decomposition, the neural drive to muscles was estimated by pooling the discharge timings of groups of identified motor neurons 39,43 . The selection of these groups depended on the analysis and is detailed for each experiment in the following. For each group of motor neurons, the corresponding neural drive was the ensemble of discharge timings of the motor neurons. The signal of delta functions cantered at each discharge of activation of each motor neuron in a defined group of neurons represents all timings of discharge of the motor neurons in the group. It is simply obtained by summing the series of discharge timings of the motor neurons in the group and corresponds physiologically to the neural drive sent to the reinnervated muscles by the defined motor-neuron group. The estimated neural drives to the reinnervated muscles were then used to map control signals into multiple degrees of freedom in the three experiments. The common procedures used in all experiments for EMG processing to extract the neural activation signals are schematically shown in Fig. 1.
Experiment 1 (classification)
Patients T1–T3 (Table 1) participated in experiment 1. The patients were asked to attempt the following tasks of their missing limb: hand opening, hand closing, wrist extension, wrist flexion, thumb adduction, thumb abduction, pronation, supination, elbow extension and elbow flexion. Due to different reinnervation profiles and levels of training, not all subjects were able to perform all these tasks. The order of the attempts was randomized. For patient T1 each attempt lasted 10 s, for patients T2 and T3 the trials lasted 5 s, with 5 s of rest between trials. Patients T1 and T2 performed each trial twice, while patient T3 repeated each trial three times. These differences in experimental choices for the different patients were due to the conditions and capabilities of the patients.
The neural drives to muscle regions were estimated, as described above, by pooling the discharge timings of groups of motor neurons. The groups were defined based on the location of the muscle units in the areas covered by the recording electrode grids. Each grid was divided into four regions and each motor neuron was associated with the muscle unit in the region of the grid where the corresponding motor unit action potential had the greatest amplitude. In this way, we defined the neural drives to reinnervated muscle sites by partitioning each grid into four parts and associating them with the ensembles of discharges of the innervating motor neurons.
Classification into the discrete classes was performed using a support vector machine classifier with linear kernel, with fivefold cross-validation. The input to the classifier was the number of discharges in each 100 ms interval for each muscle region, with a 10 ms overlap between consecutive intervals. For comparison with surface EMG classification, the EMG r.m.s. of all channels as well as the time domain features were computed in the same intervals used for spike classification and classified with the same classifier. For global EMG features, the feature space was reduced in dimensionality using PCA, retaining 95% of the signal power.
Experiment 2 (direct control).
The aim of experiment 2 was to establish if it was possible to extract a proportional command from motor unit discharges (direct control). The experiment was performed on patients T4, T5 and T6, who were all transhumeral amputees (Table 1). The patients were seated comfortably facing a computer screen. One surface EMG electrode grid was mounted over the reinnervated short head of the biceps. Initially, all participants were asked to perform a maximal voluntary contraction by attempting a hand-open gesture of their missing limb. The maximum EMG envelope during this task was taken as reference for providing feedback in percent of the maximum intensity. Each subject was then prompted to increase and decrease the intensity of muscle activity from the relaxed state to the maximal intensity, over 20 s. This contraction duration determined a slow-varying force contraction. This choice was made to assess accurate control, which requires slow variations, and in the full activation range of the muscle, to prove that the analysis methods proposed are not influenced by the strength of activation. The contractions performed by the patients represent accurate force control, as it can be seen in fine object manipulation. Conversely, experiment 3 focused on faster contractions, typical of reaching tasks.
The intensity of muscle activation was estimated using the EMG envelope, as a classic reference approach, and the neural drive, as direct neural information. These approaches were compared varying the processing interval from 50 to 500 ms, with an interval overlap of 50%. Moreover, a post processing was applied by averaging over three consecutive past intervals. For each condition, the standard deviation of the intensity estimate, after linear detrending, was computed to determine the accuracy in the control.
Experiment 3 (control of multiple degrees of freedom).
Among the tested patients, patient T6 (Table 1) volunteered for a further experimental session that involved attempted mirror movements of the missing and the contralateral arm with the purpose of testing the ability of the proposed approach to map the neural drive into multiple degrees of freedom activated concurrently during natural movements. For this purpose, motion capture data and EMG signals were recorded concurrently. Upper-limb kinematics were recorded (256 Hz sampling frequency) using a seven-camera system (Qualisys) and a set of 18 retro-reflective markers placed on the patient’s intact left upper extremity, residual right upper extremity, trunk and pelvis. High-density EMG was recorded using electrode grids located in correspondence of the residual upper arm frontal, lateral and dorsal compartments to cover the targeted reinnervations. Data were recorded during 1 static anatomical pose and 13 dynamic trials involving mirrored bilateral motions that simultaneously articulated elbow flexion, forearm pronation and wrist flexion, both in the intact and missing limb. These three degrees of freedom were chosen because in transhumeral amputees the precise proportional and simultaneous control over elbow and wrist is the biggest challenge in prosthetic fitting. Precise proportional and simultaneous control over these three degrees of freedom allows natural object reaching for transhumeral amputees, which is not possible with current systems. The hand opening–closing task was not included in this experiment since this can be achieved without simultaneous control, by a hybrid scheme 7 , whereas the focus of this experiment was on concurrent and proportional activation of degrees of freedom. The estimate of the kinematics from neural information was performed with both a signal-based and a model-based estimation approach.
An approach for extracting control signals using unsupervised subspace mapping was developed, which receives as input the full extended sources (equation (3)) and transforms them in direct control signals. The central idea of the method is to preserve the metric of the high-dimensional space where the multidimensional time series of motor-neuron discharges exists and to project the data to a subspace of dimension specified by the degrees of freedom of the prosthetic hand. The simplest of the metric projections uses the Mahalanobis distance 36 , which is defined by the covariance of discharge time series. PCA was used to explore the spatiotemporal correlation between the time series in high-dimensional space and to project the data to an orthogonal lower dimensional space. Let X n×D , be the input data matrix representing the n discharge time series over D channels. Then PCA performs a singular value decomposition (SVD) of X = UΣV T , where U and V are n × n and D × D unitary matrices, respectively, and Σ is a n × D rectangular diagonal matrix with diagonal values σ i known as the singular values of X. The columns of V are the eigenvectors of the covariance matrix and they serve as the principal directions or axes of the PCA sub-space. The principal components UΣ are projections of X on sub-space axes. PCA achieves a projection that maps most of the relevant information (variance) to a d-dimensional manifold, where d(d ≪ D) is any subspace. It can be shown that this mapping is formed by the first d eigenvectors, that is, columns of V, noted as V d . In our case, d is specified by the number of degrees of freedom to be controlled and the goal is to associate each principal component with a degree of freedom. Therefore, the problem reduces to appropriately assigning the outputs of the PCA projected data to the corresponding degree of freedom. However, the two orthogonal coordinate systems (that of PCA and that of the degrees of freedom) of the same dimension are not necessarily the same because the PCA eigenvectors are solely defined by the data, hence an orthogonal rotation R d×d is needed to align the principal directions with the external basis of degrees of freedom. Moreover, the external basis coordinates are the canonical sparse basis of R n (each component only moves one degree of freedom of the prosthesis), while the PCA eigenvectors are not sparse and are ordered by projected variance. Therefore, we sought to find the rotation matrix in the principal component subspace that is the most sparsified to mimic the canonical basis of R n . Here, we applied the VARIMAX orthogonal rotation 37,38 R VARIMAX: where are referred to as the loadings 37 . Once the PCA and VARIMAX rotation are obtained in a training set (without any kinematic labelling), the control of the prosthesis test is achieved by inputting the motor-neuron discharge time series in real time to the PCA subspace and VARIMAX projections, which can be combined to decrease the computational complexity.
The open-source software OpenSim 44 was used to scale a generic upper extremity model of the musculoskeletal geometry 45 to match the patient’s anthropometry. The musculoskeletal geometry model had seven upper extremity degrees of freedom and incorporated a total of 14 muscle–tendon units, spanning the shoulder, elbow, wrist and hand joints (Fig. 7). During the scaling process, virtual markers were placed on the generic musculoskeletal geometry model based on the position of the experimental markers from the static pose. The model anthropomorphic properties as well as the muscle–tendon unit insertion, origin and their bone wrapping points were linearly scaled on the basis of the relative distances between experimental and corresponding virtual markers 44 . Inverse kinematics was solved for three-dimensional joint angles that minimized the least-squared error between experimental and virtual marker locations during dynamic trials 46 . The generated kinematics were then used to obtain dynamically consistent joint moments via residual reduction analysis; that is, joint moments reconstructing experimental joint angles when driving forward dynamic arm simulations 47 . We refer to these as ‘experimental joint moments’.
The estimates of joint moments were based on neural data-driven musculoskeletal modelling. The motor-neuron discharges were converted into continuous neural activations using a twitch model based on a time-history dependent recursive filter and a nonlinear transfer function 48 . Experimental joint angles were used as input to a multidimensional cubic B-splines set that synthesized the OpenSim subject-specific geometry of muscle–tendon units and computed their resulting length and moment arms 49,50 . Neural activations and muscle–tendon unit length were used to control a Hill-type muscle model and estimate instantaneous length, contraction velocity and force in the muscle fibres, and strain and force in the series-elastic tendon within each muscle–tendon unit 49 . The computed forces were projected onto all upper extremity degrees of freedom simultaneously via the moment arms.
The neural-driven model was calibrated to map neural activations to individual muscle–tendon units. After the calibration, the model was used to convert neural inputs into the resulting joint moments produced in the missing limb elbow, forearm and wrist.
The codes used for the results presented in this study are available from the corresponding author on reasonable request.
The authors declare that all data supporting the findings of this study are available within the paper.
How to cite this article: Farina, D. et al. Man/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation. Nat. Biomed. Eng. 1, 0025 (2017).
This work was supported by the European Research Council Advanced Grant DEMOVE (contract #267888) (to D.F.), the Christian Doppler Research Foundation of the Austrian Federal Ministry of Science, Research and Economy (to O.C.A.), the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement number 702491 (NeuralCon) (to F.N.) and Defense Advanced Research Projects Agency (DARPA N66001-15-1-4054) (to J.P.). The authors are grateful to M. Schweisfurth and H. Rehbaum for support in the experimental measurements, M. Castronovo for support in the data analysis, and C. Hofer and S. Salminger for clinical support.