Abstract
Signals derived from the rat motor cortex can be used for controlling one-dimensional movements of a robot arm1. It remains unknown, however, whether real-time processing of cortical signals can be employed to reproduce, in a robotic device, the kind of complex arm movements used by primates to reach objects in space. Here we recorded the simultaneous activity of large populations of neurons, distributed in the premotor, primary motor and posterior parietal cortical areas, as non-human primates performed two distinct motor tasks. Accurate real-time predictions of one- and three-dimensional arm movement trajectories were obtained by applying both linear and nonlinear algorithms to cortical neuronal ensemble activity recorded from each animal. In addition, cortically derived signals were successfully used for real-time control of robotic devices, both locally and through the Internet. These results suggest that long-term control of complex prosthetic robot arm movements can be achieved by simple real-time transformations of neuronal population signals derived from multiple cortical areas in primates.
Several interconnected cortical areas in the frontal and parietal lobes are involved in the selection of motor commands for producing reaching movements in primates2, 3, 4, 5, 6, 7, 8. The involvement of these areas in many aspects of motor control has been documented extensively by serial single-neuron recordings of primate behaviour2, 3, 8, 9, and evidence for distributed representations of motor information has been found in most of these studies10, 11, 12, 13, but little is known about how these cortical areas collectively influence the generation of arm movements in real time. The advent of multi-site neural ensemble recordings in primates14 has allowed simultaneous monitoring of the activity of large populations of neurons, distributed across multiple cortical areas, as animals are trained in motor tasks15. Here we used this technique to investigate whether real-time transformations of signals generated by populations of single cortical neurons can be used to mimic in a robotic device the complex arm movements used by primates to reach for objects in space.
Microwire arrays were implanted in multiple cortical areas of two owl monkeys (Aotus trivirgatus)14, 15, 16. In the first monkey, 96 microwires were implanted in the left dorsal premotor cortex (PMd, 16 wires), left primary motor cortex (MI, 16 wires)17, 18, left posterior parietal cortex (PP, 16 wires), right PMd and MI (32 wires), and right PP cortex (16 wires). In the second monkey, 32 microwires were implanted in the left PMd (16 wires) and in the left MI (16 wires). Recordings of cortical neural ensembles began 1–2 weeks after the implantation surgery and continued for 12 months in monkey 1, and 24 months in monkey 2. During this period, the monkeys were trained in two distinct motor tasks. In task 1, animals made one-dimensional hand movements to displace a manipulandum in one of two directions (left or right) following a visual cue. In task 2, the monkeys made three-dimensional hand movements to reach for small pieces of food randomly placed at four different positions on a tray. Cortical recordings were obtained while the two subjects were trained and tested on both tasks ( Fig. 1a).
Figure 1: Experimental design.
a, Schematic diagram depicting the experimental apparatus employed to use simultaneously recorded cortical ensemble data from primate behaviour to control the movements of local and remote robotic devices. In this design, both linear and ANN models are continuously updated during the recording session. Control of a local device is achieved through a local area network (LAN), while remote control requires transmission of brain-derived signals through the Internet. b, c, Simultaneously recorded neuronal activity in five cortical areas in monkey 1 (b) and two cortical areas in monkey 2 (c) during the execution of 1D movements. PP, posterior parietal cortex; M1, primary motor cortex; PMd, dorsal premotor cortex; iM1/PMd, ipsilateral M1 and PMd.
High resolution image and legend (58K)Figure 1b and c illustrate samples of the raw neuronal data obtained while the animals performed task 1. In both monkeys, coherence analysis19, 20, 21revealed that the activity of most single neurons from each of the simultaneously recorded cortical areas was significantly correlated with both one-dimensional (Fig. 2a and b) and three-dimensional hand trajectories, although the degree and frequency range of these correlations varied considerably within and between cortical areas. We then investigated whether both linear19, 20, 21 and artificial neural network (ANN)22, 23 algorithms could be used to predict hand position in real time. For one-dimensional movements, we observed that both algorithms yielded highly significant real-time predictions in both monkeys ( Fig. 2c and d). These results were obtained in spite of the fact that the trajectories were quite complex, involving different starting positions, as well as movements at different velocities. For example, in the session represented in Fig. 2c, the activity of 27 PMd, 26 MI, 28 PP, and 19 ipsilateral MI/PMd neurons in monkey 1 allowed us to achieve an average correlation coefficient of 0.61 between the observed and predicted hand position (60-minute session, range 0.50–0.71, linear model; 0.45–0.73, ANN; P <0.00119, 20, 21). Figure 2d illustrates similar real-time results obtained by using a smaller sample of neurons (8 PMd and 27 MI) in monkey 2 (average r = 0.72, range 0.47–0.79, linear model; average r = 0.66, range 0.42–0.71, ANN, P < 0.001). No large differences in fitting accuracy were observed between linear and ANN algorithms in either animal (Fig. 2c and d, linear prediction shown as green line, ANN as red line). As shown in Fig. 2e (monkey 1) and Fig. 2f (monkey 2), the performance of both algorithms improved in the first few minutes of recordings and then reached an asymptotic level that was maintained throughout the experiment. In both monkeys, highly significant predictions of hand movement trajectories were obtained for several months.
Figure 2: Real-time control of 1D hand movements.
a, b, Coherence analysis reveals that most cortical neurons are significantly coupled with different frequency components of the movements. c, d, Observed (black) and real-time predicted 1D hand movements using linear (red) and ANN (blue) models in monkey 1 (c) and 2 ( d). e, f, Correlation coefficient variation for predicted hand movements, using linear (red) and ANN (black) models, in one recording session in monkey 1 (e) and 2 (f). g, Real-time 1D movements of a local (blue) and remote (red) robot arm obtained in monkey 1 by using the linear model.
High resolution image and legend (70K)To reduce the influence of dynamic changes in the coupling between neuronal activity and movements and other non-stationary influences in our real-time predictions, both linear and ANN models were continuously updated throughout the recording sessions. This approach significantly improved the prediction of hand trajectories. For example, when predicting the last 10 minutes of 50–100-minute sessions, the adaptive algorithm performed 55% (20 sessions, median) better than a fixed model based on the initial 10 minutes, or 20% better than a model based on the 30–40-minute segment of the session.
Because accurate hand trajectory predictions were achieved early on in each recording session and remained stable for long periods of time, we were able to use brain-derived signals to control the movements of robotic devices (Phantom, SensAble Technologies)24 in real time ( Fig. 2g). In addition, we were able to broadcast these motor control signals to multiple computer clients by using a regular Internet communication protocol (TCP/IP, Fig 1a) and control two distinct robots simultaneously: one at Duke University (Fig. 2g, blue line) and one at MIT (Fig. 2g, red line).
Next, we investigated whether the same cortical ensemble activity and models could be used to predict the complex sequences of three-dimensional hand movements used by primates in a food-reaching task (task 2). These movements involved four phases: reaching for the food, grasping the food, bringing the food to the mouth and returning to the start position (Fig. 3a and b). Because these animals were not overtrained, their movement trajectories were highly variable. For example, in the session represented in Fig. 3a (monkey 1) the dispersion of hand trajectories was 7.0 by 7.5 by 6.0 cm (or 315 cm 3); in Fig. 3b (monkey 2) the dispersion was even bigger, 11.5 by 10.5 by 10.0 cm (or 1,207.5 cm3 ). Nonetheless, in both animals, the same linear and ANN models described above provided accurate predictions of three-dimensional hand trajectories in four different directions during 25–60-minute experimental sessions (60–208 trials). Figures 3d and e illustrate several examples of observed (black lines) and predicted (red lines) sequences of three-dimensional movements produced by monkey 1 (Fig. 3d) and 2 (Fig. 3e). After initial improvements, the predictions of three-dimensional hand trajectories reached asymptotic levels that were maintained throughout the experiments (Fig. 3f and g). The three-dimensional predictions were comparable to those obtained for one-dimensional movements (monkey 1: r = 0.74, 0.72 and 0.56 for the x-, y- and z-dimensions, respectively; monkey 2: r = 0.70, 0.54 and 0.77; 20-minute averages).
Figure 3: Real-time prediction of 3D hand movements.
a, b, 3D hand movement trajectories produced by monkey 1 (a) and 2 (b) during single experimental sessions. c, schematic diagram of the four possible target locations in the food reaching task. d, e, samples of observed (black line) and real-time predicted (red line) 3D hand movement for monkey 1 (d) and 2 (e ). f, g, Correlation coefficient variation for x (black line), y (blue line) and z (red line) dimensions of predicted 3D hand movements using the linear model.
High resolution image and legend (72K)Further demonstration of the robustness of our real-time approach was obtained by investigating how well model parameters obtained for one set of hand movements could be used to predict hand trajectories to other directions. For example, by training our linear model only with hand movements directed to targets on the right (targets 1 and 2) we were able to predict accurately hand trajectories to targets on the left (targets 3 and 4). The same was true for the reverse case, that is, using parameters derived from left movements to predict movements to the right (monkey 1, r = 0.80, 0.70 and 0.67 for the x-, y- and z- dimensions; monkey 2, r = 0.68, 0.53 and 0.81; averages for both conditions). Predictions of distal (targets 2 and 4) movements by training the model only with proximal (targets 3 and 1) hand trajectories and vice versa were comparably accurate (monkey 1, r = 0.81, 0.71 and 0.74 for the x-, y- and z-dimensions; monkey 2, r = 0.69, 0.63 and 0.79; averages).
We next analysed how each of the 2–4 different cortical areas contributed
to the prediction of one-dimensional movements by calculating the average
effect of removing individual neurons, one at a time, from the neuronal population
used in each real-time session. This neuron-dropping analysis was carried
out independently for each of the cortical areas, as well as for the combination
of all of them. We found that hyperbolic functions could fit (r range
0.996–0.9996) all curves that resulted from the neuron-dropping analysis,
using both the linear and ANN models in both animals. Figure
4a–e illustrates typical neuron-dropping curves and the corresponding
hyperbolic fits. Extrapolations of the hyperbolic curves (
Fig. 4f) revealed that 90% correct real-time prediction of one-dimensional
movements could be theoretically achieved by applying the linear model to
either 480 
65.7 PMd neurons, 666 83.0 M1, 629 64.2
PP or 1,195 142 ipsilateral MI/PMd neurons in monkey 1 (average and
s.e.m., 10 sessions). In monkey 2, the same level of accuracy would require
either 376 42.3 PMd neurons or 869 127.4 MI neurons (Fig. 4g). Thus, in both monkeys significantly fewer PMd (red)
neurons would theoretically be required to achieve the same level (90%) of
one-dimensional hand movement prediction accuracy (P < 0.001, Wilcoxon),
that is, on average, PMd neurons provided the highest contribution to the
predictions. MI (light blue) and PP (dark blue) ensembles provided comparably
lower contributions, whereas neurons located in the ipsilateral MI cortex
accounted for the lowest amount of variance (yellow line). When all recorded
cortical neurons were combined, the extrapolation of the hyperbolic functions
produced identical theoretical estimates for 90% prediction accuracy in both
monkeys (monkey 1, 625 64 neurons; monkey 2, 619 73 neurons,
average and s.e.m.; P = 0.70, Wilcoxon, not significant).
Figure 4: Neuron-dropping analysis.
a, b, Neuron-dropping curves (thick lines) obtained in a single session for all cortical neurons in monkey 1 (a) and 2 ( b) are precisely fitted by hyperbolic functions (thin lines). c, d, Neuron-dropping curves (thick lines) and corresponding hyperbolic functions (thin lines) for each cortical area for monkey 1 (c) and 2 (d). e, Range of fitted hyperbolic functions, x is the number of neurons and c is the fitted constant. f, g, Estimated number of neurons per cortical area required to reach an R2 of 0.9 using the hyperbolic functions in monkey 1 (f) and 2 (g). Lower values represent higher mean contribution per neuron. Three asterisks, P < 0.001.
High resolution image and legend (69K)These results are consistent with the hypothesis that motor control signals for arm movements appear concurrently in large areas of the frontal and parietal cortices5, 6, and that, in theory, each of these cortical areas individually could be used to generate hand trajectory signals in real time. However, the differences in estimated neuronal sample required to predict hand trajectories using a single cortical area probably reflect the functional specializations of these regions. Thus, the differences observed here are consistent with previous observations that PP and MI activity are influenced by motor parameters other than hand position (for example, visual information in the case of PP8, 25, or information related to the motor periphery in the case of MI26, 27). The relative contributions of these cortical areas may possibly also change according to such factors as the demands of the particular motor task, training level or previous motor experience9, 15.
In conclusion, we demonstrated that simultaneously recorded neural ensemble activity, derived from multiple cortical areas, can be used for the generation of both one-dimensional and three-dimensional signals to control robot movements in real time. Contrary to previous off-line algorithms5, 11, 12, our real-time approach did not make any a priori assumptions about either the physiological properties (for example, shape of the tuning curve) of the single neurons, or the homogeneity of the neuronal population sample. Instead, by using random samples of cortical neurons, we obtained accurate real-time predictions of both one-dimensional and three-dimensional arm movements. In this context, our findings support the notion that motor signals derived from ensembles of cortical neurons could be used for long-term control of prosthetic limb movements28, 29. We have shown that chronically implanted microwires can yield reliable recordings in primates for at least 24 months; this suggests that a combination of denser multi-wire arrays with implantable integrated circuits, designed to handle all real-time signal processing and mathematical analysis, could one day form the basis of a brain–machine interface for allowing paralysed patients to control voluntarily the movements of prosthetic limbs30.
Methods
For more detailed methods see Supplementary Information.
Electrophysiological recordings and behavioural tasks
Multiple microelectrode arrays (NBLABS, Dennison) containing 16–32
50-
m diameter microwires were chronically implanted in different cortical
areas14, 15, 16 in two owl monkeys. Stereotaxic coordinates,
published microstimulation maps17, 18 and intraoperative neural
mapping were used to locate the premotor, primary motor and posterior parietal
cortical areas. A many-neuron acquisition processor (MNAP, Plexon) was used
to acquire and distinguish activity from single neurons16. Both
monkeys were trained and tested on two behavioural tasks. In the first task,
the subjects were trained to centre a manipulandum for a variable time and
then to move it either to left or right targets in response to a visual cue
in order to receive a juice reward. The position of the manipulandum was recorded
continuously with a potentiometer. In the second task, the monkeys were trained
to place their right hands on a small platform, located waist high and next
to their bodies. When an opaque barrier was lifted, the monkeys reached out
and grabbed a small piece of fruit from one of four fixed target locations
on a tray mounted in front of the platform. In both tasks, the location and
orientation of the wrist in three-dimensional space was continuously recorded
using a plastic strip containing multiple fibre optic sensors (Shape Tape,
Measurand) attached to the right arms of the monkeys. Analogue signals from
the strip were sampled at 200 Hz. All sessions were also videotaped.
Data analysis
Predictions of hand position based on simultaneous neural ensemble firing were obtained by applying both a linear model and ANNs to these data. In the linear model, the relationship between the neuronal discharges in the matrix X(t), and hand position (1D or 3D) in Y(t) is
where b are the Y-intercepts in the regression, and a is
a set of impulse response functions describing the weights required for the
fitting as function of time lag u. 
(t) are the residual
errors. Hence, in this model the series in X (that is, neuronal firing
in time) are convolved with the weight functions a so that the sum
of these convolutions plus b approximates Y (hand trajectory)19, 20, 21. Offline analysis was first performed to test the validity
of this model. Y-intercepts and impulse response functions with a resolution
of 5 ms were calculated using a frequency-domain approach19, 21.
Real-time prediction was achieved by using a slightly modified linear model.
Neural data representing potential feedback information from movements were
not included in the model. Each neuron's discharges were counted in 100-ms
bins. Y-intercepts and weights were calculated using standard linear regression
techniques19. The performance of this simplified real-time model
was only very slightly inferior to the complete model outlined above.
Our strategy for applying ANNs is described elsewhere14, 23. Here, we employed the same data structure described for the linear model. Several feed-forward ANNs were evaluated offline. The best results were obtained using one hidden layer with 15–20 units, linear output units, the Powell-Beale conjugate gradient training algorithm22, and an early stopping rule. Predictions obtained with ANNs were comparable to those for the linear algorithm.
During real-time prediction of hand position, up to 10 minutes of data were first collected to fit the linear and ANN models. The resulting models were then sent back to the data acquisition computer for instantaneous and continuous prediction of hand position using the neuronal ensemble data acquired in real time. During the remainder of the experimental session, both models continued to be calculated repeatedly, using the most recently recorded 10 minutes of data. All calculations were performed by software designed in MATLAB (The Mathworks, Natick, Massachusetts, USA).
Real-time control of robot arms
Predictions of hand positions were broadcasted by a TCP/IP server to one or more computer clients. One of these clients controlled the movements of a robot arm (Phantom model A1.0; Phantom, SensAble Technologies)24 locally at Duke University, and another client was used to control simultaneously the movements of a remote robot arm (Phantom DeskTop) at MIT using the Internet. Signals describing the position of the robot arm in space were recorded on each client machine.
Neuron-dropping analysis
The relation between ensemble size and the goodness of fit for our models was analysed offline by randomly removing single neurons, one at a time, and fitting the models again using the remaining population until only one neuron was left. This procedure was repeated 25–30 times for each ensemble, so that an average curve was obtained to describe R2 as a function of ensemble size. Neuron-dropping curves were obtained for all cortical areas (separately or combined) in each monkey, and simple hyperbolic functions were used to fit them with very high accuracy. Next, we used these functions to estimate the ensemble size that would theoretically be required to attain a R 2 of 0.9.