Introduction
Just as the advent of fine electrode recordings in the 1940s shifted our neurophysiological focus from the whole brain to the single neuron, the advent of multi-neuron recordings has shifted our focus to the study of the neuronal population as a whole. Single-neuron recordings can define receptive fields1, 2, but these receptive fields are problematic in that they do not specify a neuron's actual response to a stimulus, but merely its potential responses, summed across the range of conditions in which it is tested. In the somatosensory cortex, receptive fields can be so large that they cannot (by themselves) accurately specify stimulus information3. Another drawback to single-neuron recordings is that there can be a marked variability in the neurons' responses, even to controlled, discrete stimuli. In awake animals, much of this 'noise' is also related to spontaneously fluctuating brain states, such as attention or intention4, 5.
Although this coarse coding and volatility diminishes the usefulness of single-neuron recordings for extracting sensory or motor information from the brain, it increases the usefulness of multi-neuron recordings. A central tenet of parallel distributed processing theory6 is that neural information is spread across populations of neurons, and that each neuron contributes to the processing of many different informational factors. If so, one should be able to extract more information from the brain by recording simultaneously from large numbers of neurons. Over time, this basic hypothesis has sparked the development of a variety of electrode arrays and electronic devices for this purpose. Several laboratories use chronically implanted electrode arrays because they allow stable recordings of discriminated single neurons and/or field potentials from up to hundreds of electrodes over long time periods7, 8, 9, 10, 11, 12, 13, 14, 15, 16.
The development of such methods allowed neural recordings to be analyzed at the level of neuronal populations, as well as the single-neuron level. Because the same variables (e.g., neurons) are recorded simultaneously, one can directly measure the evanescent patterns of synchronous activity that typically occur in neuronal populations during spontaneous fluctuations in brain and/or behavioral state. This ability to statistically account for covariance within neuronal ensembles provides a major advantage over the alternative approach of recording the same single neurons in serial order. Whereas multi-neuron recordings can use neuronal covariance patterns to detect spontaneous changes in brain state, in serial single-neuron recordings, such variance is normally unexplained and thus must be considered as noise. One can utilize this approach to measure complex and heterogeneous changes in neuronal population activity recorded over many time frames, ranging from fractions of seconds, as in brain oscillations, to several hours or days, as in drug or hormone effects17, 18, 19, 20, 21, 22.
How then does one usefully extract the information contained within neuronal populations? Statistical approaches, such as linear or nonlinear multivariate regression and discriminant and principal components analysis (PCA), provide a more accurate estimation of neural information than the direct summation of neuronal receptive (or motor) fields, unless their unique features are laboriously measured in isolation (see ref. 23, p. 456–461 this issue). Linear regression, for example, can use neuronal populations in the motor cortex to predict arm movement and control external devices13, 14. Time-binned neural and arm position data from several training trials are used to 'fit' a regression equation that specifies a weighting for each neuron's contribution to the arm movement. Sequential arm positions can then be predicted in real time by weight-summing the neuronal population activity for each new time bin. The accuracy of this prediction (i.e., its correlation with real arm movement) monotonically increases with the number of recorded neurons11.
A major advantage of generating predictions of neural population coded outputs is that they can be directly compared to real external events (e.g., movements). Thus one can easily test the usefulness of different mathematical approaches to extract the neuronal information.
Though conventional statistics include many approaches and provide clear measures of significance24, other approaches may often be preferable because of particularities of data distribution or the underlying hypothesis. Thus, neuronal population recordings have been analyzed using Bayesian statistics, artificial neural networks and kernel analysis.
Analysis of spatiotemporally distributed information
It is becoming clear that information in the brain is coded not only in space (across neuronal populations) but also in time (temporal patterning of neuronal responses). For example, vibromechanical touch stimuli to the palm pads of rats produce discrete responses in the primary and secondary afferents, but as these signals ascend through the thalamus and cortex, they become increasingly distributed across larger portions of the somatosensory representation; they also show an increasing range of response latencies10. A different view of neural coding thus emerges from multi-neuron recordings in the thalamus and cortex (Fig. 1). When the responses to fine punctate tactile stimulation of palm pad 3 and palm pad 2 were obtained from 45 neurons recorded from a 16-electrode array covering the whole somatosensory cortical forepaw area in a rat (Fig. 1a,b) and plotted as normalized three-dimensional (3D) surface maps, both showed spatiotemporally complex patterns of neural responses that were distributed across the whole forepaw representation, even though the two stimulus sites were on closely adjacent palm pads. Any mathematical procedure used to discriminate between these two stimuli must incorporate both spatial and temporal criteria, because the significant information is spread out over many neurons and also over post-stimulus time3.
Figure 1: Neuronal population response maps.
Normalized 3D surfaces depict post-stimulus responses of the same 45 simultaneously recorded neurons (in rat S1 cortex) to vibromechanical touch of (a) palm pad 3, (b) palm pad 2 and (c) their difference. Each neuron's peristimulus responses were standardized into Z scores. (Mean and standard deviation (s.d.) of responses measured in the 15-ms period before the stimulus were used to standardize all peristimulus bins.) For each plot, the x axis shows neuron number (1–45), the y axis is response latency (-15 ms to 35 ms around stimulus), and the z axis (in s.d.) depicts standardized response level in relief and color codes (calibrated in bars at right of a–c). Panel c is also a 3D surface, but viewed from above to show the precise spatiotemporal difference pattern between palm pad 3 (positive in yellow/red) and palm pad 2 (negative in green/blue). It also shows (in red) the spatial arrangement of the 16 recording electrodes in the array, which consisted of two rows of 8 electrodes apiece (0.28 mm tip separation; 0.5 mm row separation). The whole array was chronically implanted across the forepaw representation (electrode 1 most caudolateral; electrode 16 most rostromedial). A week later, 1–4 single neurons were discriminated from each electrode, and recordings began. All stimuli used a vibromechanical stimulator with a 1-mm diameter shaft tip held just above the skin between stimuli (0.5 mm displacement, 500 
s duration, 2 Hz, 340 stimuli). Figure from J.K.C. and S. Xu, unpublished data.
Encoding multi-functional and dynamical phenomena
Distributed processing theory posits that neurons are multi-functional—that is, under different conditions they can manifest different information. As a simple example, Figure 2 shows recordings from neuronal populations in the somatosensory cortex and thalamus that are rapidly shifting between two distinct functional states: sensory response processing and spontaneous oscillations. Here we show that one population coding method (PCA) can be used to cleanly distinguish between these two functions, allowing their interactions to be studied25. These functional states cannot be differentiated in single neurons because they are distinguished by the different levels of correlation among neurons. Our PCA method involves cross-correlating the binned spiking activity of all recorded neurons over some time period and then performing successive eigenvalue rotations of this correlation matrix to yield a set of uncorrelated principal components, each representing a uniquely weighted average of the original neurons. Neuronal populations were recorded in the cortex and thalamus during rhythmic oscillations and also during 1-Hz stimulation of the forepaw (Fig. 2a). We used three neural population functions, weighted to depict principal components 1–3 (PC1–3) of the whole neuronal population (Fig. 2b). Whereas PC1 resembles an unweighted population average, PC2 segregates the paw stimuli (positive) from the main thalamocortical oscillatory regime (negative), and PC3 isolates a secondary oscillatory regime emanating from the thalamus. In statistical terms, each neuron contributes weighted portions of its variance to each of these three components, which here discriminate between different functional activities of the overall population. Although the correctness of various methods for deriving and applying the weights will be investigated for years, this general concept has proved to be quite useful for understanding population coding in neuronal populations, especially those discussed below.
Figure 2: Interactions between sensory stimuli and brain dynamics in neural populations.
(a) Spike rasters of 36 simultaneously recorded neurons in S1 cortex and 16 in VPL thalamus during 8 vibromechanical stimuli of palm pad 3 (0.5 mm displacement, 500 s duration, 1 Hz). Stimulation times at arrows. (b) Population functions encoded using weights derived from a principal components analysis (PCA) using neural data from the whole experiment. PCA involved eigenvalue decomposition of the 52 
52 correlation matrix between all neurons. Population functions for PCs 1–3 were constructed by binning the spiking data (10 ms), multiplying each neuron's firing by its weighting in PCs 1–3, and summing over all neurons. Population functions are displayed as rate meters. Bin heights scaled as weighted total firing rate (vertical axis). Figure from J.K.C. and S. Xu, unpublished data.
Extraction of neuronal population information: neural prostheses
The most visible success of multi-neuron recording to date has been in extracting information from the brain in real time and using this information to control external devices26. We originally developed such technology for online population encoding of multi-neurons in the motor cortex (M1) and thalamus (ventral lateral) in rats trained to press a lever to move a robot arm that retrieved water from a dropper27. Population encodings of the brain's 'motor signal' were electronically implemented in real time, allowing the robot arm to be moved in direct proportion to the population function amplitude. The rat thereafter obtained its daily water by using this neural signal alone to control the robot's movement to the water dropper. Over time, the rat was able to obtain its water without actually pressing the lever, suggesting that the M1 cortex neurons had learned a direct representation of the robot arm, independent of the real arm.
Similar results have now been obtained in monkeys12, 28, 29, 30, 31, derived from simultaneous recording from hundreds of neurons across the primary motor, premotor, somatosensory and parietal cortices (Fig. 3). An offline computer was used to calculate the neuronal weightings for multivariate linear regression predictive filters (as described above). An on-line computer then used these filters to convert real-time neuronal activity into neural population codes that simultaneously specified three dimensions of hand position, velocity and/or force. These population-coded outputs were then used to control a robot arm and/or a cursor on a computer screen with good accuracy (R2 = 0.5–0.8) compared with the monkey's arm itself. Though the quality and quantity of useful coding information obtained in these recordings varied across the bilaterally recorded sensorimotor cortical areas, all of these areas contributed at least some useful information to the control. This is consistent with previous findings that cortical neurons tend to be widely tuned1, 2, 3, 29, 30 and thus can be active in a wide variety of experimental conditions. Much of the success of multiple-electrode recording is attributable to this wide tuning, because the investigator can depend on obtaining useful signals from the majority of recorded neurons.
Figure 3: Brain-controlled neural prosthesis.
(a) Monkey is initially trained to sit in a chair and manipulate a pole to move a cursor toward a target depicted on a computer screen (not depicted here). This pole movement is also translated into equivalent movement of a robot arm. Three tasks are trained in serial order: (1) move pole to a particular target position, (2) squeeze force transducer on pole to reach a force target, and (3) do 1 (move) and then 2 (squeeze) in sequence. The first experiments involve 'pole control' (depicted in green) in which the pole output goes to the robot controller, which moves the robot and/or its gripper. Feedback from the robot then moves the cursor on the screen. Once this task is learned, the control of the robot is changed to 'brain control' (depicted in red), utilizing multi-neuron recordings obtained from microwire electrode arrays implanted in several contralateral and ipsilateral sensorimotor cortices, including primary motor (M1), dorsal premotor (PMd), primary somatosensory (S1) and posterior parietal (PP) cotices. Multi-neuron recordings are amplified, filtered and discriminated by an acquisition system box (Plexon Inc.), as controlled by a graphics interface on a host computer that also functions as a data server. A client computer receives spike times and field potential data from the server in real time, and uses adaptive linear or nonlinear fitting algorithms to 'predict' the pole movement, transforming the neural data into neuronal population functions suitable for controlling the robot and/or its gripper. The resulting feedback from sensors on the robot, including arm position, velocity and/or force is then represented with a cursor on the monkey's computer display. The monkey ultimately learns to use the decoded neural population prediction functions to directly control the cursor movement toward the target, often without moving the arm at all. Figure adapted from ref. 31.
Full size image (38 KB)Future of neural population recording
Thanks to these early successes, this field may soon have commercial implications. The Nicolelis laboratory has already demonstrated the ability of simultaneously recorded neurons in the brains of awake human patients to predict arm and hand movements32. Further successes will necessitate further research in a number of areas, including development of biocompatible ultra-miniaturized multi-electrode designs, implantable electronics to amplify and process these recordings, and embedded computer systems to control the output functions, such as computer screens, robots and wheelchairs. Moreover, this neuroprosthetic technology will be joined with others, such as 'functional neuromuscular stimulation', which uses stimulating electrodes to activate a paralyzed patient's muscles. As another example, our current focus is to convert the open-loop brain-controlled robotic systems into a closed-loop neural interface by using multi-electrode arrays for both recording and stimulation in the brain33. Finally, non-invasive techniques, such as EEG, MEG, fMRI and infrared (IR) methods may also be important in many of these technologies. Overall, this explosion of information will also require a major neuroinformatic effort to design database methods capable of efficiently storing the necessary data so that they can be quickly mobilized for use in controlling a robot arm or a real arm in real time.
At the same time, basic scientists will find increased impetus to work on the problem of population coding in the brain. This will require more cooperation between experimentalists, theoreticians and neuroinformaticists, particularly in the sharing of datasets. Though the current academic and funding culture tends to discourage such interactions, focused government efforts can in theory overcome these difficulties.
