Abstract
The variable responses of sensory neurons tend to be weakly correlated (spikecount correlation, r_{sc}). This is widely thought to reflect noise in shared afferents, in which case r_{sc} can limit the reliability of sensory coding. However, it could also be due to feedback from higherorder brain regions. Currently, the relative contributions of these sources are unknown. We addressed this by recording from populations of V1 neurons in macaques performing different discrimination tasks involving the same visual input. We found that the structure of r_{sc} (the way r_{sc} varied with neuronal stimulus preference) changed systematically with task instruction. Therefore, even at the earliest stage in the cortical visual hierarchy, r_{sc} structure during task performance primarily reflects feedback dynamics. Consequently, previous proposals for how r_{sc} constrains sensory processing need not apply. Furthermore, we show that correlations between the activity of single neurons and choice depend on feedback engaged by the task.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our bestvalue onlineaccess subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
 Purchase on SpringerLink
 Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Tomko, G. J. & Crapper, D. R. Neuronal variability: nonstationary responses to identical visual stimuli. Brain Res. 79, 405–418 (1974).
Henry, G. H., Bishop, P. O., Tupper, R. M. & Dreher, B. Orientation specificity and response variability of cells in the striate cortex. Vision Res. 13, 1771–1779 (1973).
Cohen, M. R. & Kohn, A. Measuring and interpreting neuronal correlations. Nat. Neurosci. 14, 811–819 (2011).
Shadlen, M. N. & Newsome, W. T. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. J. Neurosci. 18, 3870–3896 (1998).
Sompolinsky, H., Yoon, H., Kang, K. & Shamir, M. Population coding in neuronal systems with correlated noise. Phys. Rev. E 64, 051904 (2001).
Gu, Y., Angelaki, D. E. & DeAngelis, G. C. Contribution of correlated noise and selective decoding to choice probability measurements in extrastriate visual cortex. eLife 3, 1–19 (2014).
Liu, S., Dickman, J. D., Newlands, S. D., DeAngelis, G. C. & Angelaki, D. E. Reduced choicerelated activity and correlated noise accompany perceptual deficits following unilateral vestibular lesion. Proc. Natl. Acad. Sci. USA 110, 17999–18004 (2013).
Zohary, E., Shadlen, M. N. & Newsome, W. T. Correlated neuronal discharge rate and its implications for psychophysical performance. Nature 370, 140–143 (1994).
Abbott, L. F. & Dayan, P. The effect of correlated variability on the accuracy of a population code. Neural Comput. 11, 91–101 (1999).
Snippe, H. P. & Koenderink, J. J. Information in channelcoded systems: correlated receivers. Biol. Cybern. 67, 183–190 (1992).
Averbeck, B. B., Latham, P. E. & Pouget, A. Neural correlations, population coding and computation. Nat. Rev. Neurosci. 7, 358–366 (2006).
Cohen, M. R. & Maunsell, J. H. R. Attention improves performance primarily by reducing interneuronal correlations. Nat. Neurosci. 12, 1594–1600 (2009).
Graf, A. B., Kohn, A., Jazayeri, M. & Movshon, J. A. Decoding the activity of neuronal populations in macaque primary visual cortex. Nat. Neurosci. 14, 239–245 (2011).
Mitchell, J. F., Sundberg, K. A. & Reynolds, J. H. Spatial attention decorrelates intrinsic activity fluctuations in macaque area V4. Neuron 63, 879–888 (2009).
Johnson, K. O. Sensory discrimination: decision process. J. Neurophysiol. 43, 1771–1792 (1980).
MorenoBote, R. et al. Informationlimiting correlations. Nat. Neurosci. 17, 1410–1417 (2014).
Haefner, R. M., Gerwinn, S., Macke, J. H. & Bethge, M. Inferring decoding strategies from choice probabilities in the presence of correlated variability. Nat. Neurosci. 16, 235–242 (2013).
Nienborg, H. & Cumming, B. Correlations between the activity of sensory neurons and behavior: how much do they tell us about a neuron’s causality? Curr. Opin. Neurobiol. 20, 376–381 (2010).
Shadlen, M. N., Britten, K. H., Newsome, W. T. & Movshon, J. A. A computational analysis of the relationship between neuronal and behavioral responses to visual motion. J. Neurosci. 16, 1486–1510 (1996).
Bair, W., Zohary, E. & Newsome, W. T. Correlated firing in macaque visual area MT: time scales and relationship to behavior. J. Neurosci. 21, 1676–1697 (2001).
Lee, D., Port, N. L., Kruse, W. & Georgopoulos, A. P. Variability and correlated noise in the discharge of neurons in motor and parietal areas of the primate cortex. J. Neurosci. 18, 1161–1170 (1998).
Smith, M. A. & Kohn, A. Spatial and temporal scales of neuronal correlation in primary visual cortex. J. Neurosci. 28, 12591–12603 (2008).
Kohn, A. & Smith, M. A. Stimulus dependence of neuronal correlation in primary visual cortex of the macaque. J. Neurosci. 25, 3661–3673 (2005).
Callaway, E. M. Feedforward, feedback and inhibitory connections in primate visual cortex. Neural Netw. 17, 625–632 (2004).
Sillito, A. M., Cudeiro, J. & Jones, H. E. Always returning: feedback and sensory processing in visual cortex and thalamus. Trends Neurosci. 29, 307–316 (2006).
Ruff, D. A. & Cohen, M. R. Attention can either increase or decrease spike count correlations in visual cortex. Nat. Neurosci. 17, 1591–1597 (2014).
Cohen, M. R. & Newsome, W. T. Contextdependent changes in functional circuitry in visual area MT. Neuron 60, 162–173 (2008).
Lange, R. D. & Haefner, R. M. Inferring the brain’s internal model from sensory responses in a probabilistic inference framework. Preprint at bioRxiv https://doi.org/10.1101/081661 (2016).
Haefner, R. M., Berkes, P. & Fiser, J. Perceptual decisionmaking as probabilistic inference by neural sampling. Neuron 90, 649–660 (2016).
Cumming, B. G. & Nienborg, H. Feedforward and feedback sources of choice probability in neural population responses. Curr. Opin. Neurobiol. 37, 126–132 (2016).
Nienborg, H. & Cumming, B. G. Decisionrelated activity in sensory neurons may depend on the columnar architecture of cerebral cortex. J. Neurosci. 34, 3579–3585 (2014).
Nienborg, H. & Cumming, B. G. Psychophysically measured task strategy for disparity discrimination is reflected in V2 neurons. Nat. Neurosci. 10, 1608–1614 (2007).
Ahumada, A. J. Jr. Perceptual classification images from Vernier acuity masked by noise. Perception 25, 2 (1996).
Ecker, A. S., Denfield, G. H., Bethge, M. & Tolias, A. S. On the structure of neuronal population activity under fluctuations in attentional state. J. Neurosci. 36, 1775–1789 (2016).
Britten, K. H., Newsome, W. T., Shadlen, M. N., Celebrini, S. & Movshon, J. A. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis. Neurosci. 13, 87–100 (1996).
Crapse, T. B. & Basso, M. A. Insights into decision making using choice probability. J. Neurophysiol. 114, 3039–3049 (2015).
Nienborg, H., Cohen, M. R. & Cumming, B. G. Decisionrelated activity in sensory neurons: correlations among neurons and with behavior. Annu. Rev. Neurosci. 35, 463–483 (2012).
Moran, J. & Desimone, R. Selective attention gates visual processing in the extrastriate cortex. Science 229, 782–784 (1985).
McAdams, C. J. & Maunsell, J. H. Effects of attention on orientationtuning functions of single neurons in macaque cortical area V4. J. Neurosci. 19, 431–441 (1999).
Goris, R. L. T., Movshon, J. A. & Simoncelli, E. P. Partitioning neuronal variability. Nat. Neurosci. 17, 858–865 (2014).
Nienborg, H. & Cumming, B. G. Decisionrelated activity in sensory neurons reflects more than a neuron’s causal effect. Nature 459, 89–92 (2009).
Wimmer, K. et al. Sensory integration dynamics in a hierarchical network explains choice probabilities in cortical area MT. Nat. Commun. 6, 6177 (2015).
Kwon, S. E., Yang, H., Minamisawa, G. & O’Connor, D. H. Sensory and decisionrelated activity propagate in a cortical feedback loop during touch perception. Nat. Neurosci. 19, 1243–1249 (2016).
Treue, S. & Martínez Trujillo, J. C. Featurebased attention influences motion processing gain in macaque visual cortex. Nature 399, 575–579 (1999).
James, W. The Principles of Psychology. (Holt, New York, 1890).
Posner, M. I. & Petersen, S. E. The attention system of the human brain. Annu. Rev. Neurosci. 13, 25–42 (1990).
Tajima, C. I. et al. Population code dynamics in categorical perception. Sci. Rep. 6, 22536 (2016).
Knill, D. C. & Richards, W. Perception as Bayesian Inference. (Cambridge University Press, Cambridge, 1996).
Von Helmholtz, H. Handbuch der physiologischen Optik. 9, (Voss, Leipzig, 1867).
Churchland, M. M. et al. Stimulus onset quenches neural variability: a widespread cortical phenomenon. Nat. Neurosci. 13, 369–378 (2010).
Judge, S. J., Richmond, B. J. & Chu, F. C. Implantation of magnetic search coils for measurement of eye position: an improved method. Vision Res. 20, 535–538 (1980).
Cumming, B. G. & Parker, A. J. Binocular neurons in V1 of awake monkeys are selective for absolute, not relative, disparity. J. Neurosci. 19, 5602–5618 (1999).
Burgess, A. E. & Colborne, B. Visual signal detection. IV. Observer inconsistency. J. Opt. Soc. Am. A. 5, 617–627 (1988).
Kang, I. & Maunsell, J. H. R. Potential confounds in estimating trialtotrial correlations between neuronal response and behavior using choice probabilities. J. Neurophysiol. 108, 3403–3415 (2012).
Higham, N. J. Computing a nearest symmetric positive semidefinite matrix. Linear Algebra Appl. 103, 103–118 (1988).
Acknowledgements
We thank B. Wurtz and J. McFarland for useful discussions; R. Krauzlis, B. Conway and A. Ghazizadeh for comments on an earlier version of the manuscript; and B. Nagy, I. Bunea and D. Parker for veterinary care. This work was supported by the National Eye Institute (Intramural Research Program for A.G.B. and B.G.C. and R01EY028811 for R.M.H.) .
Author information
Authors and Affiliations
Contributions
A.G.B. and B.G.C. conceived and designed the experiments. A.G.B. performed the experiments and all aspects of the analysis. A.G.B. and B.G.C. wrote the paper. R.M.H. advised and assisted with the data analysis and the paper. B.G.C. advised at all stages.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Integrated supplementary information
Supplementary Figure 1 Subjects’ psychophysical kernels reflect task instruction.
a. To perform psychophysical reverse correlation, we first summarized the orientation energy of each trial (see Methods). Example shown for several 0%signal trials, with black and gray lines used to indicate the choice elicited (example session, monkey ‘lem’). The dashed vertical lines show the associated task orientations. b. The psychophysical kernel (blue curve) is calculated as the difference between the two choiceconditioned averages (black and gray curves), with an arbitrary choice of sign. Note that the circular mean orientation of the kernel (vertical blue shaded region) is significantly offset (p < 0.001, bootstrap test, onesided) from the nominally “positive” task orientation (black dashed line), indicating the subject’s choices were primarily based on orientations deviated from those given by task instruction. All shaded regions indicate the mean±1 bootstrap SEM obtained from trial resampling. Axis labels and yaxis units are the same as in (a). c. Average kernel circular mean orientations for two subsets of recording sessions within which similar task contexts were used (same groups as in Fig. 2). Error bars indicate ±1 bootstrap SEM obtained by resampling sessions (n=17 sessions for subset 1 and 24 sessions for subset 2). The average “positive” task orientations for the two groups are shown by the dashed lines. Subjects used distinct task strategies for the two subsets of sessions, as shown by the difference in kernel orientations. Note that the subjects did not completely follow task instruction: the distance between kernel orientations is smaller than the distance between the task orientations.
Supplementary Figure 2 Individual subject comparison.
a,b. Average r_{ sc } matrices for each subject, as in Fig. 2e, are similar. Difference in overall average r_{ sc } may be due to differences in stimulus size (see Methods). c,d. The eigenvectors corresponding to the largest eigenvalues of the matrices in (a) & (b), as in Fig. 4. Error bars are as in Fig. 4. e,f. Model comparison for individual subject data. In both cases, the “single eigenvector” model outperformed the “diagonal ridge” model (p=0.005 for both animals). (The more complex regression model in Fig. 3 could not be reliably fit using the data from each subject individually due to limited sampling of tasks). As in Figs. 3 and 4, model performance is normalized relative to the explainable variance. This was computed separately for each subject in these panels. Errors bars are 1 SEM around the mean, obtained by resampling pairs (n = 289 for monkey ‘lem’ and 522 for monkey ‘jbe’). Statistical comparison performed using a onesided bootstrap test obtained in the same way. g,h. Histogram of observed CPs, from the subset of neurons (n=63 and 81 for monkeys ‘lem’ and ‘jbe,’ respectively) significantly preferring one of the two task orientations (d´>0.9 at highest signal level). The means were significantly above chance (p < 0.001 and p=0.004, bootstrap test using cell resampling, onesided). CPs that were individually significant (p < 0.05, bootstrap test using trial resampling, onesided) are shown in black. The mean CP was significantly higher in monkey ‘lem’ (p=0.02, bootstrap test using cell resampling, onesided).
Supplementary Figure 3 Taskdependent r_{sc} structure absent during passive fixation.
a. Average r_{ sc } matrix observed during blocks of trials used to measure neuronal orientation tuning, during which the animal fixated passively for reward (matrix obtained as in Fig. 2e). The latticelike pattern observed during task performance is absent. Data from only 556 pairs are shown, as not all recording sessions included the interleaved 0% signal trials. b. Eigenspectrum of the matrix in (a), with first five eigenvalues shown. The largest eigenvalue of the r_{ sc } matrix in (a) did not exceed the chance distribution (mean +/ 1 SEM in blue), unlike during task performance (Fig. 2). Note that the eigenvalues are smaller than in Fig. 4a because the variance within the matrix in (a) is smaller. Error bars are as in Fig. 4a. c. Goodnessoffit for the “single eigenvector” and “diagonal ridge” models, calculated as % variance explained relative to our estimate of the explainable variance in the data, as in Fig. 4. Error bars around the mean obtained through repeated 50fold crossvalidation from among the 556 pairs, which also provided the basis for the onesided statistical comparison (p=0.04). Note these results contrast with those during task performance (Fig. 4e).
Supplementary Figure 4 R_{sc} structure is not driven by stimulus variability.
a. a. The average, taskaligned V1 r_{ sc } matrix for orientation (identical to Fig. 2e). b. The r_{ sc } matrix attributable to covariability introduced by random fluctuations in the white noise stimuli, identified using a doublepass trialshuffling procedure (see Methods). The taskaligned latticelike structure is abolished, and the mean value of r_{ sc } is drastically reduced (note difference in color scales). The remaining structure resembles a weak, diagonal ridge (i.e. “limited range correlations”). Axis labels as in (a). c. A scatter plot of r_{ sc } before versus after the trialshuffling procedure. There was no significant correlation between the two (Pearson’s r, p=0.077, bootstrap test, onesided) and mean r_{ sc } after shuffling was not significantly above zero (mean: −0.0004, p=0.399, bootstrap test, onesided). Red cross indicates the population average.
Supplementary Figure 5 R_{sc} structure is not influenced by stimulus history.
a. Average, taskaligned r_{ sc } matrix (identical to Fig. 2e). b. Average, taskaligned r_{ sc } matrix for orientation after normalizing spike counts to remove the influence of the stimulus on the preceding trial (see Methods). Color scaling and axis labels as in (a). c. Scatter plot of r_{ sc } before and after the normalization procedure, showing virtually no difference between the two. (Gray line is the identity line, Pearson’s r shown). This demonstrates that stimulus history does not significantly influence observed r_{ sc }.
Supplementary Figure 6 R_{sc} matrix is similar after short and long intertrial intervals.
a,b. The average, taskaligned r_{ sc } matrix for orientation for two subsets of trials, obtained from a median split based on the duration of the preceding ITI for each session. The shortITI subset of trials was preceded by an average ITI of 1.1 seconds, while the longITI subset was preceded by an average ITI of 4.8 seconds. The latticelike pattern is clearly present in both subsets. No attenuation in the rsc structure was observed in the longITI subset, inconsistent with stimulus history being a major determinant. Axis labels and color scale in (b) are the same as in (a). c. Slope of (type 2) regression line of r_{ sc } values obtained using the two subsets of trials against the r_{ sc } values obtained using all trials. Values below 1 indicate an attenuation of the r_{ sc } structure and values greater than 1 indicate an amplification. The amplitude of the latticelike pattern was increased after longITI trials relative to shortITI trials (p < 0.001, bootstrap test, onesided, n=811 pairs). Error bars represent +/ 1 bootstrap SEM. d. The mean r_{ sc } value was also greater after long ITIs (p=0.012, bootstrap test, onesided, n=811 pairs). Error bars represent +/ 1 bootstrap SEM.
Supplementary Figure 7 R_{sc} structure is not generated by fixational eye movements.
a. The trialaveraged trajectories of mean binocular eye position (example session M225, monkey ‘lem’). Black and gray traces correspond to trials preceding leftward and rightward choice saccades, respectively. Red lines indicate eye position 1.5 s after stimulus onset, around which time the trajectories noticeably diverge. b. Eye position trajectories for a subset of trials from which choice could no longer be predicted, identified using LDA after having removed the last quarter of the trial. c. The Euclidean distance between the trajectories associated with the two choices increased over time. The distances before and after the trialremoval procedure are shown in blue and red, respectively, with error bars indicating +/ 1 SEM obtained from resampling trials (n=1,397 and 1,117 trials before and after trial removal, respectively). The distance expected by chance, obtained by shuffling the choice labels, is shown in black. The distance did not exceed chance levels in the subset of trials left after trial removal until 1.5s after stimulus onset. d,e. Histograms of LDA posterior probabilities, giving the likelihood that a trial was associated with a leftward choice saccade before (d) and after (e) the trialremoval procedure. The gray and black distributions represent trials followed by rightward and leftward choices, respectively. f. The taskaligned V1 r_{ sc } matrix for orientation (averaged across all sessions) after applying the trialremoval procedure to each session. This is qualitatively identical to the matrix obtained before removing trials (Fig. 2e). g. Eigenspectrum for the matrix in (f), with first five eigenvalues shown. The largest eigenvalue was significantly larger than chance (p=0.004, permutation test, onesided), similar to the results in Fig. 4a. Error bars are as in Fig. 4a. h. The eigenvector corresponding to the largest eigenvalue (+/ 1 bootstrap SEM) of the matrix in (f), comparable to Fig. 4b.
Supplementary Figure 8 Alternative fixed/taskdependent component model.
a. Pseudocolor plot showing goodnessoffit for a model describing r_{ sc } structure using fixed and taskdependent components, using an alternative approach to the one shown in Fig. 3 (see Methods). Goodnessoffit is shown for versions of such a model allowing different numbers of basis functions for the two components. The best model includes two basis functions for the taskdependent component only. Model performance is quantified under 50fold crossvalidation, and normalized as described in the Methods. b. Fixed component fits for all versions of the model whose performance is shown in (a). For the best model (circled) there is no fixed component whatsoever (i.e. zero basis functions allowed). In versions of the model in which a fixed component is allowed, there is a consistent structure resembling a weak diagonal ridge with peaks near 30°/30° and 120°/120,° consistent with the model fits using the other modeling approach (Fig. 3). c. Taskdependent component fits for all versions of the model whose performance is shown in (a). The fit from the best model is circled. Again, across versions of the model allowing different numbers of basis functions, the fits are consistent and similar to the results of the alternative model shown in Fig. 3. (Note that, due to the diagonal symmetry of correlation matrices, even fits containing only a single basis function contain two peaks/troughs).
Supplementary Figure 9 Relationship between r_{sc} structure and CP is insensitive to readout weights.
a. Readout weights were unobserved but are required to predict CP based on the observed r_{ sc } structure using the feedforward linear pooling framework. We sampled from a large distribution of random readout weight profiles that could support task performance (examples shown) to generate a distribution of CP predictions (see Methods). For illustration, uniform and sinusoidal weight profiles are shown in white and black, respectively. b. The predicted mean CP associated with the average of all readout weights are shown in Fig. 8 in the main text. Here we show the uncertainty associated with the distribution of plausible readout weight profiles. Black bars indicate mean CPs +/ 1 SEM. The predictions associated with the uniform and sinusoidal weight profiles in (a) are shown with white and black circles, respectively. Note that the variability introduced by the readout weights is comparable to the uncertainty in the measurements of r_{ sc } (compare error bars to those in Fig. 8c), demonstrating that the results are quite robust to not knowing the true weights. This is not surprising given the observed correlation matrix was nearly rank1. Since the influence of the readout weights depends on their similarity to the correlation matrix eigenvectors, a low rank matrix has fewer ways to be influenced by different readout weights.
Supplementary Figure 10 Comparison to Cohen and Newsome (2008).
a. Schematic of the task design in Cohen and Newsome (2008)^{27}, after Fig. 1c in that study. For this example neuronal pair with preferred directions indicated by the black, solid arrows, one task was chosen such that the neurons contributed to the same choice (“same pool” condition). The other task was chosen so that the neurons contributed to opposite choices (“different pool” condition). The difference in preferred direction (ΔPD) determined the locations along the purples and green dashed lines in (b) to which this pair contributed data. a. Schematic of the taskaligned r_{ sc } matrix for preferred direction, analogous to the matrix we measured (Fig. 2e). The purple and green boxes indicate the regions corresponding to pairs supporting the same and opposite choices, respectively. The dashed lines indicate the regions of the matrix it was possible to sample using their approach. c. Reproduction of Fig. 2d, showing the average, taskaligned r_{ sc } matrix for orientation. d. For the sake of comparison, we plot the r_{ sc } values from our dataset in a way analogous to the main result in the prior study (i.e. the average data sampled along the “same pool” and “different pool” lines, as given by the dashed lines in (c)). This plot is directly comparable to the main data figure from the prior study (their Fig. 4a) and shows qualitatively similar data, albeit with lower overall values of r_{ sc }. We notably failed to replicate the inversion of the main result for pairs with opposing stimulus preference. This feature of the data was explained by the authors as an effect of simultaneous attention to both motion direction and the “motion axis” (i.e. orientation). In our task, an analogous mechanism would require simultaneous attention to a single orientation and an orientation pair, which may not be possible. Therefore, it is unclear to what degree this difference reflects a true discrepancy in the data. Standard error bar and statistical comparison both obtained from a bootstrap obtained by resampling from the set of 811 pairs.
Supplementary Figure 11 The taskdependent structure of spikecount covariance.
The average, taskaligned V1 covariance matrix, calculated as describe in Methods. This is closely similar to the r_{ sc } matrix shown in Fig. 2e, demonstrating that this central observation is not a consequence of using r_{ sc } values based on normalized spike counts.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–11
Rights and permissions
About this article
Cite this article
Bondy, A.G., Haefner, R.M. & Cumming, B.G. Feedback determines the structure of correlated variability in primary visual cortex. Nat Neurosci 21, 598–606 (2018). https://doi.org/10.1038/s4159301800891
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s4159301800891
This article is cited by

Coexistence of state, choice, and sensory integration coding in barrel cortex LII/III
Nature Communications (2024)

Feedback processing in the primate brain and in AI systems
Science China Technological Sciences (2024)

Correlated variability in primate superior colliculus depends on functional class
Communications Biology (2023)

Targeted V1 comodulation supports taskadaptive sensory decisions
Nature Communications (2023)

Continuous multiplexed population representations of task context in the mouse primary visual cortex
Nature Communications (2023)