Nature Neuroscience  Brief Communication
Sensory uncertainty decoded from visual cortex predicts behavior
 Ruben S van Bergen^{1}^{, }
 Wei Ji Ma^{2}^{, }
 Michael S Pratte^{3}^{, }
 Janneke F M Jehee^{1}^{, }
 Journal name:
 Nature Neuroscience
 Volume:
 18,
 Pages:
 1728–1730
 Year published:
 DOI:
 doi:10.1038/nn.4150
 Received
 Accepted
 Published online
Bayesian theories of neural coding propose that sensory uncertainty is represented by a probability distribution encoded in neural population activity, but direct neural evidence supporting this hypothesis is currently lacking. Using fMRI in combination with a generative modelbased analysis, we found that probability distributions reflecting sensory uncertainty could reliably be estimated from human visual cortex and, moreover, that observers appeared to use knowledge of this uncertainty in their perceptual decisions.
Subject terms:
At a glance
Figures
References
 Ernst, M.O. & Banks, M.S. Nature 415, 429–433 (2002).
 Körding, K.P. & Wolpert, D.M. Nature 427, 244–247 (2004).
 Ma, W.J., Beck, J.M., Latham, P.E. & Pouget, A. Nat. Neurosci. 9, 1432–1438 (2006).
 Jazayeri, M. & Movshon, J.A. Nat. Neurosci. 9, 690–696 (2006).
 Vilares, I. & Kording, K. Ann. N Y Acad. Sci. 1224, 22–39 (2011).
 Pouget, A. Nat. Neurosci. 16, 1170–1178 (2013).
 Fetsch, C.R., Pouget, A., DeAngelis, G.C. & Angelaki, D.E. Nat. Neurosci. 15, 146–154 (2012).
 Beck, J.M. et al. Neuron 60, 1142–1152 (2008).
 Barthelmé, S. & Mamassian, P. Proc. Natl. Acad. Sci. USA 107, 20834–20839 (2010).
 Kamitani, Y. & Tong, F. Nat. Neurosci. 8, 679–685 (2005).
 Haynes, J.D. & Rees, G. Nat. Neurosci. 8, 686–691 (2005).
 Jehee, J.F.M., Brady, D.K. & Tong, F. J. Neurosci. 31, 8210–8219 (2011).
 Brouwer, G.J. & Heeger, D.J. J. Neurophysiol. 106, 2108–2119 (2011).
 Serences, J.T., Saproo, S., Scolari, M., Ho, T. & Muftuler, L.T. Neuroimage 44, 223–231 (2009).
 Appelle, S. Psychol. Bull. 78, 266–278 (1972).
 Girshick, A.R., Landy, M.S. & Simoncelli, E.P. Nat. Neurosci. 14, 926–932 (2011).
 Huttenlocher, J., Hedges, L.V. & Duncan, S. Psychol. Rev. 98, 352–376 (1991).
 Jazayeri, M. & Movshon, J.A. Nature 446, 912–915 (2007).
 Stocker, A.A. & Simoncelli, E.P. Adv. Neural Inf. Process. Syst. 20, 1409–1416 (2008).
 Wei, X.X. & Stocker, A.A. Nat. Neurosci. 18, 1509–1517 (2015).
 Kok, P., Jehee, J.F.M. & De Lange, F.P. Neuron 75, 265–270 (2012).
 Ester, E.F., Anderson, D.E., Serences, J.T. & Awh, E. J. Cogn. Neurosci. 25, 754–761 (2013).
 Brainard, D.H. Spat. Vis. 10, 433–436 (1997).
 Pelli, D.G. Spat. Vis. 10, 437–442 (1997).
 Sereno, M.I. et al. Science 268, 889–893 (1995).
 DeYoe, E.A. et al. Proc. Natl. Acad. Sci. USA 93, 2382–2386 (1996).
 Engel, S.A., Glover, G.H. & Wandell, B.A. Cereb. Cortex 7, 181–192 (1997).
 Jenkinson, M., Bannister, P., Brady, M. & Smith, S. Neuroimage 17, 825–841 (2002).
 Fischl, B., Sereno, M.I. & Dale, A.M. Neuroimage 9, 195–207 (1999).
 Kay, K.N., Naselaris, T., Prenger, R.J. & Gallant, J.L. Nature 452, 352–355 (2008).
 Jehee, J.F.M., Ling, S., Swisher, J.D., van Bergen, R.S. & Tong, F. J. Neurosci. 32, 16747–16753 (2012).
 Smith, M.A. & Kohn, A. J. Neurosci. 28, 12591–12603 (2008).
 Goris, R.L.T., Movshon, J.A. & Simoncelli, E.P. Nat. Neurosci. 17, 858–865 (2014).
Author information
Affiliations

Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, the Netherlands.
 Ruben S van Bergen &
 Janneke F M Jehee

Center for Neural Science and Department of Psychology, New York University, New York, New York, USA.
 Wei Ji Ma

Department of Psychology, Mississippi State University, Mississippi State, Mississippi, USA.
 Michael S Pratte
Contributions
R.S.v.B., W.J.M. and J.F.M.J. conceived and designed the experiments. R.S.v.B. collected data. R.S.v.B. analyzed data, with help from W.J.M., M.S.P. and J.F.M.J. R.S.v.B., W.J.M., M.S.P. and J.F.M.J. wrote the paper.
Competing financial interests
The authors declare no competing financial interests.
Author details
Ruben S van Bergen
Search for this author in:
Wei Ji Ma
Search for this author in:
Michael S Pratte
Search for this author in:
Janneke F M Jehee
Search for this author in:
Supplementary information
Supplementary Figures
 Supplementary Figure 1: Decoding results broken down for different ROIs (97 KB)
Decoding results for areas V1, V2, V3, and V1–V3 combined. (a) Decoded and presented orientations are strongly correlated in areas V1–V3 from a representative observer (r = 0.74, p ≈ 0). (b) Similar results were found across observers. That is, decoding performance, plotted as the acrosssubject mean of the correlation between actual and decoded orientations, was highly significant in areas V1, V2, V3, and V1–V3 combined (all p ≈ 0). (c) Posterior width reliably predicted the variability in decoded orientations in all visual ROIs (V1: p = 0.2×10^{–15}; for each of V2, V3 and V1–V3 combined: p ≈ 0). (d) The posterior distribution is reliably broader for more oblique orientations in areas V1 (p = 0.003), V2 (p = 0.011), V3 (p = 0.003) and V1–V3 combined (p = 0.008, see also Fig. 1b, main text). (e) Posterior width reliably predicts the trialbytrial variability in behavioral orientation estimates in visual areas V1 (p = 0.003) and V1–V3 combined (p = 0.021; see also Fig. 1c, main text), with a trend towards significance in area V2 (p = 0.09). (f) The strength of the behavioral bias away from the cardinal axes is significantly correlated with posterior width in areas V1 (p = 0.002), V2 (p = 0.005), V3 (p = 0.030) and V1–V3 combined (p = 0.017, see also Fig. 1d, main text), with smaller behavioral biases for decreasing decoded uncertainty. In all plots, error bars represent +/– 1 SE.
 Supplementary Figure 2: Comparison of different noise models (60 KB)
Comparison across different noise models. We considered four different covariance structures for the model described by equation 6. The first covariance structure was defined as:
where τ is a vector that models the standard deviation of each voxel’s Gaussian variability. The second covariance structure was specified by:
where ρ models variability shared globally across voxels, irrespective of their tuning preference. The third noise structure that we considered was defined as:
where σ^{2} specifies the variance of independent and identically normally distributed noise shared across neural populations of similar orientation preference. The final noise model additionally described local correlations between voxels due to the BOLD point spread function (PSF; see e.g. Parkes et al., Magn. Reson. Med., 2005):
This noise model assumed that the degree of shared variability due to the PSF decays exponentially with distance, with initial amplitude α and decay rate controlled by β, and where matrix D describes, for each pair of voxels, the absolute distance in millimeters between their center coordinates.
Model parameters were estimated using the fMRI data in a leaveonerunout crossvalidation procedure, using the twostep training procedure as described in Methods. After fitting each model to the training data set, we used the heldout testing data set to evaluate its performance on two relevant benchmark tests. (a) We first focused on each model’s ability to identify the presented orientation from BOLD activity in areas V1–V3. Orientation decoding performance, quantified as the circular correlation coefficient between the decoded and presented stimulus orientation, was significantly better for models 2 and 3 than for models 1 and 4 (pairwise Ztests; 2 vs. 1: Z = 9.11, p ≈ 0; 2 vs. 4: Z = 9.18, p = 1.2×10^{–10}; 3 vs. 1: Z = 9.85, p ≈ 0; 3 vs. 4: Z = 9.92, p = 3.4×10^{–12}). We found no reliable difference in orientation decoding performance between models 2 and 3 (Z = 0.74, p = 0.46). (b) We then evaluated each model’s ability to characterize the degree of uncertainty about orientation. To the extent that the decoded posterior distribution appropriately models the covariance in fMRI data, broader distributions should be linked to increased variability in the decoder’s estimates of the presented orientation. Accordingly, we divided each participant’s data into four bins of increasing posterior width, calculated the acrosstrial variability in the decoder’s orientation estimates for each of the bins, and computed the partial correlation coefficient between mean posterior width and variability in decoded orientations (while regressing out betweensubject variability). Interestingly, only models 2–4 successfully characterized a sufficient degree of uncertainty in the fMRI data (model 1: t(53) = –0.25, p = 0.81, model 2: t(53) = 2.11, p = 0.04; model 3: t(53) = 15.91, p ≈ 0; model 4: t(53) = 9.85, p = 1.4×10^{–13}). Furthermore, model 3 reliably outperformed models 2 and 4 on this test (pairwise Ztests on correlation coefficients; Z = 6.31, p = 2.8×10^{–10} and Z = 2.10, p = 0.035, respectively). Taking both benchmark tests together, these results indicate that the third model best captured the noise covariance in BOLD activity relevant to orientation decoding.
 Supplementary Figure 3: Correlation between decoder errors and behavioral errors (45 KB)
Correlation between errors in decoded orientation estimates and errors in behavioral orientation reports, with error bars corresponding to +/– 1 SE. For each participant, trials were sorted into four bins of increasing signed or absolute decoder error. Within each bin, we calculated both the mean error (signed or absolute) in decoded orientation and the mean behavioral error (signed or absolute). We then used a multiple linear regression analysis to compute partial correlations between decoder and behavioral errors, controlling for mean differences between observers. Signed decoder errors were not significantly correlated with signed behavioral errors (t(53) = 1.42, p = 0.16), nor were larger decoder errors reliably associated with larger behavioral errors (t(53) = 0.91, p = 0.12) in areas V1–V3. Why do we nevertheless find that the variance of the posterior distribution is linked to behavioral biases and acrosstrial variability in behavioral errors? To see why, consider that the errors themselves are two independent random variables. As such, the correlation between the errors must necessarily be relatively weak, even when the mean and variance of their underlying distributions are linked. This observation exemplifies the utility of our uncertainty metric, which directly reflects the variance of the underlying distributions.
 Supplementary Figure 4: Correlation between the estimated neural population response and behavioral variability (39 KB)
Estimating the neural population response. Does a channelbased approach (cf. [^{13,14}]) similarly reflect the degree of uncertainty about orientation? It is important to realize that the posterior probability distribution characterizes the amount of information contained in the pattern of BOLD responses, rather than providing a direct estimate of the neural population response. That said, our model does allow for the estimation of neural population responses at a single trial level. Specifically, the population response c is described as the (idealized) tuning curves of the population plus noise (cf. equation 1):
Thus, estimating c involves finding the most likely value for η by maximizing the joint likelihood . Differentiating this likelihood with respect to η gives the following expression for the maximum likelihood estimate (MLE):
With these equations in hand, we first computed for each independent test trial the most likely neural population response . We then calculated the dispersion (circular standard deviation across channels) of the estimated neural population response, as well as the amplitude of the channel most strongly tuned to the presented stimulus orientation. Trials were subsequently sorted into four bins of increasing dispersion or peak response amplitude value. Summary statistics (mean dispersion, mean peak amplitude, and behavioral variability) were computed across all trials in each bin. Multiple regression analysis was used to compute partial correlation coefficients between dispersion and behavioral variability, as well as between peak amplitude and behavioral variability (regressing out distance to cardinal axes and betweensubject variability). Interestingly, neither the dispersion, nor the peak amplitude, of the estimated population response reliably predicted behavioral variability (r = –0.05, p = 0.73 and r = –0.16, p = 0.25, respectively; in the figure, error bars indicate +/– 1 SE).
Why is the estimated neural population response a less reliable predictor of behavior than the posterior probability distribution? One reason may be that the posterior distribution combines many aspects of the population response in a single information metric. As such, the posterior distribution is more sensitive to changes in orientation information than any one property of the neural population response alone. It is also important to realize that reflects the most likely neural population response (i.e., a single estimate). BOLD activity, however, is rather noisy and typically consistent with a whole range of neural population responses. The posterior distribution explicitly reflects this full range of possibilities, thereby providing greater sensitivity to subtle changes in BOLD activity.
 Supplementary Figure 5: Control analyses for mean BOLD, head motion and eye movements (52 KB)
Control analyses. Partial correlation coefficients between decoded uncertainty and mean BOLD signal intensity, head motion, eye position and eye movements. We found no significant correlation between decoded uncertainty and any of these variables in areas V1–V3 (p = 0.16, p = 0.55, p = 0.27 and p = 0.20, respectively), indicating that gross BOLD amplitude, mean eye position (amount of) eye movements, and (amount of) subject head motion cannot account for the trialbytrial uncertainty in cortical stimulus representations. This furthermore rules out simple explanations in terms of the amount of attentional effort put into the task, as overall BOLD amplitude tends to increase with effort (Ress, Backus & Heeger, Nat Neurosci., 2000). Error bars represent +/– 1 SE.
 Supplementary Figure 6: Bias in orientation reports as a function of stimulus orientation (40 KB)
Behavioral orientation reports are biased away from the nearest cardinal axis. Plots show the average (signed) error across observers in the behavioral orientation judgments, as a function of stimulus orientation. Positive errors indicate clockwise deviations from the veridical stimulus orientation. For each observer, trials were binned based on stimulus orientation, and the average behavioral error was calculated within each bin. Error bars indicate +/– 1 SEM.
 Supplementary Figure 7: Effect of number of voxels on main results (93 KB)
Decoding results as a function of number of voxels included within the ROI. Within areas V1–V3, all voxels whose response to the localizer stimulus met a chosen threshold (xaxis, uncorrected pvalues) were selected for subsequent analysis. Shown is the correlation between (a) decoded and actual stimulus orientation (decoding performance), (b) decoded uncertainty and angle between stimulus orientation and nearest cardinal axis (oblique effect in decoded uncertainty), (c) decoded uncertainty and behavioral variability and (d) decoded uncertainty and behavioral bias. Error bars represent +/– 1 SE. All of these results proved robust to reasonable variations in the number of voxels included in the analysis.
 Supplementary Figure 8: Hemodynamic response function and decoding window (63 KB)
Hemodynamic response function and decoding window. (a) Time course of mean BOLD activity in areas V1–V3 over the course of a trial. Time points between 4–8 s were averaged for subsequent decoding analysis. This relatively short time window (4 s) was chosen in order to ensure that activity from the response window was excluded from analysis. (b) Temporally expanding the time window to 2–8s did not greatly affect any of our results (p = 0.004, p = 0.022 and p = 0.007, respectively). Error bars correspond to +/– 1 SE.
 Supplementary Figure 9: Simulation results supporting the twostep model parameter estimation procedure (138 KB)
Estimation of W conditioned on the assumption that σ = 0. We simulated data according to the generative model described by equation 6, with ρ = 0.05, σ = 0.3, and for five different (hypothetical) observers. These parameter values were based on actual values estimated from 18 real observers. Similarly, W was simulated (independently for each observer) by sampling at random from a weight matrix that was estimated from the data of one of the participants to ensure a realistic distribution of orientation tuning across simulated voxels. Each simulated pattern of BOLD activity contained information about stimulus orientation, with varying degrees of uncertainty due to noise. To assess trialbytrial uncertainty, we computed, for each trial of simulated data, the probability distribution over stimulus orientation given the parameters of the generative model (i.e. ). We took the circular standard deviation of this distribution as the actual degree of uncertainty in the data. We then asked how well our decoder would recover this uncertainty, having estimated the model parameters from the simulated BOLD patterns using the twostep procedure described in Methods. Panel (a) plots decoded uncertainty against the actual degree of uncertainty in the data. While the decoder is slightly biased towards smaller values of uncertainty (slope = 0.53), the correlation between actual and decoded uncertainty is reasonable (r = 0.79, p = 6.0×10^{–15}). (b) The simulations were repeated for a range of parameter values (σ and τ), in semioctave steps around the empirical values used in a (red cross). Decoded uncertainty becomes less accurate with larger degrees of noise (larger σ and τ), as more noisy data generally results in poorer parameter estimates. Overall, for realistic levels of noise, decoded uncertainty correlates well with the actual degree of uncertainty in the data. (c) To assess whether our approach is sensitive to fluctuations in neural response in particular, we additionally computed the uncertainty about orientation in the simulated data given only neural variability modeled by σ. Specifically, we computed for each trial of simulated data the posterior probability distribution , where (cf. equation 1) and σ = 0.3. The circular standard deviation of this distribution served as the actual degree of uncertainty given the neural response alone. Interestingly, the correlation between decoded uncertainty and actual uncertainty reduced only slightly, suggesting that τ and ρ contribute little to overall uncertainty. To see why τ and ρ have a relatively small influence on decoded uncertainty, consider that the noise modeled by sigma is positively correlated with changes in signal (tuning curves) – such signaldependent noise has the greatest impact on information (Smith & Kohn, J. Neurosci., 2008; Abbot & Dayan, Neural Comput., 1999; Averbeck, Latham & Pouget, Nat. Rev. Neurosci., 2006). In contrast, ρ is unrelated to voxel tuning, while the sources of noise modeled by τ can, to large extent, be averaged out across voxels. Altogether, these simulations confirm that our twostep parameter estimation approach captures a sufficient degree of uncertainty in the data.
 Supplementary Figure 10: Noise model parameter estimates (42 KB)
Noise model parameter estimates obtained from the fMRI data. (a) Distributions across subjects of the mean estimated values for ρ and σ (averaged across all training partitions of the data), shown as box plots. Boxes extend from the first to the third quartiles, with notches at the medians. Whiskers span the full range of the data except for outliers, which were defined as values deviating from the median by more than 1.5 times the interquartile range, and are shown separately as open circles. (b) Since τ contained a value for each voxel, we plot its estimates separately, showing the distribution of values across voxels within each participant. For clarity of exposition, no outliers are shown here (due to the large number of data). Otherwise, this panel follows the same conventions as in a.
PDF files
 Supplementary Text and Figures (2,985 KB)
Supplementary Figures 1–10 and Supplementary Table 1
Additional data

Supplementary Figure 1: Decoding results broken down for different ROIsHover over figure to zoom

Supplementary Figure 2: Comparison of different noise modelsHover over figure to zoom

Supplementary Figure 3: Correlation between decoder errors and behavioral errorsHover over figure to zoom

Supplementary Figure 4: Correlation between the estimated neural population response and behavioral variabilityHover over figure to zoom

Supplementary Figure 5: Control analyses for mean BOLD, head motion and eye movementsHover over figure to zoom

Supplementary Figure 6: Bias in orientation reports as a function of stimulus orientationHover over figure to zoom

Supplementary Figure 7: Effect of number of voxels on main resultsHover over figure to zoom

Supplementary Figure 8: Hemodynamic response function and decoding windowHover over figure to zoom

Supplementary Figure 9: Simulation results supporting the twostep model parameter estimation procedureHover over figure to zoom

Supplementary Figure 10: Noise model parameter estimatesHover over figure to zoom