Evidence for model-based encoding of Pavlovian contingencies in the human brain

Prominent accounts of Pavlovian conditioning successfully approximate the frequency and intensity of conditioned responses under the assumption that learning is exclusively model-free; that animals do not develop a cognitive map of events. However, these model-free approximations fall short of comprehensively capturing learning and behavior in Pavlovian conditioning. We therefore performed multivoxel pattern analysis of high-resolution functional MRI data in human participants to test for the encoding of stimulus-stimulus associations that could support model-based computations during Pavlovian conditioning. We found that dissociable sub-regions of the striatum encode predictions of stimulus-stimulus associations and predictive value, in a manner that is directly related to learning performance. Activity patterns in the orbitofrontal cortex were also found to be related to stimulus-stimulus as well as value encoding. These results suggest that the brain encodes model-based representations during Pavlovian conditioning, and that these representations are utilized in the service of behavior.

parsimony, a classifier was simply trained on a binary categorization task (e.g. corresponding to reward (CS+) and non-reward (CS-) trial types), with 4 simulated Pavlovian conditioning sessions, with 30 trials per session, 15 of each trial-type, as in the empirical study. In each iteration of the simulation, we generated a multivariate dataset with normal noise and specified means (with a mean of 0.0, and standard deviation of 1.0, using the normal feature dataset data generator of the python mvpa2 library), yielding a dataset of 120 trails (samples), and 8 features (corresponding to the modal number of features in the above analyses of participant data).
To simulate within-subject variance in learning performance, we proceeded according to the following logic. In the experiment, we probed participants for whether they had acquired the Pavlovian contingencies in each of the sessions. For example, by asking them whether they would predict the delivery of juice or water after seeing a probe CS fractal. If participants answered this two-alternative question incorrectly, we assumed that they would also have made the incorrect prediction in a majority of trials from the relevant Pavlovian conditioning session. We refer to sessions for which participants answered incorrectly as "poor" learning sessions. In our simulations, we would therefore swap the classification target labels for a majority of the trials in a poor learn-ing session. Because our volunteers participated in four sessions of Pavlovian conditioning, we further investigated the effect of any possible number (0 -4) of poor learning sessions on classifier performance.
To illustrate the resulting impact on classifier performance (Supplementary Figure 1a), we simulated that participants may have entertained the incorrect prediction for various fractions ψ p of the total number of trials in poor learning sessions (.5 -1.0, in .1 increments). In addition, we also accounted for the fact that even in successful learning sessions, because participants could not know the Pavlovian contingencies in advance, they may still have entertained the incorrect prediction for a minor fraction of trails (ψ s , .1, .2, or .3). In sum, simulated average cross-validated classifier performance for participant n (ŷ n ) is a function (F ) of this participant's learning performance measures for the four sessions (X (n,1..4) ), and fractions of trials with incorrect outcome predictions in successful and poor learning sessions (ψ s n and ψ p n , respectively). To get a stable estimate of simulated classifier performanceŷ, we ran N = 1000 simulated participants for a given set X 1..4 of learning performance measures (subscripts n omitted to indicate that X 1..4 was held constant across N iterations):ŷ This revealed that simulated classification accuracy is the worst (Supplementary Figure 1a), if participants completely fail to reverse in half of the sessions, because of the cross-validation with two leave-out sessions.
In a second step, we performed an exploratory regression analysis (Supplementary Figure   1b), attempting to predict the average cross-validated classifier performance in each participant (y n ), given each participant's learning performance across the four sessions (X (n,1..4) ). Specifically, we treated the fraction ψ p and ψ s of trials in which participants entertained the incorrect prediction during poor and normal learning sessions, respectively, as free parameters in a ordinary least sum of squares (OLS) optimization: Note that the two free parameters (ψ s , ψ p ) were constrained such that they were held constant across all n participants and sessions, because we only had binary measures (e.g. correct vs. incorrect) for each participant and session, but no information regarding how close each participant was to acquiring the Pavlovian contingencies. This constraint severely limited the flexibility of the optimization procedure. Particle swarm optimization (PSO; pyswarm, version 0.6) with constraint support was used to find best-fitting parameters. To get an estimate for the robustness of the fit, we performed 100 iterations drawing random samples of 80% of participants in each iteration. On each iteration, particles were initialized with random starting values (0 <= ψ s < .5, .5 <= ψ p 1.0).
For the stimulus identity classifier and participants' explicit knowledge of stimulus-outcome con-

B Supplementary Figures
Supplementary Figure 6: Red overlay over three consecutive cortical slices of T1 image contrast indicates from which brain areas functional MRI data was acquired in all participants. Only voxels for which functional data was available for every participant were included in statistical analysis.      Table 6: Brain areas within which decoding accuracy of the Identity classifier was significantly (p < 0.005, minimum cluster extent threshold: 25 voxels) above chance, when classifier was trained on the dist. and tested on the dist. CS fractal.