Abstract
Optimal decisions across extended time horizons require value calculations over multiple probabilistic future states. Humans may circumvent such complex computations by resorting to easytocompute heuristics that approximate optimal solutions. To probe the potential interplay between heuristic and optimal computations, we develop a novel sequential decisionmaking task, framed as virtual foraging in which participants have to avoid virtual starvation. Rewards depend only on final outcomes over fivetrial blocks, necessitating planning over five sequential decisions and probabilistic outcomes. Here, we report model comparisons demonstrating that participants primarily rely on the best available heuristic but also use the normatively optimal policy. FMRI signals in medial prefrontal cortex (MPFC) relate to heuristic and optimal policies and associated choice uncertainties. Crucially, reaction times and dorsal MPFC activity scale with discrepancies between heuristic and optimal policies. Thus, sequential decisionmaking in humans may emerge from integration between heuristic and optimal policies, implemented by controllers in MPFC.
Introduction
In biological scenarios, decisionmakers should normatively evaluate the longterm consequences of their actions with respect to appropriate reward or cost functions. Foraging for food, for example, requires sequential decisionmaking to avoid starvation: foraging choices should ensure that current and future metabolic reserves remain within homeostatic boundaries. Humans in particular can use their quite sophisticated—but bounded—decisionmaking capabilities to take multiple future outcomes into account^{1,2,3,4}. For instance, when current conditions are bad, decisionmakers need to determine whether to forage immediately, or to wait until conditions improve; if current internal energy resources are low, however, waiting may lead to starvation. Conceptually similar multistep decision problems arise in many different realworld contexts, for example, in business decisions when investors have to balance immediate and delayed threats of bankruptcy.
Sequential decisionmaking requires searching over a tree of probabilistic future states. During foraging, avoiding starvation depends on the success of current and future foraging attempts, on the internal energy state of the decisionmaker, as well as on momentary and expected foraging opportunities. A modelbased tree search should take all these variables into account—with more branches to be evaluated the longer the considered time horizon. These intricate computations may exceed the capacities of the human brain and entail costs in terms of opportunities foregone due to the passage of time^{5,6,7}. Decisionmakers could simplify such quandaries by resorting to modelfree heuristics^{8, 9} and by restricting the set of considered options^{10} and actions^{11}, in order to approximate optimal multistep policies without full tree search. For example, foraging options’ momentary probabilities or magnitudes imperfectly signal expected starvation probability. A considerable literature has addressed how humans make choices on simple, economic gambles with one to three steps^{12,13,14,15,16}, but it is much less well known how humans evaluate deeper sequences^{2, 17}.
Here, we asked to what degree humans rely on optimal vs. heuristic decision policies, and how these are computed neurally. We hypothesized optimal policy computation in multimodal regions of the medial prefrontal cortex (MPFC) known to integrate economic decision variables and to evaluate prospective outcomes^{18, 19}. Our results show that participants rely on a heuristic decision policy as well as on the optimal policy—both of which relate to bloodoxygenlevel dependent (BOLD) signals in MPFC regions. The discrepancies between the two employed policies scale with reaction times (RTs) and dorsal MPFC activity.
Results
Sequential decisionmaking was framed as virtual foraging
We developed a novel task to investigate how humans evaluate complex sequences of decisions (Fig. 1). This virtual foraging task embodied a Markov decision process^{20} (Fig. 2) and contained decisionmaking aspects of foraging while neglecting biophysical affordances such as actual energy consumption and physical efforts. In line with similar monkey and human tasks^{2,3,4, 17, 21,22,23,24,25}, it thus abstracts from actual foraging^{26, 27} to reflect generic properties of sequential decisionmaking.
In our task, participants were endowed with varying “energy resources,” depicted graphically as an energy bar. Participants were financially rewarded if they “survived” over a maximum of five time steps, called “days,” within a given miniblock of trials, called “forest” (Fig. 1). That is, participants only received a payoff if at the end of five consecutive decisions within a forest their final energy level was above zero. On each day, they decided between “foraging” and “waiting.” Foraging entailed either an energy gain or loss, with a graphically signaled probability. Waiting resulted in a certain but smaller energy loss. To create nontrivial optimal policies, one of two environmental conditions, called “weather types,” occurred randomly with a probability of 0.5 on a given day within a forest. The current weather type was signaled before participants made their choice. Each forest was specified by an initial energy endowment and a distribution of gain magnitudes and probabilities for each of the two weather types (Fig. 2; see Table 1 for a list of all heuristic variables and Table 2 for all variables related to the optimal policy; see Supplementary Note 1 for written task instructions).
We computed the a priori optimal policy that minimizes starvation probability according to a finite time horizon of five days for each combination of energy state, weather type, and day within a forest. Unless otherwise specified, we refer to the optimal policy according to the normative finite time horizon of five steps. To account for noise in the decision process, we used a probabilistic version of the optimal policy (i.e., the underlying value difference between foraging and waiting, which is the decision variable) to explain participants’ choices, RTs, and fMRI data.
Participant’s choices followed primarily a heuristic policy
We tested how well the optimal policy and ten different heuristic decision variables explained participants’ choices (see Table 1 and Table 2 for explanations and examples of these variables, respectively). Normatively, participants should compute the optimal policy according to the remaining number of days, out of the five days within a forest. By definition, heuristics are myopic and do not rely on a full evaluation of all possible upcoming outcomes. As heuristics, we considered variables related to the momentary foraging options (probability of foraging success, magnitude of the possible gain, their combination in form of expected value, EV), the current internal energy state (continuous or binary), the current weather conditions, and the number of days in the current forest. Additionally, we tested two heuristic policies (“change in states” and “winstayloseshift”) that derive from past states. We finally included a myopic policy that is optimal for a horizon of one day. By design, the optimal policy shares some predictions with several of the considered heuristics most notably the probability of foraging success (Supplementary Figs. 1, 2). However, average shared variance, derived on a trialbytrial basis across participants, was sufficiently low to dissociate which variables accounted for participants’ decisions (Supplementary Figs. 1, 2).
In a Bayesian model comparison of eleven logistic regression models, the signaled probability of foraging success emerged as best single predictor of participants’ choices across two samples of participants (fMRI sample: n = 28; Bayesian model comparison, protected exceedance probability of winning model, PEP = 0.984; Fig. 3a; Supplementary Table 1; behavioral sample: n = 21; PEP = 0.999; Supplementary Fig. 3a; Supplementary Table 2). Visual inspection of plots showing posterior predictive checks demonstrate that, also qualitatively, no other model with a single predictor captured choice data as well as the model with the probability of foraging success (fMRI sample: Fig. 4a; behavioral sample: Supplementary Fig. 4a). This included models relying on the optimal policy or on a heuristic based on expected value (see also Supplementary Note 2). Since decisions were not perfectly explained by this heuristic, we next asked if the optimal policy or any other heuristic contributed to participants’ choices.
The optimal policy explained additional variance in choices
After accounting for the best heuristic, choices in the larger fMRI sample (n = 28) were best explained by the a priori optimal policy with a horizon of five time steps (Bayesian model comparison, PEP = 0.936; Fig. 3b; Supplementary Table 1). This model with two predictors also won in an extensive model comparison of all 55 pairs of candidate variables (Bayesian model comparison, PEP = 0.840; Supplementary Table 3). Visual inspection of plots showing posterior predictive checks confirm that the winning model qualitatively captured participants’ behavior (see Fig. 3c, d; Fig. 4 for the winning model; see Supplementary Figs. 5–8 for extended posterior predictive checks of all models and for parameter estimates of a full model including all candidate variables). Additionally, models including interactions between the most important heuristics did not provide a better fit than the model with the optimal policy (i.e., the model including the probability of foraging success and the optimal policy reached PEP = 0.984; Supplementary Table 1). There was no decisive evidence (according to protected exceedance probability; PEP = 0.549) that a model with three variables explained choices better than the best model with two variables (Supplementary Table 1).
In the (smaller) behavioral sample (n = 21), we found that protected exceedance probabilities did not decisively distinguish between twovariable models including the optimal policy, or the binary energy state heuristic, respectively (Bayesian model comparison, PEP = 0.298). However, loggroup Bayes factors provided decisive evidence for the same model as in the fMRI sample, including the best heuristic and the optimal policy (loggroup Bayes factors between best and secondbest model >3; Supplementary Table 2; Supplementary Table 4; Supplementary Fig. 3c, d; Supplementary Fig. 4b, c; Supplementary Fig. 9). Thus, overall our model comparisons across the two groups decisively favored the optimal policy as a predictor of participants’ choice in our task. Nevertheless, it is possible that on specific subsets of trial types (e.g., with energy state one), different variables predicted behavior better (see Supplementary Figs. 6–8; see Supplementary Note 3 for exploratory analyses on the relationship of task behavior, IQ, and questionnaires scores).
Taken together, model comparisons suggested that participants’ choices were often predicted by a heuristic policy, but additionally choices followed the normatively optimal policy.
Participants’ choices took five future states into account
Using the optimal policy implies computing future states. We performed detailed analyses to determine how many days participants thought ahead in the task. Participants were incentivized to consider a time horizon of five days. In the main task they would often not finish five days (to enhance fMRI design efficiency), but they were instructed that their payoff depended only on a random subset of ten forests that would be completed outside the scanner. When comparing models with different horizons, we found that participants’ choices were indeed best described by a timehorizon of five days (Bayesian model comparison, fMRI sample: n = 28; PEP = 1.000; behavioral sample: n = 21; PEP = 0.986; see Supplementary Fig. 10 for an illustration of the different prescriptions made by optimal policies with different time horizons and Supplementary Fig. 11 for model comparisons). This finding was corroborated when analyzing subsets of trials in which policies with different horizons made opposing prescriptions (ttests against midpoint of choice proportion: fMRI sample: n = 28; absolute tvalues > 5.2, all p’s < 10^{−4}; behavioral sample: n = 21; absolute tvalues > 1.9, all p’s < 0.065; Supplementary Table 5).
Participants used the overall best heuristic available
Does the probability of foraging success constitute a useful heuristic? This heuristic shared the largest amount of variance (0.37) with the optimal policy on a trialbytrial basis and thus the two were related by design (Supplementary Figs. 1, 2). Additionally, simulations showed that basing one’s decision exclusively on this metric would lead to a lower starvation rate (0.15) than any other of the considered metrics (all other starvation rates >0.17; Supplementary Fig. 12; Supplementary Note 4). Thus, participants relied primarily on the best available heuristic variable. Notably, this heuristic was neither primed by our task instructions nor particularly visually salient (Supplementary Note 5).
Reaction times increased with choice uncertainties
Models of choice data indicate that participants used both a heuristic policy, i.e., the probability of foraging success, and the optimal policy. Consequently, we predicted that RTs should reflect the choice uncertainties associated with these two variables. (These uncertainties were calculated on the basis of independent data from the behavioral sample.) Indeed, RTs were slower when choice uncertainties were high (see Fig. 5a–d for the fMRI sample and Supplementary Fig. 13a–d for the behavioral sample; see Supplementary Fig. 6 for mean parameter estimates and Supplementary Table 6 for statistics from a linear mixed effects model; fMRI sample: n = 28; uncertainty of probability of foraging success: t = 15.34, p < 10^{−15}; uncertainty of optimal policy: t = 2.68, p < 0.05; behavioral sample: n = 21; uncertainty of probability of foraging success: t = 5.80, p < 10^{−5}; uncertainty of optimal policy: t = 1.86, = 0.062).
Following the heuristic entailed fast choices
Given that participants’ choices appear to use heuristic and optimal policy for making their decision, the question arises how these two are computed in relation to each other. As the heuristic is easier to determine than the optimal policy, we predicted that choices following the heuristic policy should be faster than those following the optimal policy. We identified the subset of trials in the fMRI sample in which the two policies made opposite prescriptions. More specifically, we computed the policies from the behavioral sample and applied these policies to the fMRI sample to select trials with opposite choice prescriptions. Within this subset (mean proportion of trials over participants ± standard deviation, SD = 0.19 ± 0.02), mean RTs for trials in which participants’ choices followed the heuristic policy were faster than for trials in which choices followed the optimal policy (mean difference ± SD = 90.7 ± 109.2 ms; t(27) = 4.40; p < 0.001; p < 0.001; mean predicted choices according to the used policy did not differ between these trials; p > 0.4).
In addition, a linear mixed effects model of RT data provided evidence for a relatively more pronounced influence of the choice uncertainty of the heuristic compared to the choice uncertainty of the optimal policy (as can be seen by comparing the tvalues for the uncertainty of the heuristic vs. the tvalues for the uncertainty of the optimal policy; fMRI sample: 15.34 vs. 2.68; behavioral sample: 5.80 vs. 1.86; Supplementary Table 6). That is, choice uncertainty under the heuristic policy related more strongly to RTs than choice uncertainty under the optimal policy.
Reaction times increased with discrepancies between policies
Next, we addressed whether both policies are integrated. In this case, decisions should take longer when the two policies make discrepant prescriptions. We quantified these discrepancies between the two variables as the absolute differences in choice probabilities. Indeed, decisions were slower when discrepancies between the decision variables were larger (linear mixed effects model, fMRI sample: n = 28; t = 6.55, p < 10^{−5}; behavioral sample: n = 21; t = 2.94, p < 0.01). This effect was present in addition to influences of choice uncertainties (fMRI sample: Fig. 5e; behavioral sample: Supplementary Fig. 13e).
Visual inspection of plots showing posterior predictive checks indicate that logtransformed RT data were qualitatively captured by a model that included the heuristic and the optimal policies themselves, their associated choice uncertainties, and the discrepancies in the choice probabilities of the two policies (see Fig. 4; Supplementary Figs. 6, 7; Supplementary Fig. 13). Motivated by this model of RT data, we also tested whether interactions of choice uncertainties or discrepancy influenced choices but found no decisive effects (Bayesian model comparison, fMRI sample: n = 28; highest PEP = 0.521; Supplementary Table 1; behavioral sample: n = 21; highest PEP = 0.202; Supplementary Table 2).
MPFC reflected variables guiding choice
We next assessed how heuristic and optimal policies are neurally computed (on a trialbytrial basis). To do so, we implemented a main general linear model (GLM) that included the variables from the winning choice model and the RT model as parametric modulators during the choice phase: the probabilities of foraging success, the value differences between foraging and waiting according to the optimal policy (time horizon of five days), their associated choice uncertainties, and the discrepancies in choice probabilities, as well as logtransformed RTs. Given the central role parts of the MPFC in decisionmaking, we specifically focus on this region in the presentation of our results. All described clusters are wholebrain familywise error (FWE) corrected for multiple comparisons at p < 0.05 with a clusterdefining threshold of p < 0.001.
The momentary probability of foraging success, i.e., the variable underlying the heuristic policy, showed a positive relation with BOLD signals in a posterior part of the dorsal MPFC (DMPFC, extending into presupplementary motor area, preSMA; peak voxel x;y;z(MNI) = 8; 29; 50; t = 5.80), in bilateral intraparietal sulcus (IPS; left: x;y;z(MNI) = −42; −45; 51; t = 4.42; right: x;y;z(MNI) = 47; −42; 50; t = 5.37), and the left frontal pole (x;y;z(MNI) = −42; 47; −5; t = 4.28), among other regions (Fig. 6a, see Supplementary Table 7 for fMRI results in the choice phase). The same variable showed a negative relation with signals in the perigenual anterior cingulate cortex (ACC), extending into the ventral MPFC (VMPFC; x;y;z(MNI) = 6; 33; 6; t = 6.27; Fig. 6b).
The optimal policy showed a positive relation with activity in perigenual ACC (extending into VMPFC; x;y;z(MNI) = 6; 50; 6; t = 4.53) and midcingulate cortex (x;y;z(MNI) = 2; 14; 29; t = 5.31; Fig. 6c). That is, parts of the MPFC were relatively more active when waiting was favored by the heuristic and when foraging was favored by the optimal policy. This suggests an overall involvement of the MPFC in computing differences in choice value of the variables that explained participants’ behavior.
MPFC reflected choice uncertainties and discrepancies
Lower choice uncertainty of the heuristic was related to increased BOLD signals in an anterior part of the VMPFC (x;y;z(MNI) = 9; 59; −2; t = 6.24), dorsal MPFC regions (x;y;z(MNI) = 3; 56; 33; t = 4.50) as well as to the inferior frontal gyrus (IFG; x;y;z(MNI) = 48; 35; −5; t = 7.37) and the posterior cingulate cortex (x;y;z(MNI) = 15; −27; 39; t = 7.63), among other regions (Fig. 7a; Supplementary Table 7).
Lower choice uncertainty of the optimal policy scaled with activity in DMPFC, extending into ACC (x;y;z(MNI) = 6; 42; 38; t = 4.81), and in the IFG (x;y;z(MNI) = 48; 26; −9; t = 5.88; Fig. 7b; Supplementary Table 7). That is, we found increased BOLD signals with increasing choice certainty of both heuristic and optimal policy in regions of the MPFC.
Crucially, as in our RT data, we found evidence for an integrated computation of the heuristic and the optimal policy: DMPFC activity correlated in a trialbytrial fashion with the discrepancies between the two policies (i.e., the absolute differences in their decision variables; x;y;z(MNI) = −9; 36; 38; t = 8.25; Fig. 7c; Supplementary Table 7). This DMPFC region extended into preSMA and ACC. The same metric correlated with BOLD signals in bilateral dorsal striatum (left: x;y;z(MNI) = −11; 12; 3; t = 7.16; right: x;y;z(MNI) = 11; 20; −3; t = 7.66) and bilateral IFG (left:x;y;z(MNI) = −30; 26; −5; t = 6.17; right: x;y;z(MNI) = 50; 26; −11; t = 6.85).
Additional fMRI analyses corroborated MPFC involvement
All relationships of the relevant model variables—as described above for the main GLM—emerged with logtransformed RTs as parametric modulator in the GLM (Fig. 7d). Overall, the same regions described above were also identified in a second GLM, which additionally included participants’ choices themselves as parametric modulators during the choice phase (Supplementary Table 8). The primary qualitative difference between the GLMs with and without choices as additional parametric modulator was that in the GLM including choices the DMPFC cluster related to lower choice uncertainty of the optimal policy failed to reach significance with FWE correction for multiple comparisons at p < 0.05 (with a clusterdefining threshold of p < 0.001; cf. Fig. 7b).
For the sake of completeness, we set up a third GLM that only included participants’ choices as parametric modulator (please refer to Supplementary Fig. 14 and Supplementary Table 9).
In a fourth GLM, we specifically analyzed how the values of the chosen options related to BOLD signals (Supplementary Fig. 15; Supplementary Table 10). The probabilities of foraging success according to the options chosen by participants scaled positively with the left frontal pole (x;y;z(MNI) = −47; 39; 6; t = 4.63), in a similar region as described above for the probabilities of foraging success according to the presented foraging options (i.e., x;y;z(MNI) = −42; 47; −5; t = 4.28). The values of optimal policy according to the chosen options’ values were positively related to the VMPFC (x;y;z(MNI) = 2; 63; 11; t = 4.76), in a similar region as described above for the value differences according to the optimal policy (i.e., x;y;z(MNI) = 6; 50; 6; t = 4.53). These findings corroborate that BOLD signals in the left frontal pole and in the VMPFC were associated with variables on which participants based their choices.
Finally, classic reward regions^{14, 15} (including VMPFC, striatum, and posterior cingulate cortex) tracked the realized outcomes, that is the impact of participants’ decisions on their internal energy state (overall peak voxel of the cluster: x;y;z(MNI) = 0; −26; 48; t = 9.01; Supplementary Fig. 16; Supplementary Table 11).
Taken together, our results show that variables relevant for participants’ choices and RTs scale with activity in multiple brain regions involved in decisionmaking—in particular in ventral and dorsal MPFC areas.
Discussion
This study addresses the neural computations required to make sequential decisions during virtual foraging, when participants had to navigate above a lower homeostatic boundary over a number of consecutive steps with probabilistic outcomes. We show that humans rely both on an easytoaccess heuristic and on an optimal policy that integrates over probabilistic future states. More specifically, we demonstrate that participants took advantage of a modelfree heuristic available at the time point of decisionmaking. This heuristic of relying on the probability of foraging success performed best overall in explaining participants’ choices among a large set of alternatives (including the foraging options’ expected values). This finding is in line with studies showing that participants often base their choices on the overall probability of winning in another type of sequential decisionmaking task^{16, 28}. Importantly, choices in our task were also explained by the normatively optimal policy. Participants’ choices were therefore best explained by the combination of two metrics that constitute the end points of a spectrum between sloppy (but easy) and exact (but difficult) solutions.
We did not find decisive evidence that any linear combination of two candidate policies and variables explained our choice data better than the probability of foraging success and the optimal policy. Although it is theoretically possible that participants use a yet unknown decision policy (for discussion see refs. ^{29, 30}), such policies do not follow from the given task variables in an obvious way. Any such model would thus likely require higher complexity than the linear combination of the probability of foraging success and the optimal policy (or it would only apply to a less complex setting than the one investigated here). It is an interesting question for followup research whether on specific trial types (possibly undersampled here) participants may have used a more (or less) complex model. Also, it appears possible that in a more (or less) challenging task, participants may abandon the optimal policy in favor of a combination of two or more heuristics.
For our task, we can exclude that participants resorted to a simple heuristic of looking just one time step ahead, which would have been clearly suboptimal given that they finite time horizon was five steps in our task. Imposing different time constraints or varying the number of time steps in our task may alter the time horizon considered and shift the balance between heuristic and optimal computations. Evidence for the possibility of such dynamic alterations has been provided in a recent study by Keramati et al.^{31} which demonstrated that humans adaptively adjust the depth of planning and the reliance on habits in a threestep learning task—with time pressure leading to shallower planning.
Our behavioral data suggest interdependent—but partly distinct—computations of heuristic and optimal processes during sequential choice. But what is the relationship between the two policies? Analyses of RTs showed that the choice uncertainty of both policies—but in particular of the heuristic—slowed RTs. Intriguingly, the discrepancies in choice probability between the two policies also led to longer RTs, which provides crucial evidence for computation of heuristic and optimal policies, and their integration. We interpret these findings as pointing toward progressive computational processes such that the approximations provided by the heuristic are abandoned if the associated choice uncertainties turn out to be too high. That is, an insufficient choice certainty of the heuristic metric may suggest that it is worthwhile to engage in a deeper search in form of a fullblown optimal policy computation.
Integrated computation of heuristic and optimal policies could be understood in terms of a process that accumulates from increasingly sophisticated policies, until the associated choice uncertainties become sufficiently small. This could explain why the uncertainty of the heuristic had a larger effect on RTs than the uncertainty of the optimal policy, which may be computed later during the trial. Integration of opposing predictions may then engage an additional timeconsuming step, leading to increased RTs. In some cases, the requirement to integrate two specific types of information may be especially pertinent (for example at the energy state boundaries integrating information about the probability of foraging success and about the energy state can be crucial). Identifying the precise temporal requirements of information integration processes is an interesting and challenging avenue for future research.
Our fMRI data revealed that the heuristic variable, i.e., the probability of foraging success, was positively associated with BOLD signals in the frontal pole, the IPS, and a posterior part of the DMPFC. The same heuristic was negatively related to signals in a region of perigenual ACC extending into VMPFC. By definition, appropriate heuristics are related to the given task demands and therefore brain regions associated with any heuristic variable likely depend on the particular heuristic used. For example, the involvement of the IPS observed in the current study might be related to its role in processing numbers^{32}. On the other hand, frontal pole, DMPFC, perigenual ACC and VMPFC play roles in several decisionmaking process^{19}. Since participants were more likely to choose waiting rather than foraging when the probabilities of foraging success were small, the identified region of the perigenual ACC and VMPFC scaled positively with the relative value of “waiting” vs. “foraging,” as signaled by the heuristic. This finding may thus accord with the role of a similar region in a conceptually different sequential decisionmaking task: In the task used by McGuire and Kable^{33} participants had to adaptively decide how long to keep waiting for future rewards with different, uncertain timings. The temporal unfolding of the subjective value during different waiting periods was related to the VMPFC. In both tasks, “waiting” (either as a discrete action as in our task or as temporal persistence as in the task by McGuire and Kable^{33}) trades off current (opportunity) costs against potential gains in the future. Still, the fact that our analyses also identify regions outside the VMPFC that scale with the value of foraging (notably in the DMPFC) constitutes a difference between our findings and those by McGuire and Kable^{33}. Interestingly, the uncertainty of the heuristic variable was negatively (and not positively) related to BOLD signals in several regions, in particular in the posterior cingulate cortex, an anterior part of the VMPFC, dorsal MPFC, and IFG. Put differently, these regions showed a positive association with the “easiness” of making a decision according to the heuristic policy. Negative relations of an uncertainty metric have previously been identified a region of the posterior cingulate cortex, which was slightly more posterior than the cluster identified here^{12, 34}.
Multimodal integration regions such as MPFC and IFG were associated with the optimal policy and also the choice uncertainty of the optimal policy. These regions have often been observed in studies testing for brain activity related to modelbased learning processes^{35, 36}. Indeed, optimal decisionmaking in our task bears considerably resemblance with modelbased learning^{36,37,38} since both involve searching across a tree of probabilistic future states. Here, we demonstrate brain activity when participants evaluated decision trees with extended time horizons independent of the uncertainties arising during learning. In particular, we found that a dorsal region of the MPFC correlates negatively with the choice uncertainty of the optimal policy (i.e., positively with the “easiness” of making a decision according to the optimal policy). This region seems to be especially well positioned to integrated different types of decision signals related to reward values and actions^{18, 19}. An intriguing possibility is that this region may generally be related to the uncertainties in recursive evaluations of a tree of probabilistic future states, which is a key feature for inferring optimal solutions in many realistic tasks.
The discrepancies between the two employed policies showed a positive trialbytrial relationship with a prominent cluster in the region of the DMPFC, extending into the preSMA and the ACC (in addition to relationships bilateral dorsal striatum and IFG). The DMPFC cluster overlaps with regions classically associated with multiple types of decision discrepancies^{39, 40}. Thus, our finding relating discrepancies between the heuristic and optimal policies to a part of the DMPFC could potentially indicate that this DMPFC region becomes increasingly engaged when the computations of the two policies prescribe divergent choices such that progressive evidence accumulation and competition processes are required for making a decision.
Our virtual foraging task does not require exploring or inferring unknown, uncertain, or unobservable, states, which distinguishes it from learning tasks aimed at comparing explicitly signaled vs. previously trained values and tasks designed to assess modelbased vs. modelfree reinforcement learning^{6, 35,36,37,38, 41,42,43,44,45}. Our approach could be extended to include learning or information seeking such that the optimal policy would require reducing uncertainty about environmental states (cf. ^{45, 46}).
We have previously shown that formal models incorporating homeostasis maintenance can explain economic choices better than standard economic models^{4}. In the present study, we address the underlying choice mechanism. We show that humans minimize the probability of virtual starvation via combining heuristic and optimal policies. In general, our delineation of heuristic and optimal policy computations informs a substantial body of behavioral work supporting notions of nuanced tradeoffs between heuristic and optimal solutions^{47,48,49} or of finegrained specifications of rationality^{5, 8, 50, 51}. Our behavioral findings corroborate recent proposals that the optimality of decisionmaking depends on cognitive limitations of the decisionmaker as well as the energy and opportunity costs incurred during the decision process^{5,6,7,8, 49, 52}, such that a theoretically optimal modelbased system ceases to be optimal under some circumstances. Our fMRI results relate activity in the MPFC to the modelbased tree search implicated in the computation of the optimal policy. Brain activity associated with the discrepancies between heuristic and optimal policies points towards a mechanism of integrated computation that may well generalize over many decision scenarios with multiple probabilistic states. Thus, our study paves the way towards better understanding the computations of the neural systems involved in making ecologically optimal choices in the face of complex multistep decisionmaking scenarios.
Methods
Participants
Participants were recruited from a student population via mailing lists of local universities. Twentyone participants participated in the behavioral study (15 female; mean ± standard deviation (SD): age = 25.8 ± 4.3 years). Thirty participants were recruited for the fMRI study. Two had to be excluded (one due technical problems with the scanner and one due to excessive motion >4 mm). This resulted in a final sample of 28 participants (13 female; age = 23.5 ± 3.9 years) for the fMRI study. Due to time constraints, one participant in the fMRI sample only performed eight out of ten sessions and another only performed nine out of ten sessions. Participants were paid a showup fee (behavioral sample: CHF 20; fMRI sample: CHF 50) plus a variable amount (see Instructions and task). Both groups performed exactly the same task with the only difference that one group underwent MR scanning.
The study was conducted in accord with the Declaration of Helsinki and approved by the governmental research ethics committee (Kantonale Ethikkommission Zürich, KEKZHNr. 20130328). All participants gave written informed consent using a form approved by the ethics committee.
Mathematical framework of virtual foraging task
To probe sequential decisionmaking, we propose a toy scenario and a corresponding Markov decision process (MDP) for a huntergatherer or any foraging agent that aims at dynamically maintaining homeostasis over time. See Fig. 1 for an example trial and Fig. 2 for the corresponding transition matrix, and the associated state transition diagram. See Table 1 and Table 2 for lists of all variables described in this and the following sections. The decisionmaking agent has to keep its internal energy state s above zero, i.e., the agent “dies from starvation” upon reaching the energy state zero at any time step. Here, the energy state can have discrete values equaling one to five energy points (but our model easily extends to additional numbers of energy states without loss of generality). At each time step t, or “day,” the agent can chose to “wait” and incur a sure loss c_{w} (of one energy point) or it can “forage” in which case the agent probabilistically gains an amount g (of zero to four energy points) or incurs a cost for foraging c_{f} (of two energy points). We denote the probability of foraging success as p (i.e., the probability of gaining points during foraging). The probability of unsuccessful foraging and thus of incurring c_{f} is q = 1−p. The maximum energy is capped; in the highest energy state the agent cannot gain more but simply stays in the highest state if foraging is successful.
The agent “lives” within a given “forest” in which all relevant variables are specified (presented to participants during the “forest phase,” see Fig. 1). We included good and bad environmental conditions denoted as “good and bad weather types” that each occur with a probability of 0.5 on a given time step in a given forest. That is, per forest there are total of 12 states in the MDP: 6 (energy states) × 2 (weather types).
A core component of an MDP is the transition matrix between these different states (see Fig. 2 for one example). Starvation, i.e., the state of having zero energy is absorbing (and thus the entry in the transition matrix leading from energy state zero to energy state zero is 1). Waiting leads to a sure loss c_{w} of one point, which is formalized by the associated probabilities w in the offdiagonal below the main diagonal. Foraging can lead to either success (fp) or not (fq). These probabilities constitute the entries in the associated offdiagonals (depending on the magnitudes of g and c_{f}).
How should the agent’s optimal policy look? The agent should minimize the probability of starvation p_{starve}(n,s,t); i.e., it should minimize the probability of reaching zero energy points within a fixed and finite time horizon of n days when starting with the internal energy state s. In our case, the instructed finite time horizon n is always 5 days and the starting energy state s at the first day can be 2, 3, or 4 energy points. Participants were incentivized accordingly, i.e., they received a monetary payoff (for a random subset of forests), if their energy was above zero at day 5 and nothing otherwise. This corresponds to a simple implementation of a reward function within the framework of MDPs: All transitions to zero are associated with a “reward” of −1 and all other transitions with a reward of 0.
Starvation probability p_{starve}(n,s,t) and thus the optimal policy depend on the agent’s choice at the current time step t and the next n−1 time steps—in addition to the dependence on n and s. In our finitehorizon scenario, the starvation probability thus depends on the number of time steps t, i.e., days within a forest.
A rough and basic intuition for a more or less prototypical optimal policy under such circumstances is that in many types of forests the agent should forage when the weather is good and wait when the weather is bad; unless waiting leads to sure death on the next time step. When the energy state is high enough so that starvation is impossible within the remaining number of days (e.g., energy state is 4 and only 1 day is left of the 5 days within a forest), the agent is indifferent between the two choice options.
Derivation of optimal policy in Markov decision process
We derived the optimal policies analytically for a large number of different forests, i.e., for all combinations of p (from 0.1 to 0.9 in steps of 0.1) and gains g (from 0 to 4 in steps of 1) for a finite time horizon of n = 5 days. In our MDP, the possible policies consist of the probabilities of foraging f in each energy state s, environmental condition (“weather”), and each time step t (with the probabilities of waiting w = 1−f). To derive optimal policies in our finitehorizon scenario, we used backward induction: That is, we started from the final time step (i.e., day 5) and calculated the values of the two choice options (i.e., foraging or waiting) for each state for that last and final time step. These values depend on the possible transitions from the respective states (see the example transition matrix in Fig. 2a and the corresponding example state transition diagram in Fig. 2b). If the value of foraging is higher than the value of waiting, foraging is the deterministically better option in that state and at that time step—or vice versa. If both choice options have the same value, the optimal choice is indifferent between the two options. We then used the maximum of the two choice options’ values to calculate the values for the secondtolast time step and determined optimal choices. This procedure was repeated until arriving at the first time step (i.e., day 1).
The optimal policy thus depends on the time horizon considered. Our task was designed such that participants should use a time horizon of 5 time steps and participants were instructed and incentivized accordingly. Nevertheless, it is a possibility that participants use a different time horizon. Therefore, we calculated optimal policies for the following time horizons: n = 7, n = 6, …, n = 1. Participants may employ a longer (i.e., 7 or 6 days) or shorter (i.e., 4, 3, 2, or 1 days) time horizon than instructed. Time horizons are flexible in the sense that it is assumed that participants start in a new forest with a given time horizon n and then reduce their horizon on each day by one. If the horizon has reached one (i.e., n = 1), it will remain one. Supplementary Fig. 10 provides an illustration of how the proportions of actions prescribed by the optimal policy change according to different finite horizons.
Since the optimal policies binarize the value differences over the two choice options (either foraging or waiting is better, or they are exactly the same), they do not allow for variability in the decision process (i.e., in some cases waiting and foraging entail large value differences whereas in other cases the two choice options have quite similar values). We therefore used the continuous value differences between the two choice options as predictors of participants’ choices, RTs, and fMRI data. For brevity, we often use the term “optimal policy” to refer to the value differences between the foraging and waiting (according to the normative horizon of 5 steps in our task).
All calculations were carried out in MATLAB.
Instructions and task
Participants received detailed written task instructions, in which the different task components were introduced step by step (Supplementary Note 1). For each miniblock of trials called “forest,” participants were first presented with a screen that depicted the foraging options for the two weather types (3.5 s; Fig. 1). Foraging options were illustrated as a grid with ten subfields. Each of the ten subfields had a probability of 0.1 to become realized. The number of colored dots within the subfields indicated the foraging costs (always 2 points) or foraging gains (ranging from 0 to 4). Sides were counterbalanced for the two weather types. An energy bar at the bottom of the screen depicted the starting energy state (which was 2, 3, or 4 points). After a fixation period (3.5 s), participants had to choose between one of the two foraging options (which represented either good or bad weather) and waiting, which was symbolized by a single dot depicting the sure energy loss of one point. Sides were counterbalanced for the two options. Participants had a maximum of 2 s to make their choice, which was then indicated by an asterisk. If they failed to respond or pressed a wrong button, the words “Too slow or wrong button” appeared. After an interval of 1 s, participants saw the outcome of their choice (1 s): If they had foraged one of the ten subfields turned yellow and the corresponding energy points were added to or subtracted from their energy bar. If they had waited, one point was subtracted from the energy bar. After a variable fixation interval (between 0.5 and 3.8 s), a new day or a new forest was depicted. Participants were not explicitly cued about the current day within a forest; but they knew that they always remained a maximum of five days within a forest (so they could count down the number of days when entering a new forest). If participants had died from starvation, an empty energy bar was depicted and they were asked to press one of the two buttons to elicit a motor response. The task was presented using the MATLAB toolbox Cogent (www.vislab.ucl.ac.uk).
Participants performed a short first training session of four forests with five days. To ensure that participants could get accustomed to looking five days ahead, they performed a second training session, in which they remained for five days in each of 24 forests. Participants performed ten sessions of the actual experimental task either on a PC (behavioral sample) or in the MR scanner (fMRI sample).
Participants were incentivized to avoid starvation over five days in a forest. To strike a balance between design efficiency and the requirement to make participants integrate over a number of future time steps, the number of days in a forest followed an exponential distribution with a mean of 2.5 resulting in 40 days in 24 forests per session. Importantly, participants were instructed to always consider staying five days in a forest and were monetarily incentivized accordingly. In the end, we randomly selected one of the 24 forests for each of the ten sessions. To determine participants’ payment, they were again presented with the two weather types and their current energy state and completed the 5 days within these forests. For each forest in which they survived (regardless of the precise energy state) they received additional CHF 1.50.
Analyses of choice data and model comparison procedures
Of the total possible number of 400 days, participants in the behavioral sample had reached the starvation state on 22.2 ± 6.0 trials (behavioral sample) and 21.1 ± 5.9 trials (fMRI sample). In the remaining trials they did not make a response within the time limit in 8.6 ± 10.7 trials (behavioral sample) and 6.4 ± 8.6 trials (fMRI sample). This left 369.1 ± 13.0 trials (behavioral sample) and 368.3 ± 18.8 trials (fMRI sample) for analyses.
We used logistic regression models (implemented in the MATLAB function mnrfit) of the following form:
with the following form of the decision variable DV:
As policy, we first considered the following potential optimal or heuristic policies (see Table 1 and Table 2 for annotated lists): (1) the optimal policy according to a time horizon of 5 days (h5). In contrast to the optimal policy, the following variables should be regarded as heuristics since they do no necessitate integration over the relevant future time points: (2) momentary probability of foraging success p; (3) momentary magnitude of foraging gain g; (4) expected value (EV) of foraging; (5) current continuous energy state s (ranging from 1 to 5 points); and (6) binary energy state (indicating whether waiting would lead to sure death or not). Since waiting leads to sure death when the continuous energy state is one, the optimal policy never prescribes waiting in these situations (i.e., even when the probability of foraging success is small, the optimal policy always favors a nonzero starvation probability). Therefore, at a continuous energy state of one the prescriptions of the binary energy state variable are always in line with the optimal policy. We also included the following variables: (7) current weather type; (8) number of days past in a forest; (9) change between past and current energy states (within and across forests); and (10) a type of “winstayloseshift” (WSLS) strategy, which prescribes foraging if the energy state increased in the past trial (i.e., after a foraging win) and waiting if the energy state decreased (i.e., after waiting and after a foraging loss). In other words, WSLS is a binarized version of the change between past and current energy states. Additionally, we tested for a myopic optimal policy: (11) the optimal policy according to a time horizon of 1 day (h1).
Since the probability of foraging success emerged as best single predictor of participants’ choices, we tested whether any other decision variable explained remaining variance in models that included the probability of foraging success, p, as first predictor and one of the other policies as second predictor (Supplementary Tables 1, 2):
According to the same logic, we tested all possible models with two predictors (Supplementary Tables 3, 4) and additionally models with three predictors (Supplementary Tables 1, 2). We also tested interaction models of the general form (Supplementary Tables 1, 2):
For each model we approximated model evidence by calculating the Bayesian Information Criterion (BIC), which penalizes model complexity. We performed both fixedeffects and randomeffects analyses. The latter assume that different participants may use different models. We used the Bayesian model selection (BMS) procedure implemented in SPM12 (http://www.fil.ion.ucl.ac.uk/spm/) to calculate protected exceedance probabilities, which measure how likely it is that any given model is more frequent than all other models in the population^{53}.
Calculation of variables related to participants’ choices
Both RTs and brain activity may also relate to the choice uncertainties of the employed decision variables. Choice uncertainties can be quantified by the logistic functions that relate participants’ empirical choices to the decision variables (cf. Fig. 3; Supplementary Fig. 3). That is, choice uncertainty is higher at the inflection point of the logistic function (i.e., at the point at which participants are indifferent between the two choice options) than at the ranges of the decision variable where the values of the logistic function are close to zero (prescribing waiting) or close to one (prescribing foraging). As metrics of choice uncertainties, we therefore took the derivatives the logistic functions related to the two variables included in the winning model of participants’ choice: the probabilities of foraging success and the value differences according to the optimal policy (with a horizon of 5 days). To base the quantification of choice uncertainties for the fMRI sample on independent data, we used the mean parameter estimates of the choice models from the behavioral sample to derive the logistic functions.
To investigate the interplay between the two variables of the winning choice model, we were specifically interested in the discrepancies in the prescriptions of the two decision variables. Therefore, we took the absolute differences between the predicted choices under the heuristic of using the probabilities of foraging success and under the value differences according to the optimal policy. That is, we calculated the absolute differences between the points on the logistic functions derived from the mean parameter estimates of the choice models from the behavioral sample. We refer to this metric as “discrepancy” between the two policies.
We included choice uncertainties and discrepancies as trialbytrial estimates in RT and fMRI models (see below). Additionally, we explored whether choice uncertainties and discrepancy guided the use of the decision variables included in the winning model. Specifically, we tested the following interaction models (Supplementary Tables 1, 2):
For additional analyses, we used the relevant logistic functions derived from mean parameter estimates (from the behavioral sample) to select trials in which the heuristic policies of using the probability of foraging success and of using the EV made opposite prescriptions for choice. That is, we binarized the prescriptions of the two policies by splitting them into those above and below the midpoint of 0.5. Logistic functions were derived from the mean parameter estimates of the behavioral sample (Supplementary Note 2).
To analyze subsets of trials, in which different horizons of the optimal policy made opposing prescriptions, we simply selected trials in which the (a priori computable) optimal policy for a horizon of 5 time steps prescribed foraging and the optimal policy for another time horizon prescribed waiting; or vice versa (Supplementary Table 5; Supplementary Fig. 10).
Models of RTs
We analyzed logtransformed RTs using linear mixed effects models as implemented in the R package lmer^{54} (http://cran.rproject.org/web/packages/lme4/index.html). The independent variables in the mixed effects model were the probabilities of foraging success, the value differences of foraging vs. waiting according to the optimal policy (with a horizon of five days), the choice uncertainties of the two policies, and the discrepancies between the prescriptions of the two policies. Random effects for participants included a random intercept and random slopes for all variables. Since we had no hypotheses for interactions, we did not include any interaction terms as fixed or randomeffects. Significance levels of the fixedeffects were determined by performing loglikelihood tests, which compared the full model to models without the respective factor (Supplementary Table 6). Additionally, we performed the same analyses using untransformed RTs, which resulted in qualitatively similar results (mean ± SD of the skewness of the RT distributions: behavioral sample: untransformed 0.751 ± 0.404; logtransformed: −0.067 ± 0.928; fMRI sample: untransformed: 0.873 ± 0.335; logtransformed −0.038 ± 0.843).
To compare RTs in a subset of trials in which the heuristic policy of using the probabilities of foraging success and the value differences according to the optimal policy made opposing prescriptions, we binarized the prescriptions of the two policies by splitting them into those above and below the midpoint of 0.5. Logistic functions were derived from the mean parameter estimates of the behavioral sample. We then selected trials in which the two policies differed (mean ± SD of resulting number of trials per participant: fMRI sample: 70.4 ± 8.1; behavioral sample: 70.4 ± 6.8; the minimum number of trials included per participant was 47 trials).
MRI data acquisition
Data were recorded in a 3 T (Philips Achieva, Best, The Netherlands) MRI scanner using a 32channel head coil. Functional images were recorded using a T2*weighted echoplanar imaging (EPI) sequence (TR 2.1 s; TE 30 ms; flip angle 80°). A total of 37 axial slices were sampled for whole brain coverage (matrix size 96 × 96; inplane resolution 2.5 × 2.5 mm^{2}; slice thickness 2.8 mm; 0.5 mm gap between slices; slice tilt 0°). Imaging data were acquired in ten runs of 170 volumes each. The first five volumes of each run were discarded to obtain steadystate longitudinal magnetization. Field maps were acquired with a double echo gradient echo field map sequence, using 32 slices covering the whole head (TR 349.11 ms; TE 4.099 and 7.099 ms; matrix size, 80 × 80; inplane resolution 3 × 3 mm^{2}; slice thickness 3 mm; 0.5 mm gap between slices; slice tilt 0°). Anatomical images were acquired using a T1weighted scan (FoV 255 × 255 × 180 mm; voxel size 1 × 1 × 1 mm^{3}).
Preprocessing and nuisance regressors
All fMRI analyses were performed in SPM12. The FieldMap toolbox was used to correct for geometric distortions caused by susceptibilityinduced field inhomogeneities. Preprocessing of EPI data included rigidbody realignment to correct for head movement, unwarping, and slice time correction. EPI images were then coregistered to the individual’s T1 weighted image using a 12parameter affine transformation and normalized to the Montreal Neurological Institute (MNI) T1 reference brain template. Normalized images were smoothed with an isotropic 8 mm full width at halfmaximum Gaussian kernel.
We corrected for physiological noise using RETROICOR as implemented the MATLAB PhysIO toolbox^{55, 56} (open source code available as part of the TAPAS software collection: http://www.translationalneuromodeling.org/tapas/). We collected electrocardiogram (ECG), pulse oximeter, and breathing belt data during scanning. After quality checks, we used ECG and breathing belt data for noise correction in 19 participants, pulse oximeter and breathing belt data in two participants, and no physiological noise correction in six participants. The corresponding confound regressors as well as the six motion correction parameters estimated from the realignment procedure were entered as covariates of no interest. Regressors were convolved with the canonical HRF and low frequency drifts were excluded using a highpass filter with a 128 s cutoff.
General linear models
The three distinct phases of the task (forest, choice, and outcome phases; see Fig. 1) were entered as events with a duration of 0 s (i.e., as stick functions) into the GLMs. Choice and outcome phases in which participants had starved or for which they did not reply were not explicitly modeled.
We were mostly interested in the choice phase and ran a main GLM with a combination of variables that emerged in our analyses of behavioral and RT data. Specifically, the following variables were entered on a trialbytrial basis as parametric modulators of the choice phase: The probabilities of foraging success, the value differences of foraging vs. waiting according to the optimal policy (with a horizon of 5 days), the choice uncertainties of the two policies, the discrepancies between the prescriptions of the two policies, and RTs. A second GLM included participants’ choices (as a binary variable) as a parametric modulator in addition to the variables entered in the main GLM. A third GLM only included participants’ choices as a single parametric modulator. A fourth GLM included parametric modulators in terms of chosen (and unchosen) options (and not in terms of the presented options as in the first three GLMs). Specifically, this fourth GLM contained the a parametric modulator for the chosen values of the employed heuristic (i.e., the current probability of foraging success for trials in which the foraging option was chosen and a probability of zero when the waiting option was chosen) and a parametric modulator for the chosen values according to the optimal policy (i.e., the current values of the foraging or the waiting options) as well as parametric modulators for the corresponding unchosen values along with parametric modulators for participants’ choices and RTs.
In all four GLMs, the forest phase was parametrically modulated by the current energy state and the overall starvation probability across five days. The outcome phase was parametrically modulated by the change in energy state.
We report analyses in which regressors competed for variance (i.e., without serial orthogonalization). In the main GLM, the maximum average shared variance was below 0.5 for all combinations of regressors.
We performed a secondlevel onesample ttest on contrast images from all participants. All reported clusters are familywise error (FWE) corrected for multiple comparisons at p < 0.05 using the SPM random field theory based approach. The clusterdefining threshold was p < 0.001. At this voxelinclusion threshold the randomfield theory approach in SPM correctly controls the false positive rate^{57}.
Data availability
The behavioral data that support the findings of this study are publicly available at https://figshare.com/s/1a4d75cb4176a3fef040. The neuroimaging data that support the findings of this study are publicly available at https://neurovault.org/collections/3242/.
References
 1.
Pearson, J. M., Watson, K. K. & Platt, M. L. Decision making: the neuroethological turn. Neuron 82, 950–965 (2014).
 2.
Kolling, N., Behrens, T. E. J., Mars, R. B. & Rushworth, M. F. S. Neural mechanisms of foraging. Science 336, 95–98 (2012).
 3.
Shenhav, A., Straccia, M. A., Cohen, J. D. & Botvinick, M. M. Anterior cingulate engagement in a foraging context reflects choice difficulty, not foraging value. Nat. Neurosci. 17, 1249–1254 (2014).
 4.
Korn, C. W. & Bach, D. R. Maintaining homeostasis by decisionmaking. PLoS Comput. Biol. 11, e1004301 (2015).
 5.
Gershman, S. J., Horvitz, E. J. & Tenenbaum, J. B. Computational rationality: a converging paradigm for intelligence in brains, minds, and machines. Science 349, 273–278 (2015).
 6.
Keramati, M., Dezfouli, A. & Piray, P. Speed/accuracy tradeoff between the habitual and the goaldirected processes. PLOS Comput. Biol. 7, e1002055 (2011).
 7.
Boureau, Y. L., SokolHessner, P. & Daw, N. D. Deciding how to decide: selfcontrol and metadecision making. Trends Cogn. Sci. 19, 700–710 (2015).
 8.
Dayan, P. Rationalizable irrationalities of choice. Top. Cogn. Sci. 6, 204–228 (2014).
 9.
Dayan, P. How to set the switches on this thing. Curr. Opin. Neurobiol. 22, 1–7 (2012).
 10.
Donoso, M., Collins, A. G. E. & Koechlin, E. Foundations of human reasoning in the prefrontal cortex. Science 344, 1481–1486 (2014).
 11.
Bach, D. R. & Dayan, P. Algorithms for survival: a comparative perspective on emotions. Nat. Rev. Neurosci. 18, 311–319 (2017).
 12.
Bach, D. R. & Dolan, R. J. Knowing how much you don’t know: a neural organization of uncertainty estimates. Nat. Rev. Neurosci. 13, 572–586 (2012).
 13.
Mohr, P. N. C., Biele, G. & Heekeren, H. R. Neural processing of risk. J. Neurosci. 30, 6613–6619 (2010).
 14.
Bartra, O., McGuire, J. T. & Kable, J. W. The valuation system: a coordinatebased metaanalysis of BOLD fMRI experiments examining neural correlates of subjective value. Neuroimage 76, 412–427 (2013).
 15.
Clithero, J. A. & Rangel, A. Informatic parcellation of the network involved in the computation of subjective value. Soc. Cogn. Affect. Neurosci. 9, 1289–1302 (2014).
 16.
Venkatraman, V., Payne, J. W., Bettman, J. R., Luce, M. F. & Huettel, S. A. Separate neural mechanisms underlie choices and strategic preferences in risky decision making. Neuron 62, 593–602 (2009).
 17.
Kolling, N., Wittmann, M. & Rushworth, M. F. S. Multiple neural mechanisms of decision making and their competition under changing risk pressure. Neuron 81, 1190–1202 (2014).
 18.
Beckmann, M., JohansenBerg, H. & Rushworth, M. F. S. Connectivitybased parcellation of human cingulate cortex and its relation to functional specialization. J. Neurosci. 29, 1175–1190 (2009).
 19.
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E. & Behrens, T. E. Frontal cortex and rewardguided learning and decisionmaking. Neuron 70, 1054–1069 (2011).
 20.
Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, USA, 1998).
 21.
Hayden, B. Y., Pearson, J. M. & Platt, M. L. Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 14, 933–939 (2011).
 22.
Constantino, S. M. & Daw, N. D. Learning the opportunity cost of time in a patchforaging task. Cogn. Affect. Behav. Neurosci. Behav. Neurosci. 15, 837–853 (2015).
 23.
Bach, D. R. et al. Human hippocampus arbitrates approachavoidance conflict. Curr. Biol. 24, 541–547 (2014).
 24.
Bach, D. R. Anxietylike behavioural inhibition is normative under environmental threatreward correlations. PLoS Comput. Biol. 11, e1004646 (2015).
 25.
Bach, D. R. The cognitive architecture of anxietylike behavioural inhibition. J. Exp. Psychol. Hum. Percept. Perform. 43, 18–29 (2017).
 26.
Kamil, A. C. & Roitblat, H. L. The ecology of foraging behavior: implications for animal learning and memory. Annu. Rev. Psychol. 36, 141–169 (1985).
 27.
Stephens, D. W. Decision ecology: foraging and the ecology of animal decision making. Cogn. Affect. Behav. Neurosci. 8, 475–484 (2008).
 28.
Venkatraman, V., Payne, J. W. & Huettel, S. A. An overall probability of winning heuristic for complex risky decisions: choice and eye fixation evidence. Organ. Behav. Hum. Decis. Process. 125, 73–87 (2014).
 29.
Palminteri, S., Wyart, V. & Koechlin, E. The importance of falsification in computational cognitive modeling. Trends Cogn. Sci. 21, 425–433 (2017).
 30.
Nassar, M. R. & Gold, J. I. A healthy fear of the unknown: perspectives on the interpretation of parameter fits from computational models in neuroscience. PLoS Comput. Biol. 9, e1003015 (2013).
 31.
Keramati, M., Smittenaar, P., Dolan, R. J. & Dayan, P. Adaptive integration of habits into depthlimited planning defines a habitualgoaldirected spectrum. Proc. Natl. Acad Sci. USA 113, 12868–12873 (2016).
 32.
Nieder, A. & Dehaene, S. Representation of number in the brain. Annu. Rev. Neurosci. 32, 185–208 (2009).
 33.
McGuire, J. T. & Kable, J. W. Medial prefrontal cortical activity reflects dynamic reevaluation during voluntary persistence. Nat. Neurosci. 18, 760–766 (2015).
 34.
Bach, D. R., Hulme, O., Penny, W. D. & Dolan, R. J. The known unknowns: neural representation of secondorder uncertainty, and ambiguity. J. Neurosci. 31, 4811–4820 (2011).
 35.
Lee, S. W., Shimojo, S. & O’Doherty, J. P. Neural computations underlying arbitration between modelbased and modelfree learning. Neuron 81, 687–699 (2014).
 36.
Gläscher, J. P., Daw, N., Dayan, P. & O’Doherty, J. P. States versus rewards: dissociable neural prediction error signals underlying modelbased and modelfree reinforcement learning. Neuron 66, 585–595 (2010).
 37.
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Modelbased influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).
 38.
Wunderlich, K., Smittenaar, P. & Dolan, R. J. Dopamine enhances modelbased over modelfree choice behavior. Neuron 75, 418–424 (2012).
 39.
Botvinick, M. M., Cohen, J. D. & Carter, C. S. Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8, 539–546 (2004).
 40.
Nee, D. E., Kastner, S. & Brown, J. W. Functional heterogeneity of conflict, error, taskswitching, and unexpectedness effects within medial prefrontal cortex. Neuroimage 54, 528–540 (2011).
 41.
Cushman, F. & Morris, A. Habitual control of goal selection in humans. Proc. Natl Acad. Sci. USA 112, 13817–13822 (2015).
 42.
Daw, N. D., Niv, Y. & Dayan, P. Uncertaintybased competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).
 43.
Wunderlich, K., Dayan, P. & Dolan, R. J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 15, 786–791 (2012).
 44.
Fitzgerald, T. H. B., Seymour, B., Bach, D. R. & Dolan, R. J. Differentiable neural substrates for learned and described value and risk. Curr. Biol. 20, 1823–1829 (2010).
 45.
Stoll, F. M., Fontanier, V. & Procyk, E. Specific frontal neural dynamics contribute to decisions to check. Nat. Commun. 7, 11990 (2016).
 46.
Kaelbling, L. P., Littman, M. L. & Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artif. Intell. 101, 99–134 (1998).
 47.
Huys, Q. J. M. et al. Bonsai trees in your head: how the pavlovian system sculpts goaldirected choices by pruning decision trees. PLoS Comput. Biol. 8, e1002410 (2012).
 48.
Huys, Q. J. M. et al. Interplay of approximate planning strategies. Proc. Natl Acad. Sci. USA 112, 3098–3103 (2015).
 49.
Gigerenzer, G. & Gaissmaier, W. Heuristic decision making. Annu. Rev. Psychol. 62, 451–482 (2011).
 50.
Kacelnik, A. in Rational Animals? (eds Nudds, M. & Hurley, S.) 87–106 (Oxford University Press, Oxford, UK, 2006).
 51.
Griffiths, T. L., Lieder, F. & Goodman, N. D. Rational use of cognitive resources: levels of analysis between the computational and the algorithmic. Top. Cogn. Sci. 7, 217–229 (2015).
 52.
Summerfield, C. & Tsetsos, K. Do humans make good decisions? Trends Cogn. Sci. 19, 27–34 (2015).
 53.
Rigoux, L., Stephan, K. E., Friston, K. J. & Daunizeau, J. Bayesian model selection for group studies  revisited. Neuroimage 84, 971–985 (2014).
 54.
Baayen, R. H., Davidson, D. J. & Bates, D. M. Mixedeffects modeling with crossed random effects for subjects and items. J. Mem. Lang. 59, 390–412 (2008).
 55.
Kasper, L. et al. Cardiac Artefact Correction for Human Brainstem fMRI at 7T (Organization for Human Brain Mapping, San Francisco, CA, USA, 2009).
 56.
Kasper, L. et al. The PhysIO toolbox for modeling physiological noise in fMRI data. J. Neurosci. Methods 276, 56–72 (2017).
 57.
Eklund, A., Nichols, T. E. & Knutsson, H. Cluster failure: why fMRI inferences for spatial extent have inflated falsepositive rates. Proc. Natl Acad. Sci. USA 113, 7900–7905 (2016).
Acknowledgements
The Wellcome Trust Centre for Neuroimaging is supported by a strategic grant from the Wellcome Trust [09,1593/Z/10/Z]. C.W.K. was supported by the SFB TRR 169 during final stages of manuscript preparation. We thank Thorsten Kahnt, Gabriela Rosenblau, and Robb B. Rutledge for insightful comments on an earlier version of this manuscript as well as Giuseppe Castegnetti, Lars Kasper, Saurabh Khemka, Tessa Rusch, Matthias Staib, and Athina Tzovara for discussions, help with data acquisition, and technical support.
Author information
Affiliations
Contributions
C.W.K. and D.R.B. designed the experiment, developed the analysis procedures, and wrote the paper. C.W.K. collected and analyzed the data.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Additional information
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Korn, C.W., Bach, D.R. Heuristic and optimal policy computations in the human brain during sequential decisionmaking. Nat Commun 9, 325 (2018). https://doi.org/10.1038/s41467017027503
Received:
Accepted:
Published:
Further reading

Why and how the brain weights contributions from a mixture of experts
Neuroscience & Biobehavioral Reviews (2021)

Efficient Lévy walks in virtual human foraging
Scientific Reports (2021)

Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices
Social Cognitive and Affective Neuroscience (2020)

Dynamic integration of forward planning and heuristic preferences during multiple goal pursuit
PLOS Computational Biology (2020)

Representation of probabilistic outcomes during risky decisionmaking
Nature Communications (2020)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.