Quantifying the immediate computational effects of preceding outcomes on subsequent risky choices

Forty years ago, prospect theory introduced the notion that risky options are evaluated relative to their recent context, causing a significant shift in the study of risky monetary decision-making in psychology, economics, and neuroscience. Despite the central role of past experiences, it remains unclear whether, how, and how much past experiences quantitatively influence risky monetary choices moment-to-moment in a nominally learning-free setting. We analyzed a large dataset of risky monetary choices with trial-by-trial feedback to quantify how past experiences, or recent events, influence risky choice behavior and the underlying processes. We found larger recent outcomes both negatively influence subsequent risk-taking and positively influence the weight put on potential losses. Using a hierarchical Bayesian framework to fit a modified version of prospect theory, we demonstrated that the same risks will be evaluated differently given different past experiences. The computations underlying risky decision-making are fundamentally dynamic, even if the environment is not.


SUPPLEMENTARY MATERIAL Study Details
The four studies included in this analysis had the same basic instructions and task structure including the initial endowment, the timing of decision and feedback events on each trial, the possible monetary amounts, and the task-related payment. The studies varied by participation fee ($15-$20), number of trials (140-180), number of days (1)(2), and external manipulation. For this analysis, we excluded choices made under any experimental manipulation and on day 2 to capture participants' first experience with the risky monetary decision-making task. In Study 1, 30 participants switched between two cognitive reappraisal strategies ("regulate" and "attend") during the risky monetary choice task 1 . In the "attend", or baseline condition, participants were instructed to think about each choice independent of the other choices. In the "regulate" condition, participants were instructed to think about their choices as a portfolio ("you win some, you lose some"). Study 2 (N=37) involved no experimental manipulations during the risky choice task 2 . Study 3 was a two-day, double-blind, placebocontrolled, within-subjects study in which 47 participants were given the beta-blocker propranolol or a placebo prior to the risky monetary choice task 3 . In Study 4, a two-day, withinsubjects study, 120 participants completed a cold-pressor task or a control manipulation (a warmwater bath) with equal probability prior to the risky monetary decision-making task on each day 4 . In total, there was 64,953 choices across 234 participants. For this analysis, we excluded the choices made in the "regulate" condition (4,200 trials from Study 1), in the propranolol condition (3,432 trials from Study 3), in the cold-pressor condition (8,968 trials from Study 4), and on day 2 (an additional 7,013 trials from Study 3 and 17,967 additional trials from Study 4), leaving a total of 23,373 trials across 151 participants. See Table S1 for summaries of the demographic and methodological differences across the four studies.

Additional MCMC Estimation Details
The Stan model code is available on the Open Science Framework: https://osf.io/npd54/

Priors
Sampling priors were selected to be uninformative as possible and were normal (mean, standard deviation), uniform (lower limit, upper limit), or cauchy (location, scale) distributions (see Table S1 below). Parameters were sampled in a different space than they were applied. A transformation was used to convert sampled values to applied values (see section below, Transformation) Normal(0,10) Cauchy(0,2.5) Table S1: Priors for the group-level mean and standard deviation for each of the main parameters in the prospect theory plus model.

Transformations
For stability of estimation, the MCMC model transformed the sampled baseline parameters λ, ⍴, and µ before applying them to the data in manner identical to that used previously 4 and similar to approaches used by others 5,6 , in effect implementing a lognormal structure for those parameters. These transformations served to gracefully implement the minimum number of practical bounds on parameter values, without which models would experience numerical faults (overflow; impossible values) that would prevent successful estimation. See Transformation Rationale below for more discussion. In all cases, the values discussed in the text and in plots reflect the applied values of these parameters (i.e. after the transformation).
First baseline values for the PT+ parameters λ, ⍴, and µ, all of which have theoretical lower bounds of 0, were sampled in unbounded 'sampling' space (bounds of [-infinity, +infinity]). To transform from sampling space to 'applied' space, these unbounded values were passed through an exponential (e.g. if the sampled value was R, exp(R) gave the applied value, ⍴). An exponential transform produces strictly positive values of ⍴ for all real values of R (that is, ⍴ is bounded [0, +infinity]), thereby meeting the basic requirement that values of λ, ⍴, and µ, be above 0. All effect sizes and plots reflect the applied (transformed, bounded) values of the parameters (that is, ⍴ not R).
The exponential transform was not applied to the decision bias parameter, which is theoretically unbounded.
Second, the update parameters (i.e. the δ θ parameters), were transformed using individual softmax-based functions to gracefully constrain parameters between lower and upper bounds symmetric around zero, while allowing sampling to occur smoothly in unbounded space. The softmax equations were built so adjustment terms in 'applied' space thus had lower/upper bounds of [-1,+1] (db and λ), and [-0.25, +0.25] (⍴ and µ). All effect sizes reflect values in these applied (transformed, bounded) spaces.
In all cases, the final 95% confidence intervals for all parameters did not approach their respective bounds, suggesting that these bounds, while effective in enabling model estimation, did not interfere with identification of the most likely values of these parameters. While one cannot definitively state that there are no values of the parameters outside these bounds more likely than those we sampled, it is also not possible to identify them, as model estimation is not stable without these reasonable, psychologically plausible, and commonly-used bounds. The model was coded in Stan (see Methods) -for complete model code, including all aspects of parameter transformations, see https://osf.io/npd54/.

Transformation rationale
We have previously published this hierarchical Bayesian implementation of prospect theory 4 , which is structurally similar to that used by others 5,6 . In essence, in this approach the 'sampled' space of the parameters for rho, lambda, and mu are unbounded (i.e. with bounds of [infinity, +infinity]), but the final parameter values applied to data (what we call "applied space"; after the use of the exponential, see above) are bounded [0, +infinity]. We use the exponential to implement a lower bound of 0 for three of our four parameters (rho, lambda, and mu), for two main reasons: 1) It smoothly implements a lower bound of 0 (which is required for rho, lambda, and mu), while leaving the parameters unconstrained in the positive direction, as they have no theoretical upper bound (even if their plausible, expected, and psychologically-likely values are generally lower).
2) It gracefully allows the summation of independent terms contributing to the value of a parameter on a given trial in unbounded, 'sampled' space (e.g. the effect of previous outcomes on the loss aversion parameter) while preventing those summations from under-flowing.

Interaction between the priors and transformations
As a result of the prior distributions going through an exponential to generate softlybounded final parameters, the priors favored lower values for each of the parameters. This approach was deemed reasonable for three reasons. First, lower values of these parameters are indeed psychologically most likely (e.g. rho values tend to be reported between .5 and 1.5; lambda values between 0.5 and 4). Second, the priors, after being put through the exponential transformation are still broad and relatively uninformative. Lastly, our MCMC sampling procedure discarded the first 5,000 samples (50%) of each of the twenty chains, effectively eliminating the influence of the selected priors on the final sampled posterior distributions.   Table S3. Generalized linear modeling results using the "lme4" package (R version 3.5.0; "lme4" version 1.1-21).

Model 8 results
Fixed effects