# Subjective estimates of uncertainty during gambling and impulsivity after subthalamic deep brain stimulation for Parkinson’s disease

## Abstract

Subthalamic deep brain stimulation (DBS) for Parkinson’s disease (PD) may modulate chronometric and instrumental aspects of choice behaviour, including motor inhibition, decisional slowing, and value sensitivity. However, it is not well known whether subthalamic DBS affects more complex aspects of decision-making, such as the influence of subjective estimates of uncertainty on choices. In this study, 38 participants with PD played a virtual casino prior to subthalamic DBS (whilst ‘on’ medication) and again, 3-months postoperatively (whilst ‘on’ stimulation). At the group level, there was a small but statistically significant decrease in impulsivity postoperatively, as quantified by the Barratt Impulsiveness Scale (BIS). The gambling behaviour of participants (bet increases, slot machine switches and double or nothing gambles) was associated with this self-reported measure of impulsivity. However, there was a large variance in outcome amongst participants, and we were interested in whether individual differences in subjective estimates of uncertainty (specifically, volatility) were related to differences in pre- and postoperative impulsivity. To examine these individual differences, we fit a computational model (the Hierarchical Gaussian Filter, HGF), to choices made during slot machine game play as well as a simpler reinforcement learning model based on the Rescorla-Wagner formalism. The HGF was superior in accounting for the behaviour of our participants, suggesting that participants incorporated beliefs about environmental uncertainty when updating their beliefs about gambling outcome and translating these beliefs into action. A specific aspect of subjective uncertainty, the participant’s estimate of the tendency of the slot machine’s winning probability to change (volatility), increased subsequent to DBS. Additionally, the decision temperature of the response model decreased post-operatively, implying greater stochasticity in the belief-to-choice mapping of participants. Model parameter estimates were significantly associated with impulsivity; specifically, increased uncertainty was related to increased postoperative impulsivity. Moreover, changes in these parameter estimates were significantly associated with the maximum post-operative change in impulsivity over a six month follow up period. Our findings suggest that impulsivity in PD patients may be influenced by subjective estimates of uncertainty (environmental volatility) and implicate a role for the subthalamic nucleus in the modulation of outcome certainty. Furthermore, our work outlines a possible approach to characterising those persons who become more impulsive after subthalamic DBS, an intervention in which non-motor outcomes can be highly variable.

## Introduction

The subthalamic nucleus (STN) is a subcortical nucleus of central pathophysiological relevance for Parkinson’s disease (PD). In PD, STN neurons display abnormal patterns of burst firing1 and low-frequency synchronisation of local field potentials2. By modulating this pathological network activity, subthalamic deep brain stimulation (DBS) for PD improves motor symptoms such as bradykinesia, tremor and rigidity3. However, STN-DBS has also been linked to neuropsychiatric symptoms, particularly impulsivity4,5, an issue of substantial clinical importance. The STN is a second input station to the basal ganglia, receiving direct cortical projections from the frontal lobe (‘hyperdirect’ pathway)6. This route permits basal ganglia inhibitory tone to be directly modulated by cortical regions. Functional and structural brain imaging support the role of this pathway in motor inhibition7,8. Following STN-DBS, participants with PD make commission errors (errors in which participants execute an erroneous action)9 and take longer to cancel an action10, suggesting an impairment in action restraint. Errors in the Stroop11 and random number generation tasks12 suggest increased sensitivity to cognitive interference and impaired task-switching. When faced with decisional conflict, patients with subthalamic DBS speed up rather than slow down their decision-making13,14. These findings suggest a role for the STN in ‘braking’ cognitive-associative circuits in the basal ganglia. It is not clear, however, whether such chronometric aspects of decision-making are sufficient to explain the complex picture of impulsivity after STN-DBS15. For example, subthalamic DBS may modulate contingencies in reinforcement learning16,17, suggesting a role for the STN in valuation.

A paradigmatic approach to inference and learning under uncertainty uses Bayes’ theorem to understand how prior knowledge (represented as a probability distribution known as the prior) is combined with new information from the environment (the likelihood) in order to update beliefs (the posterior). To obtain the posterior, a Bayesian agent inverts a ‘generative’ model (that describes how noisy sensory data result from environmental states); this corresponds to perception. Inferring environmental states from noisy sensory data allows the agent to plan actions that take into account the uncertainty of the environment24. Human behaviour often closely resembles those of Bayesian agents, for example, during low level sensory processing25,26, sensorimotor learning27,28, and higher-level reasoning29, although approximations to ideal Bayesian inference are likely required for most domains of cognition30,31,32.

Critically, a Bayesian perspective can accommodate multiple forms of uncertainty, beyond sensory noise. For example, the agent’s environment might change over time. In order to account for this environmental uncertainty (or volatility), Bayesian agents are able to modulate the rate at which they learn (update their beliefs). This learning rate can be linked to an agent’s encoding of volatility33,34,35. For instance, in more volatile environments, estimates of uncertainty (and thus learning rate) should be higher so that more emphasis is given to very recent information; at the same time, predicting the longer-term consequences of actions becomes more difficult. Additionally, uncertainty around how to best translate an uncertain belief about the environment into the selection of an action that will eventually lead to a reward is yet another source of noise that could contribute to observed stochasticity in behaviour. This link between uncertainty and decision-making may be of crucial importance for impulsivity23. Furthermore, individual differences in approximate Bayesian inference plausibly contribute to inter-individual variability in behaviour. Such differences can be quantified using models with subject-specific parameters concerning, for instance, the estimation of environmental volatility36 or the formation of unusually confident or ‘precise’ beliefs37.

In prior work, we found that subject-specific parameter estimates encoding different aspects of uncertainty (such as an estimate of environmental uncertainty, the tonic volatility ω) related to a clinical measure of impulsivity in a cohort of healthy subjects19. This finding supported a mechanistic understanding of impulsivity from the perspective of uncertainty. Based on these results, we sought to assess whether similar relationships between computational measures of uncertainty and impulsivity were observed in a population of persons with PD undertaking subthalamic DBS. Moreover, given the central role of this brain region in decision-making, we investigated whether receiving subthalamic DBS was associated with changes in the encoding of uncertainty that were connected to increases in impulsivity in certain persons. Finally, we assessed whether a perioperative computational characterisation of subjective aspects of uncertainty (estimates of environmental volatility and stochasticity of belief-response mappings) would be associated with longitudinal changes in impulsivity during clinical follow up. This latter question is clinically important as the non-motor outcomes from subthalamic DBS can be varied. Some centres advocate for the use of DBS to address impulsivity and compulsivity amongst persons with PD on dopamine agonist medication, as subthalamic DBS allows these agents to be substantially reduced or even withdrawn38. However, other centres report the emergence of harmful impulsivity subsequent to DBS, in persons with no prior history of clinically-significant psychiatric symptoms39,40,41,42,43. At present, there is little evidence to guide the identification of surgical candidates at risk of postoperative impulsivity4,44.

In this analysis, we employed a similar computational framework to that previously reported19, applying a hierarchical Bayesian model (the Hierarchical Gaussian Filter, HGF) to behavioural data from 38 participants with PD who played a virtual casino before and after subthalamic DBS. By allowing participants to vary their bet size, switch between slot machines and place ‘double or nothing’ bets, we could estimate how participants not only inferred the trial-by-trial probability of winning, but also updated higher-order beliefs about the fluctuations (volatility) of a slot machine’s winning probability. Similar to the rationale outlined in prior work19, we believe that a naturalistic paradigm engenders increased behavioural engagement, allowing us to quantify behaviour that has a higher fidelity to ‘real world’ impulsivity. Additionally, model-based estimates derived from the computational framework may afford us an individual profile of how each participant represented (and responded to) environmental uncertainty. We assessed these findings against standard measures of impulsivity derived from clinical assessment and questionnaires, focusing our attention on self-reported impulsivity as measured by the Barratt Impulsiveness Scale (BIS). Our computational analysis examined how DBS changes the manner in which persons with PD engage in Bayesian belief updating, and whether changes in subject-specific estimates of uncertainty before and after DBS relate to changes in impulsivity.

## Materials and Methods

### Subjects

Prior to the commencement of data collection, the full protocol was approved by the Human Research Ethics Committees of the Royal Brisbane & Women’s Hospital, the University of Queensland, the QIMR Berghofer Medical Research Institute and UnitingCare Health. All research was performed in accordance with relevant guidelines and regulations. All participants gave written, informed consent to participate in the study.

Thirty-eight participants with PD undertaking STN-DBS were consecutively recruited at the Asia-Pacific Centre for Neuromodulation in Brisbane, Australia. All participants met the UK Brain Bank criteria for PD45. No participants met the Movement Disorder Society criteria for dementia46. The PD subtype and the Hoehn and Yahr stage at device implantation was recorded47. Patients underwent bilateral implantation of Medtronic 3389 or Boston Vercise electrodes in a single-stage procedure. Stimulation was commenced immediately using microelectrode recording data to identify the optimal contact. Further contact testing took place over the following week as an inpatient, with participants returning to the DBS centre following discharge for further stimulation titration, guided by residual motor symptoms. Further details have previously been reported48,49.

### Neuropsychiatric assessment

Impulsivity amongst participants was assessed with patient and clinician-rated instruments prior to STN-DBS and subsequently 2 weeks, 6 weeks, 13 weeks and 26 weeks postoperatively (see Supplementary Fig. 1 for a study flowchart). A range of measures were obtained, to account for the fact that impulsivity is not a unitary construct. These included the Barratt Impulsiveness Scale 11 (BIS) and second-order factors attentional, motor and non-planning18; the Questionnaire for Impulsive-Compulsive disorders in PD Rating Scale (QUIP-RS)50; the delay discounting task51; the Excluded letter fluency task (ELF)52; and the Hayling test53. Further information on these instruments is detailed in the Supplementary Information. Additional neuropsychiatric symptoms were captured with the Beck Depression Inventory II (BDI)54; the Empathy Quotient (EQ)55; the Geriatric anxiety inventory (GAI)56; and the Apathy Scale57. For each self-report scale, participants were instructed to refer to ‘the last two weeks’, in order to obtain a measurement of current ‘state’. At each visit, PD motor symptoms were assessed using the UPDRS Part III motor examination58. Dopaminergic medication was recorded and converted to a levodopa-equivalent daily dose (LEDD) value59.

### Design and setting

Participants completed the experimental task prior to DBS and at 13-weeks post-DBS. Participants were ‘on’ medication and stimulation for all assessments. We opted against a counterbalanced ‘off’ and ‘on’ DBS assessment at the same visit for several reasons. First, our aim was to provide a naturalistic insight into the subtle behavioural changes that emerge as patients transition from dopaminergic therapies to subthalamic stimulation; changes in levodopa equivalent daily dose were included as co-variates in our analyses. Second, our experience is that many patients would not tolerate the DBS ‘off’ state without severe discomfort. Thirdly, despite allowing DBS washout, plastic network effects of chronic DBS may persist and contaminate findings in an on-off design.

We employed a modified version of an established slot machine gambling paradigm validated in healthy controls19. Subjects read an instruction screen and played through 5 training trials, after which they entered a ‘virtual’ casino, starting with 2000 AUD available to gamble and playing 100 trials (Fig. 1). The win-loss likelihood of the slot machines was predetermined and changed at regular intervals. On completion of the task, participants received a small monetary reward proportional to their total winnings. The naturalistic gambling task allows for risk-taking and impulsive behaviour to be expressed and offers four actions on each trial, each of which reflect exploration, and thereby, risk-taking.

1. (i)

Bet Increase: increasing the amount wagered on consecutive trials (minimum of 5 AUD per bet, no maximum)

2. (ii)

Machine-Switch: switching between slot machines (four machines in total)

3. (iii)

Casino Switch: cashing out and switching ‘virtual’ casino days

4. (iv)

Double-Up: engaging in a secondary double-or-nothing gamble on all win trials

As in our previous work19, these responses, together with trial-wise outcome information (wins/losses), served as the input for our computational models (for a brief summary, see below). Details on the paradigm and computational modelling can be found in prior work19 and the Supplementary Material.

### Computational modelling

#### The hierarchical gaussian filter (HGF)

The HGF is a hierarchical Bayesian model34,35 (Fig. 2) where each level of the hierarchy encodes distributions of environmental variables (in ascending complexity) that evolve as Gaussian random walks. The HGF is an extension of the model presented in Behrens et al.33, and describes an agent whose learning rate is a function of his or her uncertainty. In the HGF, an agent is assumed not only to represent current environmental contingencies, but also to track how these contingencies change over time (volatility), and to what degree volatility itself is constant (tonic volatility) or may change in time (phasic volatility). Importantly, the agent modelled in the HGF employs these representations to make predictions about emerging environmental fluctuations and future sensory feedback. Furthermore, the agent is able to encode the precision of each prediction and use these precision estimates to scale trial-wise updates of beliefs about the environment and its statistical structure. Each level of the HGF is coupled such that higher states determine how quickly the next lower state evolves, with the lowest hierarchical level representing sensory events.

Inversion of this ‘perceptual model’ produces subject-specific parameter estimates that determine the nature of the coupling between levels of the HGF. Inverting this model under generic (mean-field) approximations results in analytical belief-update equations, in which trial-wise belief updates are proportional to prediction errors (PEs) weighted by uncertainty (or its inverse, precision). The subject-specific parameters shape an individual’s approximation to ideal Bayesian inference, specifically how phasic and tonic volatility impacts trial-wise estimates of uncertainty at all levels of the hierarchy. Posterior estimates of HGF parameters can thus be regarded as a compact summary of an individual’s uncertainty processing during an experiment.

Furthermore, in a ‘response model’, trial-wise beliefs are probabilistically linked to observed trial-wise decisions. Inverting both perceptual and response models allows for estimating the parameters; this corresponds to Bayesian inference (of an observer) on Bayesian inference (of an agent)24. An informal description is given below and a formal summary is provided in the Supplementary Material.

#### The perceptual model

The HGF is used to infer how an individual subject learns about hierarchically-coupled environmental quantities under different forms of uncertainty (including volatility). In our case, the lowest level of the HGF, x1, represents the trial-wise binary outcome (win or loss) in the slot machine. This derives from a sigmoid transformation of x2 representing winning probability in logit space (i.e., whether the machine is currently ‘hot’ or ‘cold’ and likely (or not) to pay out). x2 evolves as a Gaussian random walk whose step size is a function f2(x3) of a third-level variable, x3, which performs a Gaussian random walk of its own. x3 represents the slot machine’s ‘volatility’, the speed at which it fluctuates between ‘hot’ and ‘cold’ states. The coupling function f2 between levels, contains subject-specific parameters κ and ω that determine an individual’s approximation to ideal Bayesian inference. Finally, the parameter ϑ at the highest level denotes how quickly volatility itself is changing (meta-volatility). A detailed derivation of the exact equations can be found in Mathys et al.34.

More concretely, in the context of our study, parameters ω and ϑ at the second and third level of the hierarchy, respectively, encode different aspects of subjective estimates of uncertainty. Specifically, these estimates concern environmental uncertainty, i.e., hidden fluctuations (volatility) of environmental states (for details, see Mathys et al.)34. These volatility estimates are potentially important for explaining the observed behaviour because they shape participants’ belief updates about the slot machine and their ensuing choices about gambling. Parameter ω represents a subject’s estimate of tonic volatility, i.e., how quickly a slot machine could be moving from a state where it is likely to pay out (running ‘hot’) to a state where it is not (running ‘cold’) and vice versa. Parameter ϑ encodes a subject’s estimate of meta-volatility, i.e., the tendency of volatility itself to change over time. Larger values of each parameter correspond to greater uncertainty in the subject’s perceptual inference process.

#### The response model

The response model maps a subject’s beliefs (obtained by inverting the perceptual model under given parameter values) to observed gambling behaviour. Here, we use a sigmoidal response model34; if this function is steep, there is a close relationship between current perceptual beliefs and betting behaviour. Conversely, a gentler sigmoidal slope results in a more stochastic mapping of beliefs to behaviour. This response function has a parameter, β, the decision ‘temperature’ (also known as the inverse temperature), that determines the steepness of the sigmoid and thus the degree of stochasticity in the belief-to-choice mapping. The larger the value of β, the steeper the function, and the more deterministic is the relationship between a subject’s belief and their actions. In this paper, we test the following two variants of this response model:

1. (i)

‘Standard’ HGF: β = constant, i.e., the mapping from beliefs to behaviour is fixed across the experiment. This parameter is estimated for each subject.

2. (ii)

‘Uncertainty-driven’ HGF: $$\beta =1/{\sigma }_{2}^{(k)}$$, where $${\sigma }_{2}^{(k)}$$ is the variance of the inferred probability of win on trial k. That is, the response behaviour dynamically adapts to the precision of the subjects’ belief about the current probability of winning.

#### Perceptual variable

Based on previous work that examined different computational models of our slot machine paradigm19, the perceptual variable used here was simple: a binary variable in which wins were represented by 1 and losses by 0. This binary representation of win or loss in the task allows for increased interpretability of model parameters in measuring uncertainty-updating and impulsive-responding in reaction to a binary win/loss outcome.

#### Response variable

The response variable is a binary representation of actions associated with risk taking. It is constructed using a logical OR operator on four choices during the slot machine paradigm: bet increases, machine switches, double-ups and casino switches. For each trial, the response variable takes a 1 when any of these four events occur, and 0 otherwise. For details, please see Supplementary Table 1.

While these actions might at first glance appear to relate to different behaviours, they all share a common theme in that they enhance outcome variance and thus the amount of risk the player takes in the game. For example, increasing bet size from one trial to the next results in higher reward variability in the trial outcome, thereby making the player more susceptible to larger wins and losses. In aggregate, these four actions relate to a player’s risk-taking tendencies (described further in Supplementary Section 1.4).

#### Reinforcement learning

As an alternative model, we used a classical associative learning model, Rescorla-Wagner (RW), often used in reinforcement learning (RL)60. The RW model updates the probability of a win on trial k by combining the probability on trial k−1 with a PE weighted by a constant learning-rate. Hence, in contrast to the HGF, the RW model does not have a dynamic learning rate over trials, nor can it account for different forms of perceptual uncertainty—essentially, the RW model corresponds to an HGF with a fixed learning rate. Here, we combine the RW learning rule with the same sigmoidal response model described above, with free parameter β, that we estimate on a subject-specific basis. This results in a model that is (i) structurally not dissimilar but less complex than the HGF and (ii) almost identical to the RL model used in a prior investigation of learning after STN-DBS17.

#### Model inversion

The HGF and RW models were inverted using population Markov-Chain Monte Carlo (MCMC) sampling61. Parameter estimation in the HGF is classically ‘fully Bayesian’ and requires a selection of priors, which influence parameter estimation to a lesser or greater degree. In order to minimise this influence, we used a novel empirical Bayesian inference scheme for the HGF where a Gaussian group-level distribution of parameters is constructed from samples across the group. This group-level empirical prior is then used to obtain posterior parameter estimates in each subject (Supplementary Fig. 2). Subject-specific point estimates for model parameters are calculated as the median value of the subject’s posterior distribution.

Given the clinical constraints of our investigation (to reduce any burden on the participants, we only used 100 trials per episode of gambling, i.e., only half as many as in our previous work)19, it was important to ensure that our parameter estimates were robust. Therefore, in order to verify that HGF parameter estimates reliably reflected subject-specific characteristics of uncertainty encoding and decision noise, we tested our ability to recover ground-truth parameter values from simulated response data. In order to assess parameter recoverability, we used three parameter values per parameter (shown in Supplementary Fig. 4) and generated a batch of 38 synthetic response variables based on these assigned values, using the underlying trace of the slot machine as the perceptual variable. We then inverted the HGF and explored the relationship of the recovered parameter estimates, using the median of the posterior, with the ground truth values. When estimating ω and ϑ, β was held fixed; conversely, when estimating for β, ω and ϑ were fixed. This process was repeated for 10 batches across each parameter.

#### Model comparison

As described above, we considered two competing hypotheses of how subjects might incorporate uncertainty into their choice of actions, i.e., two different belief-to-choice mappings in the response model for the HGF (the ‘Standard’ and ‘Uncertainty-driven’ models). These two versions of the HGF were compared with the RW model. As we were primarily interested in the pre-DBS to post-DBS change, we selected the winning model for the pre-DBS measurements. We then evaluated if the parameter estimates of that winning model changed postoperatively. Estimates of the negative free energy (log model evidence) were computed using thermodynamic integration61. The negative free energy balances goodness of fit with a complexity penalty. Group-level free energy estimates were compared to select a winning model.

### Data analysis

#### General considerations

All computational modelling and model inversion was performed using MATLAB (Mathworks), employing custom scripts developed from the HGF toolbox version 3 in the open source software TAPAS (http://www.translationalneuromodeling.org/tapas/). Multiple regression analyses were performed using the regstats function in the MATLAB Statistics Toolbox. For all analyses involving multiple comparisons, native p-values are presented, accompanied by Holm-Bonferroni correction at α = 0.05. To test the significance of individual regressors in multiple regression models, post hoc t-tests were performed.

Neuropsychiatric assessment data from baseline, prior to DBS, was compared with data gathered at 13-weeks post-DBS, when the gambling task was repeated. To test for differences in pre-DBS and post-DBS questionnaire scores and model parameter estimates, a paired t-test was employed when the data were normally distributed and the Wilcoxon signed-rank test otherwise, where distribution was assessed using the Lilliefors test. Gambling behaviours (such as bet increases and machine switches) were also compared at both intervals. Gambling behaviours were regressed against clinical measures of impulsivity to determine significant relations. After determining the winning computational model, model parameter estimates were extracted for each participant and regressed against clinical measures of impulsivity to determine significant associations and predictors of postoperative impulsivity. Based on this previous work showing a significant association between BIS scores and both slot machine behaviour and HGF-based estimates of uncertainty encoding19, we focused our analyses on the BIS and its subscales. Perceptual model parameters were extracted in log space: ω and β are naturally estimated in log space, since they are part of exponential terms in their respective equations (see equation 5 in the supplementary material).

From a clinical perspective, we were interested in examining whether changes in the computational characterisation of individual uncertainty estimates pre- to post-DBS were associated with clinically-relevant changes in impulsivity at any time point after DBS. Our strategy to attempt prediction of clinical outcomes follows the ‘generative embedding’ approach, in which individual predictions are not derived from measured data but from parameter estimates obtained by a generative model62,63. Importantly, stimulation-dependent changes in impulsivity may evolve in an unpredictable manner subsequent to DBS, related to variations in DBS programming over time (with considerable adjustments to stimulation in the first six postoperative months). Furthermore, the optimal BIS cut-off score for clinically-significant impulsivity varies by age and disease64, with only one existing investigation specific to a PD cohort65. Therefore, we examined whether individual changes in parameter estimates associated with the maximum postoperative increase in impulsivity, as measured by the BIS, compared to baseline, across six months of longitudinal follow up.

## Results

### Participant characteristics

Participants were a predominantly middle-aged sample, with a bias towards male gender and akinetic-rigid/mixed phenotype over tremor (Table 1). Most participants had bilateral disease with consequent impairment of functioning in their activities of daily living.

### Neuropsychiatric assessment Pre- and Post-DBS

Concerning symptoms of primary interest (Table 2), there was a small but statistically-significant group-level post-DBS decrease in impulsivity, as measured by the BIS Total, compared to baseline. There was also a significant reduction of motor symptoms assessed using the UPDRS Part III Motor Examination, with a corresponding significant reduction in the requirement for dopaminergic therapy (LEDD). There were no statistically-significant changes in other behavioural measures related to impulsivity, including the Hayling test, the Excluded Letter Fluency task and the delay discounting task. Comparable to the BIS, the QUIP-RS total score demonstrated a trend towards a reduction at 13-weeks post-DBS, but this did not reach significance.

For symptoms of secondary interest (and subscales), see Supplementary Table 2. There was considerable variance between subjects across assessment scores at each interval and within subjects across the course of longitudinal follow up (Supplementary Fig. 3).

The BIS and the BDI showed a significant positive correlation at each time point (ρpre = 0.46, p = 0.003; ρpost = 0.53, p < 0.001), and both showed (near-)significant changes from pre- to post-DBS (Table 2). Therefore, to rule out that impulsivity-related findings were driven by changes in depression, the BDI was included as a covariate when regressing behaviour and model parameter estimates against BIS scores. Whilst the LEDD is conceivably related to impulsivity, it did not correlate with the BIS total (ρpre = −0.126, p = 0.450; ρpost = −0.042, p = 0.799) and was therefore not included in these regression analyses. However, the QUIP and LEDD correlated strongly at both time points (ρpre = 0.42, p = 0.008); ρpost = 0.44, p = 0.005), with LEDD decreasing significantly post-DBS. There were no significant correlations between LEDD and the other measures of impulsivity (ELF Rule Violations, Hayling AB Error Score and Delay Discount K). Based on previous work using this task and modelling framework19, we focused our attention on exploring impulsivity as measured by the BIS.

### Gambling behaviour

#### Gambling behaviour Pre- and Post-DBS

At the group level, there were no significant differences in the behaviour of participants on the slot machine from pre- to post-DBS (Supplementary Table 3). Due to subjects not engaging in the ‘casino switch’ option, this variable was eliminated from regression analyses.

#### Pre-DBS Regression of BIS scores on Gambling Behaviour

We studied the relationship between pre-DBS gambling behaviour and pre-DBS impulsivity as measured by the BIS (Table 3). The BDI was included in this regression in order to control for changes in clinical state attributable to depressive symptoms. The overall preoperative model including the BDI total was significantly associated with the BIS total score [F(4,33) = 3.024, p = 0.031]. Post-hoc t-tests on task behaviour revealed that no behavioural variable was significantly related to the BIS individually. When subscales of the BIS were examined, gambling behaviour associated significantly with the BIS Attentional subscale [F(4,33) = 4.094, p = 0.008], where higher bet sizes corresponded to higher attentional impulsivity (t(37) = 2.303, p = 0.028) (Supplementary Table 4).

#### Post-DBS regression of BIS scores on gambling behaviour

The full model of postoperative gambling behaviour was also significantly associated with BIS total score [F(4,33) = 4.920, p = 0.003] (Table 3). Again, post-hoc t-tests revealed that no task behaviour was significant on its own. When subscales of the BIS were examined, gambling behaviour correlated significantly with the BIS Attentional subscale [F(4,33) = 8.123, p < 0.001]. Post-hoc t-tests revealed that higher bet sizes (t(37) = 2.604 p = 0.014) and more frequent double or nothing gambles (t(37) = 2.589 p = 0.014) corresponded to higher BIS Attentional scores (Supplementary Table 5).

Post-DBS, higher bets and more frequent machine switches were significantly associated with higher QUIP-RS scores (Supplementary Table 6). No other measures of impulsivity were significantly associated with pre- or post-DBS slot machine activity.

#### Regression of maximum BIS Increase on Pre- to Post-DBS changes in gambling behaviour

The change in gambling behaviours between the pre- and post-DBS time points were significantly associated with the maximum postoperative increase in BIS score. [F(4,33) = 3.516, p = 0.017] (Table 3). Post-hoc t-tests revealed that the change in bet behaviour significantly was associated with maximum BIS increase (t(37) = 2.866, p = 0.007). Additionally, the change in machine switch behaviour was also significantly associated with maximum BIS increase (t(37) = 2.219, p = 0.034). In other words, changes in betting and slot machine switching behaviours after DBS indexed changes in impulsivity as assessed by the BIS.

### Computational modelling

As described above, we were interested in evaluating the role of uncertainty and its association with postoperative changes in impulsivity in our cohort. We therefore first determined, using Bayesian model comparison, which of our three models best explained pre-DBS behaviour, before evaluating whether the parameter estimates of this winning model changed postoperatively and were associated with postoperative BIS scores. Bayesian model comparison selected the ‘standard’ HGF (with a subject-specific decision temperature in the response model) as the winning model, with a group-level Bayes factor of approximately 12.5, compared to the next best model (the Rescorla-Wagner model) (Fig. 3).

#### Parameter recoverability in the HGF

We tested for parameter recoverability in the HGF, finding that ground truth parameter values for ω and β could be recovered consistently, but we were unable to reliably recover ϑ (for details, see Supplementary Fig. 4). For this reason, we restricted the following analysis to parameters ω and β when exploring the association between parameter values and questionnaire-based measures of impulsivity.

#### Changes in model parameter estimates Pre- to Post-DBS

Estimates of the HGF model perceptual parameter ω significantly increased postoperatively (t37 = −61.328, p < 0.001), and estimates of β significantly decreased (t37 = 2.124, p = 0.04), implying larger subjective estimates of uncertainty (volatility) and greater stochasticity in the selection of responses after DBS (Table 5 and Fig. 4). ω represents a subject’s estimate about the tonic component of environmental volatility; i.e., how quickly the likelihood of winning on a given slot machine might be changing, while β represents the decision noise, or the stochasticity involved in the belief-to-choice mapping process.

#### Pre-DBS regression of BIS scores on model parameter estimates

The full regression model (including the estimates of preoperative perceptual and response parameters ω and β) was significantly associated with BIS total [F(3,34) = 3.372, p = 0.03] (Table 5). Post-hoc t-tests on model parameter estimates did not reveal any single parameter to be significantly related to the BIS on its own. When subscales of the BIS were examined, the full regression model associated significantly with the BIS Attentional subscale [F(3,34) = 3.314, p = 0.031], but again no single parameter was independently significant (Supplementary Table 7).

#### Post-DBS regression of BIS scores on model parameter estimates

The full regression model was significantly associated with BIS total [F(3,34) = 10.906, p < 0.001] (Table 5). Post-hoc t-tests revealed that ω was a significant regressor (t(34) = 3.761, p = 0.001). The positive regression coefficient for ω implies that the greater the subjective estimate of uncertainty (tonic volatility), the higher the BIS score (Fig. 4). In other words, although there was a group level decrease in BIS score pre-post DBS, at an individual level the greater postoperative estimate of uncertainty (i.e., the higher the estimated volatility of the slot machine’s winning probability), the greater the postoperative impulsivity. Thus, for participants with high subjective volatility estimates, there was likely to be a postoperative increase in BIS. When subscales of the BIS were examined, model parameter estimates correlated significantly with the BIS Attentional subscale [F(3,34) = 7.777, p < 0.001] and the BIS Non-Planning subscale [F(3,34) = 7.642, p < 0.001]. Post-hoc t-tests revealed that ω was also a significant regressor in both of these associations (t(34) = 2.874, p = 0.007 for BIS Attentional and t(34) = 2.561, p = 0.015 for BIS Non-Planning) (Supplementary Table 8).

#### Change in model parameters associating with maximum change in BIS

We were interested in whether individual pre- to postoperative changes in estimates of subjective uncertainty would correlate with the maximum postoperative change in impulsivity across six months of follow up post-DBS. The pre-to-post change in model parameter estimates was calculated as the preoperative minus the postoperative parameter estimate. In the case of ω, postoperative parameter estimates were significantly higher than preoperative values (in the case of ω, higher values imply higher uncertainty), therefore the Δω value for each subject is negative and implies a post-operative increase in uncertainty. With regards to β, postoperative parameter estimate values were significantly lower than preoperative values (in the case of β, lower values imply greater stochasticity), therefore the Δβ value for most subjects is positive, implying a postoperative increase in the stochasticity of the belief-to-choice mapping. Changes in model parameter estimates were associated with the maximum post-operative increase in the BIS across all longitudinal assessments over six months [F(3,34) = 3.987, p = 0.015] (Table 5). Post-hoc t-tests revealed that Δβ was a significant regressor (t(34) = 3.312, p = 0.002). The positive regression coefficient here implies that a post-operative increase in decisional randomness related to a greater maximum increase in BIS (Fig. 4).

#### Supplementary analyses

We examined whether dopaminergic medication dosage expressed as a standardised unit (LEDD) was connected to computational model parameters and whether changes in drug doses postoperatively were connected to changes in uncertainty encoding. There was no significant relationship between pre-DBS model parameters and pre-DBS LEDD [F(2,35) = 1.008, p = 0.375] or post-DBS model parameters and post-DBS LEDD [F(2,35) = 0.266, p = 0.768]. Additionally, the pre-to-postoperative change in LEDD did not significantly relate to pre-to-postoperative change in BIS (ρ = 0.28, p = 0.088). Also, the maximum decrease in LEDD across six months of follow up did not significantly relate to the postoperative maximum increase in BIS (ρ = 0.29, p = 0.073),

However, there was a significant relationship between the change in model parameter estimates and the maximum post-operative decrease in LEDD [F(2,35) = 4.032, p = 0.027] (Table 5). The BDI was not included in these regression models, as it is not a confound when examining a relationship between model parameter estimates and the LEDD. Post-hoc t-tests showed that Δβ was a significant regressor (t(34) = 2.832, p = 0.008). The positive regression coefficient here implies that a post-operative increase in decisional stochasticity was observed in patients who had a larger post-operative decrease in LEDD (Fig. 4). However, this relationship appeared to be driven by an outlier participant with a particularly large perioperative decrease in LEDD. When this participant was removed, the relationship was no longer statistically significant. We have removed the outlier in Fig. 4 but a full plot including the outlier can be found in Supplementary Fig. 5.

## Discussion

In this study, we employed a naturalistic gambling task and a hierarchical Bayesian model (for inference on subject-specific estimates of uncertainty) in order to investigate impulsive decision-making in participants with PD undertaking subthalamic DBS. Gambling behaviour associated with a ‘gold-standard’ questionnaire (BIS) measure of impulsivity, with post-DBS changes in gambling behaviours indexing postoperative changes in impulsivity. We also found that parameter estimates representing subjective estimates of environmental uncertainty (volatility) changed significantly from pre- to postoperative conditions. In particular, there was a significant increase in ω, that reflects a gambler’s estimate of how quickly the probability of winning on a given slot machine was changing (volatility), There was also a postoperative decrease in a second parameter, β, that captures the decision noise in a player’s belief-to-choice mapping. Notably, these model-based estimates of uncertainty related to postoperative impulsivity. The greater the postoperative estimate of ω, the greater the postoperative BIS score. In other words, the more a participant perceived the pay-out tendency of a slot machine to be changing after DBS, the more impulsive they rated themselves. Additionally, the higher the pre- to postoperative decrease in estimates of β, the higher the postoperative increase in BIS score across six months of longitudinal follow up. In other words, the more a participant became indiscriminate in their belief-to-choice mapping after DBS, the more impulsive they rated themselves.

Our gambling task utilised a multivariate response variable (bet increase, machine switch, casino switch and double-up) that captured different aspects of impulsivity and explorative behaviour. Furthermore, by employing a generative model that mapped observed responses to perceptual states, we were able to infer directly upon subject-specific parameters defining individual differences in uncertainty encoding. This is an important point of difference from a purely behavioural analysis, in which responses can have more than one (ambiguous) proximate cause. In the HGF, parameters are mathematically defined and have a concrete influence upon learning at different levels of the hierarchy (see Mathys et al. 2011, 2014 for simulations that demonstrate these effects)34,35.

What is the significance of individual differences in uncertainty encoding? Increased estimates of environmental uncertainty accelerate the rate of learning at higher hierarchical levels, which could engender maladaptive learning at lower levels of the hierarchy. A high learning rate suppresses the influence of top-down expectations, and may impair learning about probabilistically aberrant events. In a recent investigation employing the HGF to model surprise about unexpected events, persons with autism learned more quickly about environmental volatility than controls without autism66. However, at lower levels of the hierarchy, the tendency to believe that environmental instability is unstable resulted in smaller prediction errors (surprise) when events violated expectations. In other words, when the world is judged to be unstable and unpredictable, an agent differentiates less between expected and unexpected outcomes. This offers a similar but computationally distinct account of the stimulation-related learning changes described in a previous study17, in which reduced positive and negative instrumental outcome sensitivity was reported as a consequence of neurostimulation. Similar to prior work, we found a positive relationship between model-based estimates of uncertainty and impulsivity19,20,22,23. A plausible computational account of impulsivity is that high subjective uncertainty leads to lack of predictability and thus increases a tendency for short-term reward seeking and exploration.

We established that the ‘standard’ HGF best explained the gambling behaviour of our participants, in favour of a Rescorla-Wagner model or an ‘uncertainty-driven’ HGF. Importantly, the distinction between the ‘standard’ and ‘uncertainty-driven’ HGF models pertains only to the modelling of responses (the perceptual model is identical), in which the ‘standard’ HGF employs a fixed decision temperature and the ‘uncertainty-driven’ HGF a dynamic belief-to-response mapping based on online estimates of uncertainty (of beliefs about winning probability). These model comparison results suggest that our participants incorporate estimates of volatility into their prediction of reward probability but do not vary the stochasticity of their responses in response to these estimates. This is an interesting point of difference from the findings amongst younger, healthy males who completed a similar (albeit much longer) version of this task19 and future work will corroborate whether this finding of a static decision temperature is also observed amongst other neurodegenerative disorders.

In our participants, neurostimulation may interact with the physiology of the STN and alter the computations it implements. A tripartite functional organisation of the STN into limbic, associative and motor subregions is suggested by primate and human studies67,68, with electrode implantation targeted to the dorsolateral sensorimotor region to address motor symptoms of PD69. Yet, the small size of the STN means that dispersion of electrical charge from a stimulating contact in this region could still modulate subthalamic regions with greater connectivity to fronto-striatal networks. The more ventral and medial the stimulating contact, the more likely these networks are to be affected by DBS. Previous investigations have suggested that the site of subthalamic stimulation can modulate cognitive70 and psychiatric symptoms49,71,72. How could STN-DBS modulate uncertainty? From a computational perspective, the STN has been considered to implement a ‘delay’ on cognitive-associative circuits in the basal ganglia, allowing more information to be gathered to guide the most appropriate behavioural policy, suppressing impulsive and potentially error-prone responding13,14. It is possible that by modulating the decision threshold, STN-DBS could alter the bound for evidence accumulation and thus uncertainty in the representation of the reward environment73,74. Further work employing drift diffusion modelling to quantify rates of evidence accumulation and decision boundaries after STN-DBS may be illuminating, having previously helped to elucidate the mechanisms underlying hallucinations in PD75. Further work is also required to determine if the site of stimulation affects the magnitude of changes in uncertainty estimation observed here and specifically if cognitive-associative or sensorimotor regions of the STN are most implicated in these shifts.

We did not observe a cross-sectional relationship between dopaminergic medication (expressed as LEDD) and uncertainty encoding, nor did we observe a longitudinal relationship between LEDD and self-reported impulsivity. However, there was a longitudinal relationship between changes in model parameter estimates and the maximum reduction in LEDD during longitudinal follow up. Specifically, the greater the increase in decision noise (the greater the decrease in β), the greater the postoperative reduction in LEDD. It is difficult to be certain about whether this is a causal relationship and it may be an epiphenomenon of effective subthalamic DBS: One of the benefits of the STN (as opposed to other surgical targets in DBS for PD such as the internal segment of the globus pallidus) is that it allows for significant postoperative reduction in dopaminergic medication. Therefore, this apparent relationship could well be mediated by the effect of electrical stimulation – increasing indiscriminate responding and leading to a reduced requirement for dopaminergic therapy. Moreover, the finding that this relationship no longer held after the removal of an outlying participant decreases the confidence in this result.

There are likely to be fundamental differences in the computational operations subserved by the STN and dopamine in decision-making and impulsive behaviour. We have discussed the chronometric role of the STN is setting a decision bound and delaying impulsive choice, whereas dopamine is likely to have an essential role in reinforcement learning and reward evaluation76,77,78,79,80,81,82,83. In a non-surgical population, persons with PD withdrawn from medication display a characteristic impairment in reward learning and may show enhanced punishment sensitivity84. However, whilst dopamine replacement enhances the ability to learn from positive outcomes, learning from negative outcomes is impaired84. Thus, if postoperative LEDD reduction were a principal driver of a change in behaviour subsequent to DBS, then a selective impairment in positive outcome representation would be expected. However, from the HGF perspective, an agent with increased uncertainty at higher levels would be expected to show both decreased reward and punishment learning, as surprise to both positive and negative unexpected outcomes would be reduced (see Lawson et al.)66. This suggests that LEDD changes may have a secondary role, but further careful experiments will be necessary to address this question. For example, the goal of this behavioural analysis was to relate a computational marker of uncertainty (over all trials) to impulsivity, but future neuroimaging investigations could model trial-wise positive and negative reward prediction errors and relate this to trial-wise brain activity. Participants could also be tested prior to STN-DBS ‘on’ and ‘off’ medication (although in our cohort, participants were too impaired by their movement disorder to tolerate this and a group of less severely-affected individuals would be required).

We did not observe significant correlations between behaviour or parameters inferred from slot machine play with other estimates of impulsivity including the excluded letter fluency task, the Hayling test and the delay discounting task. This reflects the multifaceted nature of impulsivity, which may implicate discrete subcortical and cortical regions and may evidence differential patterns of expression amongst impulsive endophenotypes85,86. For example, the Hayling and ELF tasks are more commonly included amongst measures of task-switching and conflict interference, whilst the delay discounting task assesses impatience. Alternative paradigms may be required to capture participant-wise behaviour amongst these constructs.

We acknowledge methodological limitations of our investigation. The lack of a counterbalanced on-off stimulation design means that we cannot directly infer that stimulation underlies the observed changes in perceptual modelling observed in our participants (rather than, for example, practice effects or time). Specifically, for our participants the pre-DBS session was the first time they had performed the task, and so changes in postoperative behaviour could also be attributable to greater familiarity with the task and perhaps the inherent volatility of the reward structure. However, we suggest that a strength of our longitudinal design is that it is more reflective of the natural clinical course taken by persons with PD in the clinic. Moreover, our participants simply would not have tolerated an extended DBS washout and we hypothesise that the younger age of participants in the study of Seymour et al. may have facilitated their crossover design17. Nevertheless, it would be important to consider future experiments that could resolve this question, for example, selecting a cohort of younger PD participants who could tolerate a washout of stimulation, or testing a cohort of PD participants without DBS twice, 13-weeks apart.

Unfortunately, in this study, we were unable to utilise estimates of meta-volatility in our analysis as ϑ could not be robustly recovered from simulated data. This failure to recover ϑ might result from the limited number of trials (100) completed by each participant, which limits the amount of information that can be gathered to update estimates of this higher-level HGF parameter from the population prior. Again, the disability of our participant cohort prohibited a greater number of trials (as employed in previous studies using this paradigm)19, but this could be considered in future studies using younger or less severely-affected participants.

In summary, this study suggests that subjective estimates of uncertainty pertaining to environmental volatility and the stochasticity in belief-to-choice mapping change after subthalamic DBS for PD and relate significantly to postoperative impulsivity. Increased estimates of environmental uncertainty (volatility) and increased noise in the decision process may contribute to impulsivity as a clinically relevant form of maladaptive behaviour. Uncertainty elevates the learning rate and suppresses top-down expectations, which may blunt error signalling in a series of trial-wise outcomes. Similarly, a consistent decision rule with regards to acting on an internal model of the world is important to make appropriate decisions based on what has been learned. We therefore posit a cognitive mechanism for the genesis of impulsive behaviour in this population. Finally, our results demonstrate that a naturalistic assessment of gambling behaviour in a virtual casino is useful for investigating impulsivity in PD. The potential of our model to explain changes in impulsivity through game play could be most valuable in PD, given the significant, but poorly quantified risks relating to surgical (neurostimulation) and medical (dopamine agonist) treatments. If those at a higher risk of neuropsychiatric harm could be identified, this would improve the nature of treatment choice and informed consent and the effectiveness of clinical follow-up.

## Data Availability

The HGF toolbox is part of the open source TAPAS software and available for download at http://www.translationalneuromodeling.org/tapas. The gambling paradigm is provided for download on a git repository at https://github.com/saeepaliwal/breakspear_slot_machine.git. The analysis pipeline is provided at https://github.com/saeepaliwal/dbs_pd_analysis_pipeline.git. A de-identified data set containing neuropsychiatric assessment and gambling data can be provided by Dr Philip Mosley (Philip.Mosley@qimrberghofer.edu.au) on application, subject to institutional review board approval.

## References

1. 1.

Vila, M. et al. Evolution of changes in neuronal activity in the subthalamic nucleus of rats with unilateral lesion of the substantia nigra assessed by metabolic and electrophysiological measurements. The European journal of neuroscience 12, 337–344 (2000).

2. 2.

Brown, P. et al. Dopamine dependency of oscillations between subthalamic nucleus and pallidum in Parkinson’s disease. The Journal of neuroscience: the official journal of the Society for Neuroscience 21, 1033–1038 (2001).

3. 3.

Schuepbach, W. M. et al. Neurostimulation for Parkinson’s disease with early motor complications. New England Journal of Medicine 368, 610–622, https://doi.org/10.1056/NEJMoa1205158 (2013).

4. 4.

Mosley, P. E. & Marsh, R. The psychiatric and neuropsychiatric symptoms after subthalamic stimulation for Parkinson’s disease. Journal of Neuropsychiatry and Clinical Neurosciences 27, 19–26, https://doi.org/10.1176/appi.neuropsych.14040069 (2015).

5. 5.

Jahanshahi, M., Obeso, I., Baunez, C., Alegre, M. & Krack, P. Parkinson’s disease, the subthalamic nucleus, inhibition, and impulsivity. Movement disorders 30, 128–140, https://doi.org/10.1002/mds.26049 (2015).

6. 6.

Nambu, A., Tokuno, H. & Takada, M. Functional significance of the cortico-subthalamo-pallidal ‘hyperdirect’ pathway. Neurosci Res 43, 111–117 (2002).

7. 7.

Aron, A. R., Behrens, T. E., Smith, S., Frank, M. J. & Poldrack, R. A. Triangulating a cognitive control network using diffusion-weighted magnetic resonance imaging (MRI) and functional MRI. The Journal of neuroscience: the official journal of the Society for Neuroscience 27, 3743–3752, https://doi.org/10.1523/JNEUROSCI.0519-07.2007 (2007).

8. 8.

Rae, C. L., Hughes, L. E., Anderson, M. C. & Rowe, J. B. The prefrontal cortex achieves inhibitory control by facilitating subcortical motor pathway connectivity. The Journal of neuroscience: the official journal of the Society for Neuroscience 35, 786–794, https://doi.org/10.1523/JNEUROSCI.3093-13.2015 (2015).

9. 9.

Hershey, T. et al. Stimulation of STN impairs aspects of cognitive control in PD. Neurology 62, 1110–1114 (2004).

10. 10.

Obeso, I., Wilkinson, L., Rodriguez-Oroz, M. C., Obeso, J. A. & Jahanshahi, M. Bilateral stimulation of the subthalamic nucleus has differential effects on reactive and proactive inhibition and conflict-induced slowing in Parkinson’s disease. Experimental Brain Research 226, 451–462, https://doi.org/10.1007/s00221-013-3457-9 (2013).

11. 11.

Witt, K. et al. Deep brain stimulation of the subthalamic nucleus improves cognitive flexibility but impairs response inhibition in Parkinson disease. Archives of Neurology 61, 697–700, https://doi.org/10.1001/archneur.61.5.697 (2004).

12. 12.

Thobois, S. et al. STN stimulation alters pallidal-frontal coupling during response selection under competition. J Cereb Blood Flow Metab 27, 1173–1184, https://doi.org/10.1038/sj.jcbfm.9600425 (2007).

13. 13.

Frank, M. J., Samanta, J., Moustafa, A. A. & Sherman, S. J. Hold your horses: impulsivity, deep brain stimulation, and medication in parkinsonism. Science 318, 1309–1312, https://doi.org/10.1126/science.1146157 (2007).

14. 14.

Cavanagh, J. F. et al. Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold. Nature neuroscience 14, 1462–1467, https://doi.org/10.1038/nn.2925 (2011).

15. 15.

Florin, E. et al. Subthalamic stimulation modulates self-estimation of patients with Parkinson’s disease and induces risk-seeking behaviour. Brain: a journal of neurology 136, 3271–3281, https://doi.org/10.1093/brain/awt241 (2013).

16. 16.

Wagenbreth, C. et al. Deep brain stimulation of the subthalamic nucleus modulates reward processing and action selection in Parkinson patients. J Neurol 262, 1541–1547, https://doi.org/10.1007/s00415-015-7749-9 (2015).

17. 17.

Seymour, B. et al. Deep brain stimulation of the subthalamic nucleus modulates sensitivity to decision outcome value in Parkinson’s disease. Sci Rep 6, 32509, https://doi.org/10.1038/srep32509 (2016).

18. 18.

Patton, J. H., Stanford, M. S. & Barratt, E. S. Factor structure of the Barratt impulsiveness scale. Journal of Clinical Psychology 51, 768–774 (1995).

19. 19.

Paliwal, S., Petzschner, F. H., Schmitz, A. K., Tittgemeyer, M. & Stephan, K. E. A model-based analysis of impulsivity using a slot-machine gambling paradigm. Frontiers in human neuroscience 8, 428, https://doi.org/10.3389/fnhum.2014.00428 (2014).

20. 20.

FitzGerald, T. H., Schwartenbeck, P., Moutoussis, M., Dolan, R. J. & Friston, K. Active inference, evidence accumulation, and the urn task. Neural Comput 27, 306–328, https://doi.org/10.1162/NECO_a_00699 (2015).

21. 21.

Averbeck, B. B., O’Sullivan, S. S. & Djamshidian, A. Impulsive and compulsive behaviors in Parkinson’s disease. Annu Rev Clin Psychol 10, 553–580, https://doi.org/10.1146/annurev-clinpsy-032813-153705 (2014).

22. 22.

Djamshidian, A. et al. Decision making, impulsivity, and addictions: do Parkinson’s disease patients jump to conclusions? Movement disorders 27, 1137–1145, https://doi.org/10.1002/mds.25105 (2012).

23. 23.

Averbeck, B. B. et al. Uncertainty about mapping future actions into rewards may underlie performance on multiple measures of impulsivity in behavioral addiction: evidence from Parkinson’s disease. Behav Neurosci 127, 245–255, https://doi.org/10.1037/a0032079 (2013).

24. 24.

Daunizeau, J. et al. Observing the observer (I): meta-bayesian models of learning and decision-making. PloS one 5, e15554, https://doi.org/10.1371/journal.pone.0015554 (2010).

25. 25.

Weiss, Y., Simoncelli, E. P. & Adelson, E. H. Motion illusions as optimal percepts. Nature neuroscience 5, 598–604, https://doi.org/10.1038/nn858 (2002).

26. 26.

Petzschner, F. H., Glasauer, S. & Stephan, K. E. A Bayesian perspective on magnitude estimation. Trends in cognitive sciences 19, 285–293, https://doi.org/10.1016/j.tics.2015.03.002 (2015).

27. 27.

Wolpert, D. M., Ghahramani, Z. & Jordan, M. I. An internal model for sensorimotor integration. Science 269, 1880–1882 (1995).

28. 28.

Kording, K. P. & Wolpert, D. M. Bayesian integration in sensorimotor learning. Nature 427, 244–247, https://doi.org/10.1038/nature02169 (2004).

29. 29.

Tenenbaum, J. B., Griffiths, T. L. & Kemp, C. Theory-based Bayesian models of inductive learning and reasoning. Trends in cognitive sciences 10, 309–318, https://doi.org/10.1016/j.tics.2006.05.009 (2006).

30. 30.

Friston, K. The free-energy principle: a rough guide to the brain? Trends in cognitive sciences 13, 293–301, https://doi.org/10.1016/j.tics.2009.04.005 (2009).

31. 31.

Griffiths, T. L., Chater, N., Kemp, C., Perfors, A. & Tenenbaum, J. B. Probabilistic models of cognition: exploring representations and inductive biases. Trends in cognitive sciences 14, 357–364, https://doi.org/10.1016/j.tics.2010.05.004 (2010).

32. 32.

Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to grow a mind: statistics, structure, and abstraction. Science 331, 1279–1285, https://doi.org/10.1126/science.1192788 (2011).

33. 33.

Behrens, T. E., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. Learning the value of information in an uncertain world. Nature neuroscience 10, 1214–1221, https://doi.org/10.1038/nn1954 (2007).

34. 34.

Mathys, C. D. et al. Uncertainty in perception and the Hierarchical Gaussian Filter. Frontiers in human neuroscience 8, 825, https://doi.org/10.3389/fnhum.2014.00825 (2014).

35. 35.

Mathys, C., Daunizeau, J., Friston, K. J. & Stephan, K. E. A bayesian foundation for individual learning under uncertainty. Frontiers in human neuroscience 5, 39, https://doi.org/10.3389/fnhum.2011.00039 (2011).

36. 36.

Vossel, S. et al. Spatial attention, precision, and Bayesian inference: a study of saccadic response speed. Cerebral cortex 24, 1436–1450, https://doi.org/10.1093/cercor/bhs418 (2014).

37. 37.

Schwartenbeck, P. et al. Optimal inference with suboptimal models: addiction and active Bayesian inference. Medical hypotheses 84, 109–117, https://doi.org/10.1016/j.mehy.2014.12.007 (2015).

38. 38.

Lhommee, E. et al. Subthalamic stimulation in Parkinson’s disease: restoring the balance of motivated behaviours. Brain: a journal of neurology 135, 1463–1477, https://doi.org/10.1093/brain/aws078 (2012).

39. 39.

Mosley, P. E., Marsh, R., Perry, A., Coyne, T. & Silburn, P. Persistence of Mania After Cessation of Stimulation Following Subthalamic Deep Brain Stimulation. Journal of Neuropsychiatry and Clinical Neurosciences Published Online in Advance of Print (2018).

40. 40.

Smeding, H. M. et al. Pathological gambling after bilateral subthalamic nucleus stimulation in Parkinson disease. Journal of Neurology, Neurosurgery, and Psychiatry 78, 517–519, https://doi.org/10.1136/jnnp.2006.102061 (2007).

41. 41.

Lim, S. Y. et al. Dopamine dysregulation syndrome, impulse control disorders and punding after deep brain stimulation surgery for Parkinson’s disease. Journal of clinical neuroscience: official journal of the Neurosurgical Society of Australasia 16, 1148–1152, https://doi.org/10.1016/j.jocn.2008.12.010 (2009).

42. 42.

Halbig, T. D. et al. Subthalamic deep brain stimulation and impulse control in Parkinson’s disease. European journal of neurology: the official journal of the European Federation of Neurological Societies 16, 493–497, https://doi.org/10.1111/j.1468-1331.2008.02509.x (2009).

43. 43.

Amami, P. et al. Impulse control behaviours in patients with Parkinson’s disease after subthalamic deep brain stimulation: de novo cases and 3-year follow-up. Journal of Neurology, Neurosurgery, and Psychiatry 86, 562–564, https://doi.org/10.1136/jnnp-2013-307214 (2015).

44. 44.

Voon, V., Kubu, C., Krack, P., Houeto, J. L. & Troster, A. I. Deep brain stimulation: neuropsychological and neuropsychiatric issues. Movement disorders 21(Suppl 14), S305–327, https://doi.org/10.1002/mds.20963 (2006).

45. 45.

Hughes, A. J., Daniel, S. E., Kilford, L. & Lees, A. J. Accuracy of clinical diagnosis of idiopathic Parkinson’s disease: a clinico-pathological study of 100 cases. Journal of Neurology, Neurosurgery, and Psychiatry 55, 181–184 (1992).

46. 46.

Emre, M. et al. Clinical diagnostic criteria for dementia associated with Parkinson’s disease. Movement disorders 22, 1689–1707, https://doi.org/10.1002/mds.21507 (2007).

47. 47.

Hoehn, M. M. & Yahr, M. D. Parkinsonism: onset, progression and mortality. Neurology 17, 427–442 (1967).

48. 48.

Mosley, P. E., Breakspear, M., Coyne, T., Silburn, P. & Smith, D. Caregiver burden and caregiver appraisal of psychiatric symptoms are not modulated by subthalamic deep brain stimulation for Parkinson’s disease. NPJ Parkinsons Disease 4, 12, https://doi.org/10.1038/s41531-018-0048-2 (2018).

49. 49.

Mosley, P. E. et al. The site of stimulation moderates neuropsychiatric symptoms after subthalamic deep brain stimulation for Parkinson’s disease. NeuroImage: Clinical 18, 996–1006, https://doi.org/10.1016/j.nicl.2018.03.009 (2018).

50. 50.

Weintraub, D. et al. Questionnaire for impulsive-compulsive disorders in Parkinson’s Disease–Rating Scale. Movement disorders 27, 242–247, https://doi.org/10.1002/mds.24023 (2012).

51. 51.

Kirby, K. N., Petry, N. M. & Bickel, W. K. Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls. Journal of experimental psychology. General 128, 78–87 (1999).

52. 52.

Shores, E. A., Carstairs, J. R. & Crawford, J. R. Excluded Letter Fluency Test (ELF): Norms and Test–Retest Reliability Data for Healthy Young Adults. Brain Impairment 7, 26–32, https://doi.org/10.1375/brim.7.1.26 (2006).

53. 53.

Burgess, P. W., Shallice, T. & Thames Valley Test Company. The Hayling and Brixton tests. (Thames Valley Test Company, 1997).

54. 54.

Beck, A. T., Ward, C. H., Mendelson, M., Mock, J. & Erbaugh, J. An Inventory for Measuring Depression. Archives of General Psychiatry 4, 561–571, https://doi.org/10.1001/archpsyc.1961.01710120031004 (1961).

55. 55.

Baron-Cohen, S. & Wheelwright, S. The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. Journal of Autism and Developmental Disorders 34, 163–175 (2004).

56. 56.

Pachana, N. A. et al. Development and validation of the Geriatric Anxiety Inventory. International Psychogeriatrics 19, 103–114, https://doi.org/10.1017/S1041610206003504 (2007).

57. 57.

Starkstein, S. E. et al. Reliability, validity, and clinical correlates of apathy in Parkinson’s disease. The Journal of neuropsychiatry and clinical neurosciences 4, 134 (1992).

58. 58.

Goetz, C. G. et al. Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS): Process, format, and clinimetric testing plan. Movement disorders 22, 41–47, https://doi.org/10.1002/mds.21198 (2007).

59. 59.

Evans, A. H. et al. Punding in Parkinson’s disease: its relation to the dopamine dysregulation syndrome. Movement disorders 19, 397–405, https://doi.org/10.1002/mds.20045 (2004).

60. 60.

Rescorla, R. A. & Wagner, A. W. In Classical Conditioning II: Current Research and Theory (eds Black, A. H. & Prokasy, W. F.) 64–99 (Appleton-Century-Crofts, 1972).

61. 61.

Aponte, E. A. et al. mpdcm: A toolbox for massively parallel dynamic causal modeling. Journal of neuroscience methods 257, 7–16, https://doi.org/10.1016/j.jneumeth.2015.09.009 (2016).

62. 62.

Brodersen, K. H. et al. Dissecting psychiatric spectrum disorders by generative embedding. Neuroimage Clin 4, 98–111, https://doi.org/10.1016/j.nicl.2013.11.002 (2014).

63. 63.

Brodersen, K. H. et al. Generative embedding for model-based classification of fMRI data. PLoS Comput Biol 7, e1002079, https://doi.org/10.1371/journal.pcbi.1002079 (2011).

64. 64.

Stanford, M. S. et al. Fifty years of the Barratt Impulsiveness Scale: An update and review. Personality and Individual Differences 47, 385–395, https://doi.org/10.1016/j.paid.2009.04.008 (2009).

65. 65.

Voon, V. et al. Factors associated with dopaminergic drug-related pathological gambling in Parkinson disease. Archives of Neurology 64, 212–216, https://doi.org/10.1001/archneur.64.2.212 (2007).

66. 66.

Lawson, R. P., Mathys, C. & Rees, G. Adults with autism overestimate the volatility of the sensory environment. Nature neuroscience 20, 1293–1299, https://doi.org/10.1038/nn.4615 (2017).

67. 67.

Haynes, W. I. & Haber, S. N. The organization of prefrontal-subthalamic inputs in primates provides an anatomical substrate for both functional specificity and integration: implications for Basal Ganglia models and deep brain stimulation. The Journal of neuroscience: the official journal of the Society for Neuroscience 33, 4804–4814, https://doi.org/10.1523/JNEUROSCI.4674-12.2013 (2013).

68. 68.

Lambert, C. et al. Confirmation of functional zones within the human subthalamic nucleus: patterns of connectivity and sub-parcellation using diffusion weighted imaging. NeuroImage 60, 83–94, https://doi.org/10.1016/j.neuroimage.2011.11.082 (2012).

69. 69.

Wodarg, F. et al. Stimulation site within the MRI-defined STN predicts postoperative motor outcome. Movement disorders 27, 874–879, https://doi.org/10.1002/mds.25006 (2012).

70. 70.

Hershey, T. et al. Mapping Go-No-Go performance within the subthalamic nucleus region. Brain: a journal of neurology 133, 3625–3634, https://doi.org/10.1093/brain/awq256 (2010).

71. 71.

Mallet, L. et al. Stimulation of subterritories of the subthalamic nucleus reveals its role in the integration of the emotional and motor aspects of behavior. Proceedings of the National Academy of Sciences of the United States of America 104, 10661–10666, https://doi.org/10.1073/pnas.0610849104 (2007).

72. 72.

Welter, M. L. et al. Optimal target localization for subthalamic stimulation in patients with Parkinson disease. Neurology 82, 1352–1361, https://doi.org/10.1212/WNL.0000000000000315 (2014).

73. 73.

Pote, I. et al. Subthalamic nucleus deep brain stimulation induces impulsive action when patients with Parkinson’s disease act under speed pressure. Experimental brain research 234, 1837–1848, https://doi.org/10.1007/s00221-016-4577-9 (2016).

74. 74.

Herz, D. M. et al. Mechanisms Underlying Decision-Making as Revealed by Deep-Brain Stimulation in Patients with Parkinson’s Disease. Current biology: CB 28, 1169–1178.e1166, https://doi.org/10.1016/j.cub.2018.02.057 (2018).

75. 75.

O’Callaghan, C. et al. Visual Hallucinations Are Characterized by Impaired Sensory Evidence Accumulation: Insights From Hierarchical Drift Diffusion Modeling in Parkinson’s Disease. Biol Psychiatry Cogn Neurosci Neuroimaging 2, 680–688, https://doi.org/10.1016/j.bpsc.2017.04.007 (2017).

76. 76.

Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

77. 77.

Kishida, K. T. et al. Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward. Proceedings of the National Academy of Sciences of the United States of America 113, 200–205, https://doi.org/10.1073/pnas.1513619112 (2016).

78. 78.

Haber, S. N. & Knutson, B. The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35, 4–26, https://doi.org/10.1038/npp.2009.129 (2010).

79. 79.

Basar, K. et al. Nucleus accumbens and impulsivity. Prog Neurobiol 92, 533–557, https://doi.org/10.1016/j.pneurobio.2010.08.007 (2010).

80. 80.

Wittmann, B. C., Daw, N. D., Seymour, B. & Dolan, R. J. Striatal activity underlies novelty-based choice in humans. Neuron 58, 967–973, https://doi.org/10.1016/j.neuron.2008.04.027 (2008).

81. 81.

Tanaka, S. C., Balleine, B. W. & O’Doherty, J. P. Calculating consequences: brain systems that encode the causal effects of actions. The Journal of neuroscience: the official journal of the Society for Neuroscience 28, 6750–6755, https://doi.org/10.1523/JNEUROSCI.1808-08.2008 (2008).

82. 82.

Abler, B., Walter, H., Erk, S., Kammerer, H. & Spitzer, M. Prediction error as a linear function of reward probability is coded in human nucleus accumbens. NeuroImage 31, 790–795, https://doi.org/10.1016/j.neuroimage.2006.01.001 (2006).

83. 83.

Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879, https://doi.org/10.1038/nature04766 (2006).

84. 84.

Frank, M. J., Seeberger, L. C. & O’Reilly, R. C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306, 1940–1943, https://doi.org/10.1126/science.1102941 (2004).

85. 85.

Nombela, C., Rittman, T., Robbins, T. W. & Rowe, J. B. Multiple modes of impulsivity in Parkinson’s disease. PloS one 9, e85747, https://doi.org/10.1371/journal.pone.0085747 (2014).

86. 86.

Robbins, T. W., Gillan, C. M., Smith, D. G., de Wit, S. & Ersche, K. D. Neurocognitive endophenotypes of impulsivity and compulsivity: towards dimensional psychiatry. Trends in cognitive sciences 16, 81–91, https://doi.org/10.1016/j.tics.2011.11.009 (2012).

## Acknowledgements

The authors gratefully acknowledge the commitment of patients and caregivers who contributed their time to this study. The authors acknowledge the ongoing support of St Andrew’s War Memorial Hospital and the Herston Imaging Research Facility. PEM was supported by an early career fellowship from the Queensland government’s ‘Advance Queensland’ initiative, a Royal Brisbane & Woman’s Hospital Foundation Research Grant and a young investigator grant from the Royal Australian and New Zealand College of Psychiatrists. He received an unrestricted educational grant from Medtronic. MB was supported by the National Health and Medical Research Council (118153, 10371296, 1095227) and the Australian Research Council (CE140100007). KES was supported by the University of Zurich and the René and Susanne Braginsky Foundation.

## Author information

Authors

### Contributions

Paliwal: Task design, statistical analysis, writing of first draft of manuscript*. Mosley: Data collection, statistical analysis, writing the first draft of the manuscript*. Breakspear: Critical comments on study design and manuscript. Coyne: Supervision of data collection, critical comments on manuscript. Silburn: Supervision of data collection, critical comments on manuscript. Aponte: Collaborated on model inversion and cross validation. Mathys: Collaborated on model inversion and cross validation. Stephan: Task design, supervision of data analysis, contributions to manuscript writing*.

### Corresponding author

Correspondence to Philip E. Mosley.

## Ethics declarations

### Competing Interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Paliwal, S., Mosley, P.E., Breakspear, M. et al. Subjective estimates of uncertainty during gambling and impulsivity after subthalamic deep brain stimulation for Parkinson’s disease. Sci Rep 9, 14795 (2019). https://doi.org/10.1038/s41598-019-51164-2

• Accepted:

• Published:

• ### The structural connectivity of subthalamic deep brain stimulation correlates with impulsivity in Parkinson’s

• Philip E Mosley
• , Saee Paliwal
• , Katherine Robinson
• , Terry Coyne
• , Peter Silburn
• , Marc Tittgemeyer
• , Klaas E Stephan
• , Alistair Perry
•  & Michael Breakspear

Brain (2020)

• ### Neuroimaging and electrophysiology meet invasive neurostimulation for causal interrogations and modulations of brain states

• Gabriel Gonzalez-Escamilla
• , Muthuraman Muthuraman
• , Dumitru Ciolac
• , Volker A. Coenen
• , Alfons Schnitzler
•  & Sergiu Groppa

NeuroImage (2020)

• ### Deep brain stimulation and recordings: Insights into the contributions of subthalamic nucleus in cognition

• Neil M. Drummond
•  & Robert Chen

NeuroImage (2020)

• ### The structural connectivity of discrete networks underlies impulsivity and gambling in Parkinson’s disease

• Philip E Mosley
• , Saee Paliwal
• , Katherine Robinson
• , Terry Coyne
• , Peter Silburn
• , Marc Tittgemeyer
• , Klaas E Stephan
• , Michael Breakspear
•  & Alistair Perry

Brain (2019)