The ability to decode mood state over time from neural activity could enable closed-loop systems to treat neuropsychiatric disorders. However, this decoding has not been demonstrated, partly owing to the difficulty of modeling distributed mood-relevant neural dynamics while dealing with the sparsity of mood state measurements. Here we develop a modeling framework to decode mood state variations from multi-site intracranial recordings in seven human subjects with epilepsy who self-reported their mood state intermittently over multiple days. We built dynamic neural encoding models of mood state and corresponding decoders for each individual and demonstrated that mood state variations over time can be decoded from neural activity. Across subjects, the decoders largely recruited neural signals from limbic regions, whose spectro-spatial features were tuned to mood variations. The dynamic models also provided an analytical tool to compute the timescales of the decoded mood state. These results provide an initial line of evidence indicating the feasibility of mood state decoding.
Subscribe to Journal
Get full journal access for 1 year
only $20.83 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Calhoon, G.G. & Tye, K.M. Resolving the neural circuits of anxiety. Nat. Neurosci. 18, 1394–1404 (2015).
Dan, R. et al. Separate neural representations of depression, anxiety and apathy in Parkinson's disease. Sci. Rep. 7, 12164 (2017).
Dmochowski, J.P., Sajda, P., Dias, J. & Parra, L.C. Correlated components of ongoing EEG point to emotionally laden attention - a possible marker of engagement? Front. Hum. Neurosci. 6, 112 (2012).
Drevets, W.C. Neuroimaging and neuropathological studies of depression: implications for the cognitive-emotional features of mood disorders. Curr. Opin. Neurobiol. 11, 240–249 (2001).
Drysdale, A.T. et al. Resting-state connectivity biomarkers define neurophysiological subtypes of depression. Nat. Med. 23, 28–38 (2017).
Haroush, K. & Williams, Z.M. Neuronal prediction of opponent's behavior during cooperative social interchange in primates. Cell 160, 1233–1245 (2015).
Kragel, P.A., Knodt, A.R., Hariri, A.R. & LaBar, K.S. Decoding spontaneous emotional states in the human brain. PLoS Biol. 14, e2000106 (2016).
Kupfer, D.J., Frank, E. & Phillips, M.L. Major depressive disorder: new clinical, neurobiological, and treatment perspectives. Lancet 379, 1045–1055 (2012).
Mayberg, H.S. Modulating dysfunctional limbic-cortical circuits in depression: towards development of brain-based algorithms for diagnosis and optimised treatment. Br. Med. Bull. 65, 193–207 (2003).
Mayberg, H.S. et al. Reciprocal limbic-cortical function and negative mood: converging PET findings in depression and normal sadness. Am. J. Psychiatry 156, 675–682 (1999).
Zeng, L.-L. et al. Identifying major depression using whole-brain functional connectivity: a multivariate pattern analysis. Brain 135, 1498–1507 (2012).
Clark, L.A. & Watson, D. Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications. J. Abnorm. Psychol. 100, 316–336 (1991).
Posner, J., Russell, J.A. & Peterson, B.S. The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17, 715–734 (2005).
Clark, D.L., Brown, E.C., Ramasubbu, R. & Kiss, Z.H.T. Intrinsic local beta oscillations in the subgenual cingulate relate to depressive symptoms in treatment-resistant depression. Biol. Psychiatry 80, e93–e94 (2016).
Leuchter, A.F., Cook, I.A., Hunter, A.M., Cai, C. & Horvath, S. Resting-state quantitative electroencephalography reveals increased neurophysiologic connectivity in depression. PLoS One 7, e32508 (2012).
Neumann, W.J. et al. Different patterns of local field potentials from limbic DBS targets in patients with major depressive and obsessive compulsive disorder. Mol. Psychiatry 19, 1186–1192 (2014).
Malone, D.A. Jr. et al. Deep brain stimulation of the ventral capsule/ventral striatum for treatment-resistant depression. Biol. Psychiatry 65, 267–275 (2009).
Mayberg, H.S. et al. Deep brain stimulation for treatment-resistant depression. Neuron 45, 651–660 (2005).
Schlaepfer, T.E. et al. Deep brain stimulation to reward circuitry alleviates anhedonia in refractory major depression. Neuropsychopharmacology 33, 368–377 (2008).
Clark, D.A. & Beck, A.T. Cognitive theory and therapy of anxiety and depression: convergence with neurobiological findings. Trends Cogn. Sci. 14, 418–424 (2010).
Grimm, S. et al. Imbalance between left and right dorsolateral prefrontal cortex in major depression is linked to negative emotional judgment: an fMRI study in severe major depressive disorder. Biol. Psychiatry 63, 369–376 (2008).
Lee, B.-T., Lee, B.C., Choi, I.G., Lyoo, I.K. & Ham, B.J. The neural substrates of affective processing toward positive and negative affective pictures in patients with major depressive disorder. Prog. Neuropsychopharmacol. Biol. Psychiatry 31, 1487–1492 (2007).
Bouchard, K.E., Mesgarani, N., Johnson, K. & Chang, E.F. Functional organization of human sensorimotor cortex for speech articulation. Nature 495, 327–332 (2013).
Korzeniewska, A. et al. Ictal propagation of high frequency activity is recapitulated in interictal recordings: effective connectivity of epileptogenic networks recorded with intracranial EEG. Neuroimage 101, 96–113 (2014).
Yaffe, R.B. et al. Reinstatement of distributed cortical oscillations occurs with precise spatiotemporal dynamics during successful memory retrieval. Proc. Natl. Acad. Sci. USA 111, 18727–18732 (2014).
Nahum, M. et al. Immediate Mood Scaler: tracking symptoms of depression and anxiety using a novel mobile mood scale. JMIR Mhealth Uhealth 5, e44 (2017).
Friedman, J., Hastie, T. & Tibshirani, R. The Elements of Statistical Learning (Springer, New York, 2001).
Van Overschee, P. & De Moor, B. Subspace Identification for Linear Systems: Theory, Implementation and Applications (Kluwer Academic, Norwell, Massachusetts, USA, 1996).
Fitzgerald, P.B., Laird, A.R., Maller, J. & Daskalakis, Z.J. A meta-analytic study of changes in brain activation in depression. Hum. Brain Mapp. 29, 683–695 (2008).
Schieber, M.H. Constraints on somatotopic organization in the primary motor cortex. J. Neurophysiol. 86, 2125–2143 (2001).
Andersen, R.A., Kellis, S., Klaes, C. & Aflalo, T. Toward more versatile and intuitive cortical brain-machine interfaces. Curr. Biol. 24, R885–R897 (2014).
Brandman, D.M., Cash, S.S. & Hochberg, L.R. Review: human intracortical recording and neural decoding for brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 1687–1696 (2017).
Hatsopoulos, N.G. & Suminski, A.J. Sensing with the motor cortex. Neuron 72, 477–487 (2011).
Hotson, G. et al. Individual finger control of a modular prosthetic limb using high-density electrocorticography in a human subject. J. Neural Eng. 13, 026017–26017 (2016).
Leuthardt, E.C., Schalk, G., Wolpaw, J.R., Ojemann, J.G. & Moran, D.W. A brain-computer interface using electrocorticographic signals in humans. J. Neural Eng. 1, 63–71 (2004).
Nicolelis, M.A. & Lebedev, M.A. Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nat. Rev. Neurosci. 10, 530–540 (2009).
Sadtler, P.T. et al. Neural constraints on learning. Nature 512, 423–426 (2014).
Sajda, P., Muller, K.-R. & Shenoy, K. Brain-computer interfaces. IEEE Signal Process. Mag. 25, 16–17 (2008).
Schwartz, A.B., Cui, X.T., Weber, D.J. & Moran, D.W. Brain-controlled interfaces: movement restoration with neural prosthetics. Neuron 52, 205–220 (2006).
Shanechi, M.M. Brain-machine interface control algorithms. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 1725–1734 (2017).
Shanechi, M.M. et al. Rapid control and feedback rates enhance neuroprosthetic control. Nat. Commun. 8, 13825 (2017).
Shenoy, K.V. & Carmena, J.M. Combining decoder design and neural adaptation in brain-machine interfaces. Neuron 84, 665–680 (2014).
Thakor, N.V. Translating the brain-machine interface. Sci. Transl. Med. 5, 210ps17 (2013).
Wang, W. et al. An electrocorticographic brain interface in an individual with tetraplegia. PLoS One 8, e55344 (2013).
Yanagisawa, T. et al. Electrocorticographic control of a prosthetic arm in paralyzed patients. Ann. Neurol. 71, 353–361 (2012).
Habel, U., Klein, M., Kellermann, T., Shah, N.J. & Schneider, F. Same or different? Neural correlates of happy and sad mood in healthy males. Neuroimage 26, 206–214 (2005).
Mak, A.K., Hu, Z.G., Zhang, J.X., Xiao, Z.W. & Lee, T.M. Neural correlates of regulation of positive and negative emotions: an fMRI study. Neurosci. Lett. 457, 101–106 (2009).
Sartorius, N., Ustün, T.B., Lecrubier, Y. & Wittchen, H.U. Depression comorbid with anxiety: results from the WHO study on psychological disorders in primary health care. Br. J. Psychiatry Suppl. 1996, 38–43 (1996).
American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Publishing, Arlington, Virginia, 2013).
Russell, J.A. Core affect and the psychological construction of emotion. Psychol. Rev. 110, 145–172 (2003).
Kroenke, K., Spitzer, R.L. & Williams, J.B. The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med. 16, 606–613 (2001).
Spitzer, R.L., Kroenke, K., Williams, J.B. & Löwe, B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch. Intern. Med. 166, 1092–1097 (2006).
Hermann, B.P., Seidenberg, M. & Bell, B. Psychiatric comorbidity in chronic epilepsy: identification, consequences, and treatment of major depression. Epilepsia 41 (Suppl. 2), S31–S41 (2000).
Harden, C.L. et al. A pilot study of mood in epilepsy patients treated with vagus nerve stimulation. Epilepsy Behav. 1, 93–99 (2000).
George, M.S. et al. Vagus nerve stimulation for the treatment of depression and other neuropsychiatric disorders. Expert Rev. Neurother. 7, 63–74 (2007).
Etkin, A. & Wager, T.D. Functional neuroimaging of anxiety: a meta-analysis of emotional processing in PTSD, social anxiety disorder, and specific phobia. Am. J. Psychiatry 164, 1476–1488 (2007).
Goldstein, R.Z. & Volkow, N.D. Dysfunction of the prefrontal cortex in addiction: neuroimaging findings and clinical implications. Nat. Rev. Neurosci. 12, 652–669 (2011).
Tracey, I. & Bushnell, M.C. How neuroimaging studies have challenged us to rethink: is chronic pain a disease? J. Pain 10, 1113–1120 (2009).
Yang, Y . & Shanechi, M.M. Generalized binary noise stimulation enables time-efficient identification of input-output brain network dynamics. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2016, 1766–1769 (2016).
Yang, Y ., Chang, E.F. & Shanechi, M.M. Dynamic tracking of non-stationarity in human ECoG activity. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2017, 1660–1663 (2017).
Fischl, B. FreeSurfer. Neuroimage 62, 774–781 (2012).
Mazziotta, J. et al. A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). Phil. Trans. R. Soc. Lond. B 356, 1293–1322 (2001).
Yekutieli, D. & Benjamini, Y. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Stat. Plan. Inference 82, 171–196 (1999).
Donoghue, J.P. Bridging the brain to the world: a perspective on neural interface systems. Neuron 60, 511–521 (2008).
Hsieh, H.-L. & Shanechi, M.M. Optimizing the learning rate for adaptive estimation of neural encoding models. PLoS Comput. Biol. 14, e1006168 (2018).
Kao, J.C. et al. Single-trial dynamics of motor cortex and their applications to brain-machine interfaces. Nat. Commun. 6, 7759 (2015).
Pistohl, T., Ball, T., Schulze-Bonhage, A., Aertsen, A. & Mehring, C. Prediction of arm movement trajectories from ECoG-recordings in humans. J. Neurosci. Methods 167, 105–114 (2008).
Shanechi, M.M., Orsborn, A.L. & Carmena, J.M. Robust brain-machine interface design using optimal feedback control modeling and adaptive point process filtering. PLOS Comput. Biol. 12, e1004730 (2016).
So, K., Dangi, S., Orsborn, A.L., Gastpar, M.C. & Carmena, J.M. Subject-specific modulation of local field potential spectral power during brain-machine interface control in primates. J. Neural Eng. 11, 026002 (2014).
Stavisky, S.D., Kao, J.C., Nuyujukian, P., Ryu, S.I. & Shenoy, K.V. A high performing brain-machine interface driven by low-frequency local field potentials alone and together with spikes. J. Neural Eng. 12, 036009 (2015).
Yang, Y. & Shanechi, M.M. An adaptive and generalizable closed-loop system for control of medically induced coma and other states of anesthesia. J. Neural Eng. 13, 066019 (2016).
Anderson, B.D. & Moore, J.B. Optimal Filtering (Dover, New York, 1979).
Cook, R.D. & Weisberg, S. Residuals and Influence in Regression (Chapman and Hall, New York, 1982).
Burgos-Robles, A. et al. Amygdala inputs to prefrontal cortex guide behavior amid conflicting cues of reward and punishment. Nat. Neurosci. 20, 824–835 (2017).
Ebmeier, K.P., Donaghey, C. & Steele, J.D. Recent developments and current controversies in depression. Lancet 367, 153–167 (2006).
Kim, S.-Y. et al. Diverging neural pathways assemble a behavioural state from separable features in anxiety. Nature 496, 219–223 (2013).
Michopoulos, V. et al. Association of CRP genetic variation and CRP level with elevated PTSD symptoms and physiological responses in a civilian population with high levels of trauma. Am. J. Psychiatry 172, 353–362 (2015).
Mulders, P.C., van Eijndhoven, P.F., Schene, A.H., Beckmann, C.F. & Tendolkar, I. Resting-state functional connectivity in major depressive disorder: A review. Neurosci. Biobehav. Rev. 56, 330–344 (2015).
Namburi, P. et al. A circuit mechanism for differentiating positive and negative associations. Nature 520, 675–678 (2015).
Tye, K.M. et al. Dopamine neurons modulate neural encoding and expression of depression-related behaviour. Nature 493, 537–541 (2013).
This research was partially funded by the Defense Advanced Research Projects Agency (DARPA) under Cooperative Agreement Number W911NF-14-2-0043 (to M.M.S. and E.F.C.), issued by the Army Research Office contracting office in support of DARPA's SUBNETS program. The views, opinions and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the US Government.
The authors declare no competing financial interests.
Integrated supplementary information
Supplementary Figure 1 Mood state ranges in each subject and validation of IMS against PHQ-9 and GAD-7.
(a) Range of mood state variations observed in each subject. The lower and upper bounds of the boxes represent the 25th and 75th percentiles of the IMS scores, the middle line represents the median and the whiskers represent the minimum and maximum scores for the subject. Number of samples in each subject is equal to the number of IMS points in that subject (Supplementary Table 3). (b) Correlation of PHQ-9 (red, number of samples equal to 110) and GAD-7 (blue, number of samples equal to 93) scores with the IMS score (adapted from Figure 2.D of Nahum et al.26*; the vertical axis is transferred to the equivalent IMS scoring that is used in this study, i.e., scoring each question from -3 to +3). This data shows the range of IMS values reported for various values of PHQ-9 and GAD-7 across this large population. The range of IMS observed in this study across our subjects is marked on the plots with the green shaded area. The vertical axes in a and b cover the total possible range of IMS (-72 to +72). The horizontal axis in b covers the total range of PHQ-951 (0 to 27) and GAD-752 (0 to 21).26. Nahum, M. et al. Immediate mood scaler: tracking symptoms of depression and anxiety using a novel mobile mood scale. JMIR mHealth and uHealth 5, e44 (2017).51. Kroenke, K., Spitzer, R. L. & Williams, J. B. The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med. 16, 606–613 (2001).52. Spitzer, R. L., Kroenke, K., Williams, J. B. & Löwe, B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch. Intern. Med. 166, 1092–1097 (2006).* Copyright ©Mor Nahum, Thomas M Van Vleet, Vikaas S Sohal, Julie J Mirzabekov, Vikram R Rao, Deanna L Wallace, Morgan B Lee, Heather Dawes, Alit Stark-Inbar, Joshua Thomas Jordan, Bruno Biagianti, Michael Merzenich, Edward F Chang. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org/2017/4/e44/), 12.04.2017. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/).
(a) Progressive region selection. Progressive region selection starts with a network size of one (Net. Size = 1, i.e., one region), and progressively increases the network size until decoding becomes possible (with FDR control to correct for multiple comparisons). At each stage, model selection and model fitting are performed with leave-one-out cross-validation using the training IMS; then the corresponding decoder is tested on the test IMS. To assess significance, the same procedure is performed on sets of random IMS points to compute the random-test P value. (b) Model fitting for a candidate neural encoding model specified by a given network, a given number of PCs, and a given neural state dimension. PCA and LSSM are fitted purely based on neural features, unsupervised with respect to IMS points. Regression model is fitted using the training IMS with regularization. (c) Model selection. Model selection involves fitting several candidate neural encoding models by considering different networks, number of PCs and neural state dimensions. For each candidate neural encoding model, prediction error for training IMS and sensitivity to training IMS are computed and a neural encoding model with small values for both measures is selected. (d) Decoding. The selected neural encoding model is then used to build a decoder to predict the test IMS. Note that the test IMS is never seen in model fitting in b or in model selection in c, which together specify the decoder. The decoder extracts neural features from the selected network, linearly transforms them to PCs, extracts the neural state via Kalman filtering and applies a linear regression to them to predict mood state at the time of the test IMS. We evaluate the decoder only at the discrete time-points that the questionnaire was filled, i.e., that a true measured IMS point was observed.
(a–h) Cross-validated predictions for an example (randomly selected) set of random IMS points is shown pooled across subjects in a and within individuals in b–h. Figure convention is the same as in Fig. 2. Random-test P value is noted on each plot (Online Methods). Number of IMS samples in a is 87 and in b–h is equal to the number of IMS points for each subject (Supplementary Table 3).
Supplementary Figure 4 Cross-validated prediction error for random and permuted sets of IMS points, compared with the true set of IMS points.
Cross-validated prediction error is quantified by the normalized root mean square error (NRMSE). (a–h) Distribution of the prediction error for sets of random (dark blue) and permuted (light blue) IMS points are shown pooled across subjects in a and within individuals in b–h. Cross-validated prediction error for the true IMS points (corresponding to the results shown in Fig. 2 and Table 1) is shown as a red vertical line in each plot. The random and permuted test P values are computed as the probability that NRMSE for random and permuted sets of IMS will be equal to or smaller than that of the true IMS points (area under the distribution tail to the left of the vertical line), respectively. (i, j) Bar plots for average cross-validated prediction errors are shown pooled across subjects in i and within individual subjects in j. In all cases, a two-sided t-test comparing true IMS NRMSE with random and permuted IMS average NRMSEs resulted in P < 10−207 (t > 39 and degrees of freedom > 998 for all t-tests), which is indicated by ***. Error bars represent s.e.m. Number of samples is equal to the number of random and permuted sets, i.e. 109QUOTE 109 in a and i and 1000 in b–h and j.
Prediction of IMS points that were the only IMS point (i.e., report) obtained in a day was significant across subjects. The cross-validated IMS predictions in each subject are z-scored based on the mean and s.d. of all true IMS points in that subject. Then z-scored predictions of the IMS points that were the only IMS point in a day were pooled together across all four subjects that had such IMS points. The same procedure was applied to sets of random IMS points to find the random-test P value (Online Methods). Number of IMS samples is 14.
Supplementary Figure 6 Decoders could significantly predict the maximum and minimum IMS values in each subject, which were outside the range of IMS values used for decoder training.
The cross-validated IMS predictions from each subject are z-scored based on mean and s.d. of all true IMS points in that subject. Then the z-scores of the maximum and minimum IMS predictions in each subject were pooled across the population. The same procedure was applied to sets of random IMS points to find the random-test P value (Online Methods). Number of IMS samples is 14.
Supplementary Figure 7 IMS depression and anxiety subscales could also be separately decoded using the same networks selected for decoding the full IMS.
(a–c) IMS depression and anxiety subscales consist of 7 and 5 of the total 24 questions, respectively26. Within cross-validation, the regression model was retrained to predict the IMS depression and anxiety subscales separately in each subject. The network, number of PCs and neural state dimension were fixed to be the same as those selected for decoding the full IMS. Despite the smaller number of questions and thus potentially a lower subscale measurement signal-to-noise ratio, the IMS depression and anxiety subscales could be significantly decoded across subjects using the same networks that were predictive of the full IMS. Cross-validated predictions were pooled across subjects by z-scoring using the true value of the scale in each subject. Cross-validated predictions versus the true value of the scales are shown for full IMS in a, for IMS depression subscale in b, and for IMS anxiety subscale in c. The random-test P value in each case is noted on the figure and is significant in all cases (Online Methods). Number of IMS and subscale samples in each case is 87. Performing a search of network, number of PCs and neural state dimensions for each IMS subscale separately led to similar results. Moreover, an additional correlation analysis showed that in subjects whose two subscales had different variations over time (all but EC108 whose two subscales had almost equal variance, i.e., 4.2% difference vs. more than 30% difference in the other six subjects), the decoded values for the two subscales also showed a similar difference in their variations. In these subjects, the ratio of the variances of the two subscales, and the ratio of the variances of the decoded values for the two subscales were significantly correlated (Pearson’s P = 0.023, number of samples is 6).26. Nahum, M. et al. Immediate mood scaler: tracking symptoms of depression and anxiety using a novel mobile mood scale. JMIR mHealth and uHealth 5, e44 (2017).
Supplementary Figure 8 Limbic regions alone were sufficient for decoding mood state in all subjects except EC137, whose decoding still required the limbic regions.
(a–g) The best small mood-predictive network sufficient for decoding in each subject. In each panel, lateral (L), medial (M) and orbital (O) views of the brain are shown. The areas corresponding to key limbic regions based on labeling from the FreeSurfer software61 are shaded with color. Each recording channel is also marked with the color of the anatomical region to which it is assigned based on FreeSurfer labels and expert examination (with the latter having priority in case of any difference). All displayed cingulate coverage was in dorsal ACC, except for one intracranial electrode in EC87, which is marked as ventral ACC on the figure (Supplementary Table 2).61. Fischl, B. FreeSurfer. Neuroimage 62, 774–781 (2012).
Supplementary Figure 10 Extending the search to all regions consistently and robustly selected the same regions as those selected in a search within the limbic regions alone.
(a–e) Figure convention is the same as in Supplementary Fig. 8. Note that in EC108, the same AMYG electrode as the limbic search was selected (Supplementary Fig. 8); this electrode had some channels (indicated in black) that were not verified as being exactly in AMYG and thus were not included in the limbic search but were included in the all-region search.
(a–b) The cross-validated normalized mean-absolute error is shown when using only limbic regions or using only other regions for decoding across the population in a and in each individual in b. This analysis was performed for the six subjects that had similar number of neural features inside (97 ± 54) and outside (96 ± 48) limbic regions (two-sided t-test P = 0.98, t = 0.032 and degrees of freedom = 5). Number of samples in a is 75 (total number of IMS points across the six subjects) and in b is equal to the number of IMS points for each subject (Supplementary Table 3). Bars represent mean and error bars represent s.e.m. In b, individual data points are overlaid on each bar plot as black dots (the vertical axis is broken to accommodate the visualization of outliers). Decoding with features outside limbic regions failed at the population level (a). In four of the six subjects, decoding was not significant without the limbic regions (b); only in EC108 and EC166, a mood-predictive network was identified outside the limbic regions in both cases consisting of inferior temporal cortex, which has been implicated in prior studies as mood-relevant81–83 (Supplementary Table 7). Significance caption: n.s. (not significant), random-test P > 0.05; *, random-test P < 0.05; **, random-test P < 0.005; ***, random-test P < 0.005.81. Morris, J. S. et al. A neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain 121, 47–57 (1998).82. Sabatinelli, D., Bradley, M. M., Fitzsimmons, J. R. & Lang, P. J. Parallel amygdala and inferotemporal activation reflect emotional intensity and fear relevance. Neuroimage 24, 1265–1270 (2005).83. Sato, W., Kochiyama, T., Yoshikawa, S., Naito, E. & Matsumura, M. Enhanced neural activity in response to dynamic facial expressions of emotion: an fMRI study. Brain Res. Cogn. Brain Res. 20, 81–91 (2004).
Supplementary Figure 12 Decoding performance using only features from one frequency band compared to using features from all five frequency bands.
(a–b) The cross-validated normalized mean-absolute error is shown for predictions pooled across the population in a and for each individual in b. Within cross-validation, the selected network, the number of PCs, and the neural state dimension for each band were kept the same as those for all bands combined. Number of samples in a is 87 and in b is equal to the number of IMS points for each subject (Supplementary Table 3). Bars represent mean and error bars represent s.e.m. In b, individual data points are overlaid on each bar plot as black dots (the vertical axis is broken to accommodate the visualization of outliers). Asterisks indicate significantly predictive decoding using the corresponding band(s) (*, random-test P < 0.05; **, random-test P < 0.005; ***, random-test P < 0.0005). All frequency bands were predictive in one or more subjects even when used exclusively. Bands with significantly worse decoding error compared with all bands combined (one-sided Wilcoxon signed-rank P < 0.05) are marked with an upward red triangle. No band had significantly lower error than all bands combined. Also, on average, decoding error achieved when the decoder used all bands combined was significantly lower than when it used any single band exclusively as shown in a. Performing a search of the network, the number of PCs and the neural state dimension for each individual band separately led to similar results.
Supplementary Figure 13 Analytically estimated power spectral density (PSD) of the decoded mood state in each subject.
(a–f) The horizontal axis is the timescale (inverse of frequency) and the vertical axis is proportional to the log of PSD at the timescale (dB units). Area under the curve (in original power units) over a certain set of timescales is proportional to how prominent the decoded mood state variations are at those timescales. For example, 70% of the variations in the decoded mood state occurred at timescales of hours or slower (shown as the shaded area in each plot). This analysis was not applicable to EC150 as its neural encoding model was the special case of direct regression (i.e., an LSSM with all eigenvalues of the A matrix equal to 0). Timescale unit abbreviations: h: hours; m: minutes; s: seconds.
Supplementary Figure 14 Mood state decoders provided a slowly changing prediction of mood state around the measured IMS points.
Cross-validated prediction of mood state over time in a 20-minute window centered around each measured IMS point is shown in grey. The true value of the measured IMS point is marked with a black cross. The decoder used to obtain the grey trace in each subplot for prediction around each IMS point is trained only using the other IMS points in the subject (i.e., cross-validation). For 2 IMS points, decoding is shown for less than 10 minutes after the IMS point since the subject was going to sleep immediately after the completion of the IMS report rendering the neural features unreliable (shaded as blue). For comparison, traces of an overfit decoder trained with a degenerate version of our method and without cross-validation is shown in green. This overfit decoder was constructed by deliberately removing several components in our modeling framework, which are designed to prevent overfitting (i.e., progressive region selection, regularization in regression, and the sensitivity measure and inner-level cross-validation for model selection; Online Methods). Moreover, we fully removed the leave-one-out cross-validation and trained the overfit decoder on all IMS points. So to obtain the green trace, the IMS point in each subfigure is used in the training of the overfit decoder. In contrast to our decoder (grey trace), the overfit decoder results in mood traces (green) that often have large variations even within minutes. The s.d. of the overfit green trace during a 20-minute window around each IMS point was significantly larger than the s.d. of our cross-validated decoded trace across all subjects (one-sided Wilcoxon signed-rank P = 8.5×10−15, number of samples is 87).
About this article
Cite this article
Sani, O., Yang, Y., Lee, M. et al. Mood variations decoded from multi-site intracranial human brain activity. Nat Biotechnol 36, 954–961 (2018). https://doi.org/10.1038/nbt.4200
Nonlinear sparse partial least squares: an investigation of the effect of nonlinearity and sparsity on the decoding of intracranial data
Journal of Neural Engineering (2020)
Classification of Facial Expressions for Intended Display of Emotions Using Brain–Computer Interfaces
Annals of Neurology (2020)
Annual Review of Pharmacology and Toxicology (2020)
Development of a neural interface for high-definition, long-term recording in rodents and nonhuman primates
Science Translational Medicine (2020)
Neuroscience Research (2020)