Abstract

Analysis methods in cognitive neuroscience have not always matched the richness of fMRI data. Early methods focused on estimating neural activity within individual voxels or regions, averaged over trials or blocks and modeled separately in each participant. This approach mostly neglected the distributed nature of neural representations over voxels, the continuous dynamics of neural activity during tasks, the statistical benefits of performing joint inference over multiple participants and the value of using predictive models to constrain analysis. Several recent exploratory and theory-driven methods have begun to pursue these opportunities. These methods highlight the importance of computational techniques in fMRI analysis, especially machine learning, algorithmic optimization and parallel computing. Adoption of these techniques is enabling a new generation of experiments and analyses that could transform our understanding of some of the most complex—and distinctly human—signals in the brain: acts of cognition such as thoughts, intentions and memories.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    , , , & Time course EPI of human brain function during task activation. Magn. Reson. Med. 25, 390–397 (1992).

  2. 2.

    et al. Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc. Natl. Acad. Sci. USA 89, 5675–5679 (1992).

  3. 3.

    et al. Intrinsic signal changes accompanying sensory stimulation: functional brain mapping with magnetic resonance imaging. Proc. Natl. Acad. Sci. USA 89, 5951–5955 (1992).

  4. 4.

    , , & Localization of cognitive operations in the human brain. Science 240, 1627–1631 (1988).

  5. 5.

    et al. Statistical parametric maps in functional imaging: a general linear approach. Hum. Brain Mapp. 2, 189–210 (1994).

  6. 6.

    et al. A parametric study of prefrontal cortex involvement in human working memory. Neuroimage 5, 49–62 (1997).

  7. 7.

    & Selective averaging of rapidly presented individual trials using fMRI. Hum. Brain Mapp. 5, 329–340 (1997).

  8. 8.

    et al. Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold. Magn. Reson. Med. 33, 636–647 (1995).

  9. 9.

    & Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and localisation in cluster inference. Neuroimage 44, 83–98 (2009).

  10. 10.

    SPM: a history. Neuroimage 62, 791–800 (2012).

  11. 11.

    AFNI: what a long strange trip it's been. Neuroimage 62, 743–747 (2012).

  12. 12.

    BrainVoyager--past, present, future. Neuroimage 62, 748–756 (2012).

  13. 13.

    , , , & FSL. Neuroimage 62, 782–790 (2012).

  14. 14.

    & Multi-voxel pattern analysis of fMRI data. in The Cognitive Neurosciences (eds. Gazzaniga, M. & Mangun, R.) 911–920 (MIT Press, 2014).

  15. 15.

    et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).

  16. 16.

    & Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685 (2005).

  17. 17.

    , , & Orientation decoding depends on maps, not columns. J. Neurosci. 31, 4792–4804 (2011).

  18. 18.

    & Inferring brain-computational mechanisms with models of activity measurements. Phil. Trans. R. Soc. B 371, 20160278 (2016).

  19. 19.

    , , & Sparse logistic regression for whole-brain classification of fMRI data. Neuroimage 51, 752–764 (2010).

  20. 20.

    , & Searchlight analysis: promise, pitfalls, and potential. Neuroimage 78, 261–269 (2013).

  21. 21.

    & Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 338, 270–273 (2012).

  22. 22.

    , & Hippocampal and ventral medial prefrontal activation during retrieval-mediated learning supports novel inference. Neuron 75, 168–179 (2012).

  23. 23.

    , , & Pruning of memories by context-based prediction error. Proc. Natl. Acad. Sci. USA 111, 8997–9002 (2014).

  24. 24.

    , , & Cortical reinstatement mediates the relationship between content-specific encoding activity and subsequent recollection decisions. Cereb. Cortex 24, 3350–3364 (2014).

  25. 25.

    , & Linking pattern completion in the hippocampus to predictive coding in visual cortex. Nat. Neurosci. 19, 665–667 (2016).

  26. 26.

    et al. Memory consolidation by replay of stimulus-specific neural activity. J. Neurosci. 33, 19373–19383 (2013).

  27. 27.

    , & Distributed patterns of reactivation predict vividness of recollection. J. Cogn. Neurosci. 27, 2000–2018 (2015).

  28. 28.

    , & Confounds in multivariate pattern analysis: theory and rule representation case study. Neuroimage 77, 157–165 (2013).

  29. 29.

    , & Representational similarity analysis - connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).

  30. 30.

    & Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).

  31. 31.

    , , & Hippocampal activity patterns carry information about objects in temporal context. Neuron 81, 1165–1178 (2014).

  32. 32.

    , & A probability distribution over latent causes, in the orbitofrontal cortex. J. Neurosci. 36, 7817–7828 (2016).

  33. 33.

    , & Shaping of object representations in the human medial temporal lobe based on temporal regularities. Curr. Biol. 22, 1622–1627 (2012).

  34. 34.

    & Neural differentiation tracks improved recall of competing memories following interleaved study and retrieval practice. Cereb. Cortex 25, 3994–4008 (2015).

  35. 35.

    & Briefly cuing memories leads to suppression of their neural representations. J. Neurosci. 34, 8010–8020 (2014).

  36. 36.

    , & Insight reconfigures hippocampal-prefrontal memories. Curr. Biol. 25, 821–830 (2015).

  37. 37.

    , & Learning-related representational changes reveal dissociable integration and separation signatures in the hippocampus and prefrontal cortex. Nat. Commun. 6, 8151 (2015).

  38. 38.

    , , , & Retrieval induces adaptive forgetting of competing memories via cortical pattern suppression. Nat. Neurosci. 18, 582–589 (2015).

  39. 39.

    , & Experience-dependent hippocampal pattern differentiation prevents interference during subsequent learning. Nat. Commun. 7, 11066 (2016).

  40. 40.

    et al. What do differences between multi-voxel and univariate analysis mean? How subject-, voxel-, and trial-level variance impact fMRI analysis. Neuroimage 97, 271–283 (2014).

  41. 41.

    , , , & Mind the drift - improving sensitivity to fMRI pattern information by accounting for temporal pattern drift. Preprint at (2015).

  42. 42.

    Decoding fMRI brain states in real-time. Neuroimage 56, 440–454 (2011).

  43. 43.

    et al. Real-time fMRI neurofeedback: progress and challenges. Neuroimage 76, 386–399 (2013).

  44. 44.

    et al. Control over brain activation and pain learned by using real-time functional MRI. Proc. Natl. Acad. Sci. USA 102, 18626–18631 (2005).

  45. 45.

    et al. Neurocognitive therapeutics: from concept to application in the treatment of negative attention bias. Biol. Mood Anxiety Disord. 5, 1 (2015).

  46. 46.

    et al. Volitional reduction of anterior cingulate cortex activity produces decreased cue craving in smoking cessation: a preliminary real-time fMRI study. Addict. Biol. 18, 739–748 (2013).

  47. 47.

    , , & Perceptual learning incepted by decoded fMRI neurofeedback without stimulus presentation. Science 334, 1413–1415 (2011).

  48. 48.

    , , , & Closed-loop training of attention with real-time brain imaging. Nat. Neurosci. 18, 470–475 (2015).

  49. 49.

    , & Real-time functional magnetic resonance imaging. Magn. Reson. Med. 33, 230–236 (1995).

  50. 50.

    et al. Online analysis of functional MRI datasets on parallel platforms. J. Supercomput. 11, 295–318 (1997).

  51. 51.

    et al. Physiological self-regulation of regional brain activity using real-time functional magnetic resonance imaging (fMRI): methodology and exemplary data. Neuroimage 19, 577–586 (2003).

  52. 52.

    , & Direct instrumental conditioning of neural activity using functional magnetic resonance imaging-derived reward feedback. J. Neurosci. 27, 7498–7507 (2007).

  53. 53.

    , , & Covert neurofeedback without awareness shapes cortical network spontaneous connectivity. Proc. Natl. Acad. Sci. USA 113, E2413–E2420 (2016).

  54. 54.

    , & Differential magnetic resonance neurofeedback modulations across extrinsic (visual) and intrinsic (default-mode) nodes of the human cortex. J. Neurosci. 35, 2588–2595 (2015).

  55. 55.

    et al. Meta-analysis of real-time fMRI neurofeedback studies using individual participant data: how is brain regulation mediated? Neuroimage 124 Pt A: 806–812 (2016).

  56. 56.

    et al. When the brain is prepared to learn: enhancing human learning using real-time fMRI. Neuroimage 59, 846–852 (2012).

  57. 57.

    & A method for real-time visual stimulus selection in the study of cortical object perception. Neuroimage 133, 529–548 (2016).

  58. 58.

    & The computational neurobiology of learning and reward. Curr. Opin. Neurobiol. 16, 199–204 (2006).

  59. 59.

    , , & The neural basis of loss aversion in decision-making under risk. Science 315, 515–518 (2007).

  60. 60.

    , & Neural correlates of mentalizing-related computations during strategic interactions in humans. Proc. Natl. Acad. Sci. USA 105, 6741–6746 (2008).

  61. 61.

    , , , & Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).

  62. 62.

    et al. Reinforcement learning in multidimensional environments relies on attention mechanisms. J. Neurosci. 35, 8145–8157 (2015).

  63. 63.

    , , , & Model-based choices involve prospective neural activity. Nat. Neurosci. 18, 767–772 (2015).

  64. 64.

    , , & Two anatomically and computationally distinct learning signals predict changes to stimulus-outcome associations in hippocampus. Neuron 89, 1343–1354 (2016).

  65. 65.

    (Cameron) et al. A reduced-dimension fMRI shared response model. in Advances in Neural Information Processing Systems 28 (eds. Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M. & Garnett, R.) 460–468 (Curran Associates, Inc., 2015).

  66. 66.

    et al. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron 72, 404–416 (2011).

  67. 67.

    , & Training fMRI classifiers to discriminate cognitive states across multiple subjects. in Advances in Neural Information Processing Systems 16 (eds. Thrun, S., Saul, L.K. & Schölkopf, P.B.) (2003).

  68. 68.

    , & Predicting the integration of overlapping memories by decoding mnemonic processing states during learning. Neuroimage 124, 323–335 (2016).

  69. 69.

    , & Decoding the large-scale structure of brain function by classifying mental States across individuals. Psychol. Sci. 20, 1364–1372 (2009).

  70. 70.

    et al. Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity. Nat. Neurosci. 18, 1664–1671 (2015).

  71. 71.

    et al. A model of representational spaces in human cortex. Cereb. Cortex 26, 2919–2934 (2016).

  72. 72.

    et al. Shared memories reveal shared structure in neural activity across individuals. Nat. Neurosci. 20, 115–125 (2017).

  73. 73.

    & Building a science of individual differences from fMRI. Trends Cogn. Sci. 20, 425–443 (2016).

  74. 74.

    et al. A neuromarker of sustained attention from whole-brain functional connectivity. Nat. Neurosci. 19, 165–171 (2016).

  75. 75.

    , , & Topographic factor analysis: a Bayesian model for inferring brain networks from neural data. PLoS One 9, e94914 (2014).

  76. 76.

    Functional interactions as big data in the human brain. Science 342, 580–584 (2013).

  77. 77.

    , & Top-down attention switches coupling between low-level and high-level areas of human visual cortex. Proc. Natl. Acad. Sci. USA 109, 14675–14680 (2012).

  78. 78.

    , , , & Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cereb. Cortex 22, 158–165 (2012).

  79. 79.

    et al. Multi-task connectivity reveals flexible hubs for adaptive task control. Nat. Neurosci. 16, 1348–1355 (2013).

  80. 80.

    , & Network scaling effects in graph analytic studies of human resting-state fMRI data. Front. Syst. Neurosci. 4, 22 (2010).

  81. 81.

    , , & Full correlation matrix analysis (FCMA): an unbiased method for task-related functional connectivity. J. Neurosci. Methods 251, 108–119 (2015).

  82. 82.

    & Complex brain networks: graph theoretical analysis of structural and functional systems. Nat. Rev. Neurosci. 10, 186–198 (2009).

  83. 83.

    , & Two's company, three (or more) is a simplex: algebraic-topological tools for understanding higher-order structure in neural data. Preprint at (2016).

  84. 84.

    et al. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191–1195 (2008).

  85. 85.

    A spreading activation theory of memory. J. Verbal Learn. Verbal Behav. 22, 261–295 (1983).

  86. 86.

    , & On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97, 332–361 (1990).

  87. 87.

    & Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).

  88. 88.

    Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1, 417–446 (2015).

  89. 89.

    , , , & Deep neural networks predict hierarchical spatio-temporal cortical dynamics of human visual object recognition. Preprint at (2016).

  90. 90.

    & Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).

  91. 91.

    Tracking problem solving by multivariate pattern analysis and Hidden Markov Model algorithms. Neuropsychologia 50, 487–498 (2012).

  92. 92.

    et al. Full correlation matrix analysis of fMRI data on Intel Xeon Phi coprocessors. in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Supercomputing) (ACM, 2015).

  93. 93.

    et al. Enabling factor analysis on thousand-subject neuroimaging datasets. Preprint at (2016).

  94. 94.

    et al. Large scale distributed deep networks. in Advances in Neural Information Processing Systems 25 (eds. Pereira, F., Burges, C.J.C., Bottou, L. & Weinberger, K.Q.) 1223–1231 (Curran Associates, Inc., 2012).

  95. 95.

    et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci. Data 3, 160044 (2016).

  96. 96.

    et al. Dynamic reconfiguration of the default mode network during narrative comprehension. Nat. Commun. 7, 12141 (2016).

  97. 97.

    et al. Real-time full correlation matrix analysis of fMRI data. in 2016 IEEE International Conference on Big Data (Big Data 2016) (Curran Associates, Inc., in the press).

  98. 98.

    et al. Mapping between natural movie fMRI responses and word-sequence representations. Preprint at (2016).

  99. 99.

    et al. A searchlight factor model approach for locating shared information in multi-subject fMRI analysis. Preprint at (2016).

  100. 100.

    et al. Mapping brain activity at scale with cluster computing. Nat. Methods 11, 941–950 (2014).

Download references

Acknowledgements

We thank C. Chen, M. Regev, Y. Wang, and H. Zhang for assistance and the members of our labs at Princeton University and Intel Labs for their numerous invaluable contributions to the work described herein. This work was made possible by support from Intel Corporation, the John Templeton Foundation, NIH grants R01 EY021755 and R01 MH069456, and NSF grant MRI BCS1229597. The opinions expressed in this publication do not necessarily reflect the views of these agencies.

Author information

Affiliations

  1. Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, USA.

    • Jonathan D Cohen
    • , Nathaniel Daw
    • , Uri Hasson
    • , Yael Niv
    • , Kenneth A Norman
    • , Jonathan Pillow
    •  & Nicholas B Turk-Browne
  2. Department of Psychology, Princeton University, Princeton, New Jersey, USA.

    • Jonathan D Cohen
    • , Nathaniel Daw
    • , Uri Hasson
    • , Yael Niv
    • , Kenneth A Norman
    • , Jonathan Pillow
    •  & Nicholas B Turk-Browne
  3. Department of Computer Science, Princeton University, Princeton, New Jersey, USA.

    • Barbara Engelhardt
    •  & Kai Li
  4. Department of Electrical Engineering, Princeton University, Princeton, New Jersey, USA.

    • Peter J Ramadge
  5. Intel Labs, Intel Corporation, Santa Clara, California, USA.

    • Theodore L Willke

Authors

  1. Search for Jonathan D Cohen in:

  2. Search for Nathaniel Daw in:

  3. Search for Barbara Engelhardt in:

  4. Search for Uri Hasson in:

  5. Search for Kai Li in:

  6. Search for Yael Niv in:

  7. Search for Kenneth A Norman in:

  8. Search for Jonathan Pillow in:

  9. Search for Peter J Ramadge in:

  10. Search for Nicholas B Turk-Browne in:

  11. Search for Theodore L Willke in:

Contributions

All authors helped conceive the manuscript and wrote one or more sections. N.B.T.-B. edited the manuscript. N.B.T.-B. revised the manuscript with input from J.D.C., K.A.N. and P.J.R. Author order was determined alphabetically.

Competing interests

All authors receive research support from Intel Corporation. T.L.W. is employed by Intel Corporation.

Corresponding author

Correspondence to Nicholas B Turk-Browne.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nn.4499

Further reading