Review Article | Published:

Neuroimaging

Decoding mental states from brain activity in humans

Nature Reviews Neuroscience volume 7, pages 523534 (2006) | Download Citation

Subjects

Abstract

Recent advances in human neuroimaging have shown that it is possible to accurately decode a person's conscious experience based only on non-invasive measurements of their brain activity. Such 'brain reading' has mostly been studied in the domain of visual perception, where it helps reveal the way in which individual experiences are encoded in the human brain. The same approach can also be extended to other types of mental state, such as covert attitudes and lie detection. Such applications raise important ethical issues concerning the privacy of personal thought.

Key points

  • Understanding whether cognitive and perceptual states can be decoded from brain activity alone is a fundamental question in cognitive neuroscience. It is not only relevant for scientific theories of how information is encoded in the brain, but also has important practical and ethical implications.

  • Non-invasive techniques such as functional MRI (fMRI) can be used to record signals related to brain activity in humans from many locations in the brain simultaneously. However, many conventional approaches to analysing these data rely on considering signal changes at each location independently of all the other locations in the brain

  • These conventional approaches have proven successful in elucidating many aspects of the relationship between cognitive and mental states and brain activity. However, recent advances in data analysis procedures raise the possibility of deciphering additional and complementary information from neuroimaging data.

  • Recently, a powerful approach has emerged that applies pattern-recognition techniques to neuroimaging data. The new strategy is to decode a person's current mental state by learning to recognize characteristic spatial patterns of brain activity associated with different mental states. This takes into account not just activity at single locations but the full spatial pattern of activity. Such pattern-based decoding reveals that substantially more information is encoded in fMRI signals than was previously recognised.

  • These new approaches have a particular use in addressing the question of how information regarding perceptual and cognitive states is encoded in the human brain. Pattern-based decoding has now been successfully used to reveal the principles underlying the representation of objects in the ventral visual pathway. It can also reveal conscious and unconscious sensory representations of individual features, and can be used to track dynamic changes in the contents of consciousness over time.

  • Decoding approaches therefore provide a particularly sensitive way to determine what types of information are represented in the spatially distributed pattern of brain responses recorded with current neuroimaging techniques. However, for more general applications, important technical and methodological barriers remain to be overcome, including the ability of such approaches to generalize across individuals and different cognitive and perceptual states.

  • As these techniques have the possibility to reveal covert or unconscious mental states, they raise important ethical and privacy concerns. These can be addressed within existing ethical frameworks, but nevertheless necessitate careful and considered engagement by the neuroimaging community.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    Emerging ethical issues in neuroscience. Nature Neurosci. 5, 1123–1129 (2002).

  2. 2.

    & Functional magnetic resonance imaging (fMRI) 'brain reading': detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19, 261–270 (2003). This study compares various classification techniques and outlines important principles of decoding-based fMRI research.

  3. 3.

    & Mental imagery of faces and places activates corresponding stimulus-specific brain regions. J. Cogn. Neurosci. 12, 1013–1023 (2000).

  4. 4.

    et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001). One of the first studies using pattern-based analysis to investigate the nature of object representations in the human ventral visual cortex.

  5. 5.

    , & Patterns of activity in the categorical representation of objects. J. Cogn. Neurosci. 15, 704–717 (2003).

  6. 6.

    et al. Classifying instantaneous cognitive states from FMRI data. AMIA Annu. Symp. Proc. 465–469 (2003).

  7. 7.

    , , , & Learning to decode cognitive states from brain images. Machine Learning 57, 145–175 (2004).

  8. 8.

    & Decoding the visual and subjective contents of the human brain. Nature Neurosci. 8, 679–685 (2005). The first application of multivariate classification to reveal processing of features in the primary visual cortex represented below the resolution of fMRI.

  9. 9.

    & Predicting the orientation of invisible stimuli from activity in primary visual cortex. Nature Neurosci. 8, 686–691 (2005). This study directly compares perceptual performance with the performance of a decoder trained on fMRI-signals from the early visual cortex.

  10. 10.

    & Predicting the stream of consciousness from activity in human visual cortex. Curr. Biol. 15, 1301–1307 (2005). This work reveals the potential power of multivariate decoding to track perception quasi-online on a second-to-second basis.

  11. 11.

    & Decoding motion direction from activity in human visual cortex. J. Vision 5, 152a (2005).

  12. 12.

    , & Information-based functional brain mapping. Proc. Natl Acad. Sci. USA 103, 3863–3868 (2006). This study introduces the 'searchlight' approach that searches across the entire brain for specific local patterns that encode information about a cognitive or perceptual state.

  13. 13.

    , , , & Support vector machines for temporal classification of block design fMRI data. Neuroimage 26, 317–329 (2005).

  14. 14.

    , , , & Classifying brain states and determining the discriminating activation patterns: support vector machine on functional MRI data. Neuroimage 28, 980–995 (2005).

  15. 15.

    , , & Partially distributed representation of objects and faces in ventral temporal cortex. J. Cogn. Neurosci. 17, 580–590 (2005).

  16. 16.

    , , & Category-specific cortical activity precedes retrieval during memory search. Science 310, 1963–1966 (2005).

  17. 17.

    , & Predicting performance from functional imaging data: methods matter. Neuroimage 20, 615–624 (2003).

  18. 18.

    & On the nature of the BOLD fMRI contrast mechanism. Magn. Reson. Imaging 22, 1517–1531 (2004).

  19. 19.

    et al. Face recognition in human extrastriate cortex. J. Neurophysiol. 71, 821–825 (1994).

  20. 20.

    , & The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17, 4302–4311 (1997).

  21. 21.

    & A cortical representation of the local visual environment. Nature 392, 598–601 (1999).

  22. 22.

    et al. FMRI of human visual cortex. Nature 369, 525 (1994).

  23. 23.

    et al. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science 268, 889–893 (1995).

  24. 24.

    et al. Inferring behavior from functional brain images. Nature Neurosci. 1, 549–550 (1998).

  25. 25.

    , , & A cortical region consisting entirely of face-selective cells. Science 311, 670–674 (2006).

  26. 26.

    , , & A cortical area selective for visual processing of the human body. Science 293, 2470–2473 (2001).

  27. 27.

    et al. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients, Brain 123, 291–307 (2000).

  28. 28.

    , , , & Domain specificity in visual cortex. Cereb. Cortex Dec 7 2005 (doi: 10.1093/cercor/bhj086).

  29. 29.

    , & Face perception is mediated by a distributed cortical network. Brain Res. Bull. 67, 87–93 (2005).

  30. 30.

    , , & Towards direct visualization of the internal shape space by fMRI. Psychobiology 26, 309–321 (1998). This early work is the first application of multivariate techniques to the study of object representation. Of special interest is the demonstration that shape space is reflected in the similarity space between evoked cortical responses.

  31. 31.

    & How distributed is visual category information in human occipito-temporal cortex? An fMRI study. Neuron 35, 1157–1165 (2002).

  32. 32.

    Mechanisms of visual object recognition: monkey and human studies. Curr. Opin. Neurobiol. 7, 523–529 (1997).

  33. 33.

    , & Optical imaging of functional organization in the monkey inferotemporal cortex. Science 272, 1665–1668 (1996).

  34. 34.

    & Geometry of orientation and ocular dominance columns in monkey striate cortex. J. Neurosci. 13, 4114–4129 (1993).

  35. 35.

    The Principles of Psychology (Henry Holt, New York, 1890).

  36. 36.

    & Visual competition. Nature Rev. Neurosci. 3, 13–21 (2002).

  37. 37.

    , & Eye-specific effects of binocular rivalry in the human lateral geniculate nucleus. Nature 438, 496–499 (2005).

  38. 38.

    & A method for investigating binocular rivalry in real-time with the steady-state VEP. Vision Res. 37, 2401–2408 (1997).

  39. 39.

    , , & Fast readout of object identity from macaque inferior temporal cortex. Science 310, 863–866 (2005).

  40. 40.

    , , , & Intersubject synchronization of cortical activity during natural vision. Science 303, 1634–1640 (2004).

  41. 41.

    & Functional brain mapping during free viewing of natural scenes. Hum. Brain Mapp. 21, 75–85 (2004),

  42. 42.

    Eye Movements and Vision (Plenum, New York, 1967).

  43. 43.

    , & The updating of the representation of visual space in parietal cortex by intended eye movements. Science 255, 90–92 (1992).

  44. 44.

    et al. Classifying spatial patterns of brain activity with machine learning methods: application to lie detection. Neuroimage 28, 663–668 (2005). This study is the first demonstration that multivariate decoding can be applied to lie detection.

  45. 45.

    et al. Telling truth from lie in individual subjects with fast event-related fMRI. Hum. Brain Mapp. 26, 262–272 (2005).

  46. 46.

    et al. Detecting deception using functional magnetic resonance imaging. Biol. Psychiatry 58, 605–613 (2005).

  47. 47.

    Conscious and unconscious perception: experiments on visual masking and word recognition. Cognit. Psychol. 15, 197–237 (1983).

  48. 48.

    & Using direct and indirect measures to study perception without awareness. Percept. Psychophys. 44, 563–575 (1988).

  49. 49.

    et al. Imaging unconscious semantic priming. Nature 395, 597–600 (1998)

  50. 50.

    , & Are we aware of neural activity in primary visual cortex? Nature 375, 121–123 (1995)

  51. 51.

    et al. Performance on indirect measures of race evaluation predicts amygdala activation. J. Cogn. Neurosci. 12, 729–738 (2000).

  52. 52.

    , , , Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential). The unconscious initiation of a freely voluntary act. Brain 106, 623–642 (1983).

  53. 53.

    & On the relation between brain potentials and the awareness of voluntary movements. Exp. Brain. Res. 126, 128–133 (1999).

  54. 54.

    Tremendous New Power to be Unleashed. Kansas City Journal-Post (10 Sep 1933).

  55. 55.

    & Beyond the visible — imaging the human brain with light. J. Cereb. Blood Flow Metab. 23, 1–18 (2003).

  56. 56.

    , , & Invariance between subjects of brain wave representations of language. Proc. Natl Acad. Sci. USA 96, 12953–12958 (1999).

  57. 57.

    et al. Spatial registration and normalisation of images. Hum. Brain Mapp. 2, 165–189 (1995).

  58. 58.

    , , & High resolution intersubject averaging and a coordinate system for the cortical surface. Hum. Brain Mapp. 8, 272–284 (1999).

  59. 59.

    , & Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10, 14–23 (2006).

  60. 60.

    The columnar organization of the neocortex. Brain 120, 701–722 (1997).

  61. 61.

    & The cortical column: a structure without a function. Philos. Trans. R. Soc. Lond. B Biol. Sci. 360, 837–862 (2005).

  62. 62.

    Can cognitive processes be inferred from neuroimaging data? Trends Cogn. Sci. 10, 59–63 (2006).

  63. 63.

    et al. Separable neural components in the processing of black and white faces. Psychol. Sci. 15, 806–813 (2004).

  64. 64.

    et al. An fMRI investigation of the impact of interracial contact on executive function. Nature Neurosci. 6, 1323–1328 (2003).

  65. 65.

    , & Neural correlates of conscious self-regulation of emotion. J. Neurosci. 21, RC165 (2001).

  66. 66.

    , , & Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. Neuroimage 16, 331–348 (2002).

  67. 67.

    & Neuroimaging of emotion and personality: scientific evidence and ethical considerations. Brain Cogn. 50, 414–431 (2002).

  68. 68.

    , & Neuroimaging and personality disorders. Curr. Psychiatry Rep. 7, 65–72 (2005).

  69. 69.

    , & Neuroimaging in psychopathy. Aust. N. Z. J. Psychiatry 39, 856–865 (2005).

  70. 70.

    et al. Reduced prefrontal and increased subcortical brain functioning assessed using positron emission tomography in predatory and affective murderers. Behav. Sci. Law 16, 319–332 (1998).

  71. 71.

    et al. Limbic activation during cue-induced cocaine craving. Am. J. Psychiatry 156, 11–18 (1999).

  72. 72.

    et al. Neural correlates of behavioral preference for culturally familiar drinks. Neuron 44, 379–387 (2004).

  73. 73.

    , , & A general mechanism for perceptual decision-making in the human brain. Nature 431, 859–862 (2004).

  74. 74.

    et al. Differences in cerebral blood flow and glucose utilization in vegetative versus locked-in patients. Ann. Neurol. 22, 673–682 (1987).

  75. 75.

    , , , & Residual cognitive function in comatose, vegetative and minimally conscious states. Curr. Opin. Neurol. 18, 726–733 (2005).

  76. 76.

    Neuroethics: the practical and the philosophical. Trends Cogn. Sci. 9, 34–40 (2005).

  77. 77.

    Brain scam? Nature Neurosci. 7, 1015 (2004).

  78. 78.

    & Will the law come running? The potential role of 'brain fingerprinting' in crime investigation and adjudication in Australia. J. Law Med. 13, 204–222 (2005).

  79. 79.

    Occipital alpha rhythm, eye position and lens accommodation. Nature 214, 975–977 (1967).

  80. 80.

    Actions from thoughts. Nature 409, 403–407 (2001).

  81. 81.

    , , , & Cognitive neural prosthetics. Trends Cogn. Sci. 8, 486–493 (2004).

  82. 82.

    et al. Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG analysis. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 127–131 (2003).

  83. 83.

    , , , & Brain–computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791 (2002).

  84. 84.

    & Control of a two-dimensional movement signal by a noninvasive brain–computer interface in humans. Proc. Natl Acad. Sci. USA 101, 17849–17854 (2004).

  85. 85.

    , & Category-specific visual responses of single neurons in the human median temporal lobe. Nature Neurosci. 3, 946–953 (2000).

  86. 86.

    , , , & Invariant visual representation by single neurons in the human brain. Nature 435, 1102–1107 (2005). Here, multivariate decoding is applied to simultaneous recordings of spike trains from the human medial temporal lobe.

  87. 87.

    & Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9, 1707–1711 (1998).

  88. 88.

    et al. A spelling device for the paralysed. Nature 398, 297–298 (1999).

  89. 89.

    et al. Self-regulation of local brain activity using real-time functional magnetic resonance imaging (fMRI). J. Physiol. (Paris) 98, 357–373 (2004).

  90. 90.

    , & Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. J. Neurosci. 19, 8036–8042 (1999).

  91. 91.

    & Who can catch a liar? Am. Psychol. 46, 913–920 (1991).

  92. 92.

    The systolic blood pressure symptoms of deception. J. Exp. Psy. 2, 117–163 (1917).

  93. 93.

    History of the polygraph, an instrument for the detection of deception. Biomed. Eng. 8, 154–156 (1973).

  94. 94.

    The inspiration–expiration ratio during truth and falsehood. J. Exp. Psy. 4, 1–23 (1921).

  95. 95.

    & A comparison of physiological indices in detection of deception. Psychophysiol. 4, 329–339 (1968).

  96. 96.

    Detecting deception: the promise and the reality of voice stress analysis. J. Forensic Sci. 27, 340–351 (1982).

  97. 97.

    , & Seeing through the face of deception. Nature 415, 35 (2002).

  98. 98.

    , , , & Comparison of polygraph data obtained from individuals involved in mock crimes and actual crime investigations. J. Appl. Psychol. 89, 1099–1105 (2004).

  99. 99.

    Tremor in the Blood: Uses and Abuses of the Lie Detector (McGraw-Hill, New York, 1981).

  100. 100.

    , & Mental and physical countermeasures reduce the accuracy of polygraph tests. J. App. Psy. 79, 252–259 (1994).

  101. 101.

    & Using brain MERMER testing to detect knowledge despite efforts to conceal. J. Forensic Sci. 46, 135–143 (2001).

  102. 102.

    et al. Behavioral and functional anatomical correlates of deception in humans. Neuroreport 12, 2849–2853 (2001).

  103. 103.

    et al. Neural correlates of telling lies: a functional magnetic resonance imaging study at 4 Tesla. Acad. Radiol. 12, 164–172 (2005).

  104. 104.

    , , , & Neural correlates of different types of deception: an fMRI investigation. Cereb. Cortex 13, 830–836 (2003).

  105. 105.

    et al. Lie detection by functional magnetic resonance imaging. Hum. Brain Mapp. 15, 157–164 (2002).

  106. 106.

    Imaging orientation selectivity: decoding conscious perception in V1. Nature Neurosci. 8, 541–542 (2005).

  107. 107.

    , & Functional imaging and neural information coding. Neuroimage 21, 1083–1095 (2004).

  108. 108.

    How much cortex can a vein drain? Downstream dilution of activation-related cerebral blood oxygenation changes. Neuroimage 16, 1062–1067 (2002).

  109. 109.

    The Human Brain (Springer, New York, 1999).

Download references

Acknowledgements

This work was supported by the Wellcome Trust and the Mind–Science Foundation. We thank V. Lamme for bringing the reference to Nikola Tesla to our attention, and thank J. Driver, C. Frith and K.-E. Stephan for helpful comments on the manuscript.

Author information

Affiliations

  1. Max Planck Institute for Cognitive and Brain Sciences, Stephanstrasse 1a, 04103 Leipzig, Germany.

    • John-Dylan Haynes
  2. Wellcome Department of Imaging Neuroscience, Institute of Neurology, University College London, 12 Queen Square, London WC1N 3BG, UK.

    • John-Dylan Haynes
    •  & Geraint Rees
  3. Institute of Cognitive Neuroscience, University College London, Alexandra House, 17 Queen Square, London WC1N 3AR, UK.

    • John-Dylan Haynes
    •  & Geraint Rees

Authors

  1. Search for John-Dylan Haynes in:

  2. Search for Geraint Rees in:

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to John-Dylan Haynes.

Glossary

Multivariate analysis

An analytical technique that considers (or solves) multiple decision variables. In the present context, multivariate analysis takes into account patterns of information that might be present across multiple voxels measured by neuroimaging techniques.

Univariate analysis

Univariate statistical analysis considers only single-decision variables at any one time. Conventional brain imaging data analyses are mass univariate in that they consider how responses vary at very many single voxels, but consider each individual voxel separately.

Blood-oxygen-leveldependent (BOLD) signal

Functional MRI measures local changes in the proportion of oxygenated blood in the brain; the BOLD signal. This proportion changes in response to neural activity. Therefore, the BOLD signal, or haemodynamic response, indicates the location and magnitude of neural activity.

Primary visual cortex

Considered to be the first visual cortical area in primates, and receives the majority of its input from the retina via the lateral geniculate nucleus.

Voxel

A voxel is the three-dimensional (3D) equivalent of a pixel; a finite volume within 3D space. This corresponds to the smallest element measured in a 3D anatomical or functional brain image volume.

Pattern vector

A vector is a set of one or more numerical elements. Here, a pattern vector is the set of values that together represent the value of each individual voxel in a particular spatial pattern.

Orientation tuning

Many neurons in the mammalian early visual cortex evoke spikes at a greater rate when the animal is presented with visual stimuli of a particular orientation. The stimulus orientation that evokes the greatest firing rate for a particular cell is known as its preferred orientation, and the orientation tuning curve of a cell describes how that firing rate changes as the orientation of the stimulus is varied away from the preferred orientation.

Spatial anisotropy

An anisotropic property is one where a measurement made in one direction differs from the measurement made in another direction. For example, the orientation tuning preferences of neurons in V1 change in a systematic but anisotropic way across the surface of the cortex.

Electroencephalogram

(EEG). The continuously changing electrical signal recorded from the scalp in humans that reflects the summated postsynaptic potentials of cortical neurons in response to changing cognitive or perceptual states. The EEG can be measured with extremely high temporal resolution.

Magnetoencephalography

A non-invasive technique that allows the detection of the changing magnetic fields that are associated with brain activity on the timescale of milliseconds.

About this article

Publication history

Published

DOI

https://doi.org/10.1038/nrn1931

Further reading