Article | Published:

Decoding the visual and subjective contents of the human brain

Nature Neuroscience volume 8, pages 679685 (2005) | Download Citation

Subjects

Abstract

The potential for human neuroimaging to read out the detailed contents of a person's mental state has yet to be fully explored. We investigated whether the perception of edge orientation, a fundamental visual feature, can be decoded from human brain activity measured with functional magnetic resonance imaging (fMRI). Using statistical algorithms to classify brain states, we found that ensemble fMRI signals in early visual areas could reliably predict on individual trials which of eight stimulus orientations the subject was seeing. Moreover, when subjects had to attend to one of two overlapping orthogonal gratings, feature-based attention strongly biased ensemble activity toward the attended orientation. These results demonstrate that fMRI activity patterns in early visual areas, including primary visual cortex (V1), contain detailed orientation information that can reliably predict subjective perception. Our approach provides a framework for the readout of fine-tuned representations in the human brain and their subjective contents.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    & Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. (Lond.) 160, 106–154 (1962).

  2. 2.

    & Receptive fields and functional architecture of monkey striate cortex. J. Physiol. (Lond.) 195, 215–243 (1968).

  3. 3.

    Orientation selectivity, preference, and continuity in monkey striate cortex. J. Neurosci. 12, 3139–3161 (1992).

  4. 4.

    & Relationships between orientation-preference pinwheels, cytochrome oxidase blobs, and ocular-dominance columns in primate striate cortex. Proc. Natl. Acad. Sci. USA 89, 11905–11909 (1992).

  5. 5.

    A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biol. Cybern. 58, 35–49 (1988).

  6. 6.

    , & Information processing with population codes. Nat. Rev. Neurosci. 1, 125–132 (2000).

  7. 7.

    et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).

  8. 8.

    & Functional magnetic resonance imaging (fMRI) 'brain reading': detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19, 261–270 (2003).

  9. 9.

    , & Patterns of activity in the categorical representations of objects. J. Cogn. Neurosci. 15, 704–717 (2003).

  10. 10.

    , , & The organization of orientation selectivity throughout macaque visual cortex. Cereb. Cortex 12, 647–662 (2002).

  11. 11.

    , & High-resolution mapping of iso-orientation columns by fMRI. Nat. Neurosci. 3, 164–169 (2000).

  12. 12.

    , & Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb. Cortex 7, 181–192 (1997).

  13. 13.

    & Interactions between electrical activity and cortical microcirculation revealed by imaging spectroscopy: implications for functional brain mapping. Science 272, 551–554 (1996).

  14. 14.

    , , , & Long-term optical imaging and spectroscopy reveal mechanisms underlying the intrinsic signal and stability of cortical maps in V1 of behaving monkeys. J. Neurosci. 20, 8111–8121 (2000).

  15. 15.

    Statistical Learning Theory (Wiley, New York, 1998).

  16. 16.

    & Perceptrons – Expanded Edition: An Introduction to Computational Geometry (MIT Press, Boston, 1987).

  17. 17.

    & An oblique effect in human primary visual cortex. Nat. Neurosci. 3, 535–536 (2000).

  18. 18.

    & Complementary global maps for orientation coding in upper and lower layers of the monkey's foveal striate cortex. Exp. Brain Res. 76, 503–509 (1989).

  19. 19.

    , & Retinal ganglion cell dendritic fields in old-world monkeys are oriented radially. Brain Res. 368, 18–23 (1986).

  20. 20.

    & Attentional modulation of visual motion processing in cortical areas MT and MST. Nature 382, 539–541 (1996).

  21. 21.

    & Feature-based attention influences motion processing gain in macaque visual cortex. Nature 399, 575–579 (1999).

  22. 22.

    Visual attention: the where, what, how and why of saliency. Curr. Opin. Neurobiol. 13, 428–432 (2003).

  23. 23.

    , & Object-based attention in the primary visual cortex of the macaque monkey. Nature 395, 376–381 (1998).

  24. 24.

    et al. Task-dependent influences of attention on the activation of human primary visual cortex. Proc. Natl. Acad. Sci. USA 95, 11489–11492 (1998).

  25. 25.

    , & Global effects of feature-based attention in human visual cortex. Nat. Neurosci. 5, 631–632 (2002).

  26. 26.

    , , , & in Visual Perception: The Neurophysiological Foundations (eds. Spillman, L. & Werner, J.S.) 231–272 (Academic, San Diego, 1990).

  27. 27.

    et al. Functional analysis of primary visual cortex (V1) in humans. Proc. Natl. Acad. Sci. USA 95, 811–817 (1998).

  28. 28.

    & Orientation-specific adaptation in human visual cortex. J. Neurosci. 23, 8781–8787 (2003).

  29. 29.

    & Manifestation of scotomas created by transcranial magnetic stimulation of human visual cortex. Nat. Neurosci. 2, 767–771 (1999).

  30. 30.

    , & Neural correlates of consciousness in humans. Nat. Rev. Neurosci. 3, 261–270 (2002).

  31. 31.

    Primary visual cortex and visual awareness. Nat. Rev. Neurosci. 4, 219–229 (2003).

  32. 32.

    The Quest for Consciousness: A Neurobiological Approach (Roberts, Englewood, Colorado, USA, 2004).

  33. 33.

    Connecting cortex to machines: recent advances in brain interfaces. Nat. Neurosci. 5 (suppl.): 1085–1088 (2002).

  34. 34.

    & Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. USA 101, 17849–17854 (2004).

  35. 35.

    & Stroboscopic motion: Effects of duration and interval. Percept. Psychophys. 8, 161–164 (1970).

  36. 36.

    et al. Borders of multiple visual area in humans revealed by functional magnetic resonance imaging. Science 268, 889–893 (1995).

  37. 37.

    et al. A direct demonstration of functional specialization in human visual cortex. J. Neurosci. 11, 641–649 (1991).

  38. 38.

    et al. Area V5 of the human brain: evidence from a combined study using positron emission tomography and magnetic resonance imaging. Cereb. Cortex 3, 79–94 (1993).

  39. 39.

    et al. Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging. J. Neurosci. 15, 3215–3230 (1995).

  40. 40.

    , , , & Automated image registration: I. General methods and intrasubject, intramodality validation. J. Comput. Assist. Tomogr. 22, 139–152 (1998).

Download references

Acknowledgements

We thank D. Remus and J. Kerlin for technical support, the Princeton Center for Brain, Mind and Behavior for MRI support and J. Haxby, D. Heeger and A. Seiffert for comments. This work was supported by grants from Japan Society for the Promotion of Science and National Institute of Information and Communications Technology to Y.K., and grants R01-EY14202 and P50-MH62196 from the US National Institutes of Health to F.T.

Author information

Author notes

    • Frank Tong

    Present address: Psychology Department, Vanderbilt University, 301 Wilson Hall, 111 21st Avenue South, Nashville, Tennessee 37203, USA.

Affiliations

  1. ATR Computational Neuroscience Laboratories, 2-2-2 Hikaridai, Keihanna Science City, Kyoto 619-0288, Japan.

    • Yukiyasu Kamitani
  2. Psychology Department, Princeton University, Green Hall, Princeton, New Jersey, 08544, USA.

    • Frank Tong

Authors

  1. Search for Yukiyasu Kamitani in:

  2. Search for Frank Tong in:

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Yukiyasu Kamitani.

Supplementary information

PDF files

  1. 1.

    Supplementary Fig. 1

    Average fMRI responses to orientations for each visual area.

  2. 2.

    Supplementary Fig. 2

    Average fMRI responses for voxels with the same orientation preference, shown by visual area.

  3. 3.

    Supplementary Fig. 3

    Map of differential responses to eight orientations.

  4. 4.

    Supplementary Fig. 4

    Split-display experiments.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nn1444

Further reading