Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Decoding the visual and subjective contents of the human brain

Nature Neuroscience volume 8, pages 679–685 (2005)Cite this article

Abstract

The potential for human neuroimaging to read out the detailed contents of a person's mental state has yet to be fully explored. We investigated whether the perception of edge orientation, a fundamental visual feature, can be decoded from human brain activity measured with functional magnetic resonance imaging (fMRI). Using statistical algorithms to classify brain states, we found that ensemble fMRI signals in early visual areas could reliably predict on individual trials which of eight stimulus orientations the subject was seeing. Moreover, when subjects had to attend to one of two overlapping orthogonal gratings, feature-based attention strongly biased ensemble activity toward the attended orientation. These results demonstrate that fMRI activity patterns in early visual areas, including primary visual cortex (V1), contain detailed orientation information that can reliably predict subjective perception. Our approach provides a framework for the readout of fine-tuned representations in the human brain and their subjective contents.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Orientation decoder and ensemble orientation selectivity.
Figure 2: Decoding stimulus orientation from ensemble fMRI activity in the visual cortex.
Figure 3: Orientation selectivity across the human visual pathway.
Figure 4: Pairwise decoding performance as a function of orientation difference (all pairs from eight orientations), for grating images (pixel intensities), fMRI images (voxel intensities) and transformed grating images.
Figure 5: Orientation preference map on flattened cortical surface.
Figure 6: Simulation of one-dimensional array of columns and voxels.
Figure 7: Mind-reading of attended orientation.

References

  1. Hubel, D.H. & Wiesel, T.N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. (Lond.) 160, 106–154 (1962).

    Article  CAS  Google Scholar 

  2. Hubel, D.H. & Wiesel, T.N. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. (Lond.) 195, 215–243 (1968).

    Article  CAS  Google Scholar 

  3. Blasdel, G.G. Orientation selectivity, preference, and continuity in monkey striate cortex. J. Neurosci. 12, 3139–3161 (1992).

    Article  CAS  Google Scholar 

  4. Bartfeld, E. & Grinvald, A. Relationships between orientation-preference pinwheels, cytochrome oxidase blobs, and ocular-dominance columns in primate striate cortex. Proc. Natl. Acad. Sci. USA 89, 11905–11909 (1992).

    Article  CAS  Google Scholar 

  5. Paradiso, M.A. A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biol. Cybern. 58, 35–49 (1988).

    Article  CAS  Google Scholar 

  6. Pouget, A., Dayan, P. & Zemel, R. Information processing with population codes. Nat. Rev. Neurosci. 1, 125–132 (2000).

    Article  CAS  Google Scholar 

  7. Haxby, J.V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).

    Article  CAS  Google Scholar 

  8. Cox, D.D. & Savoy, R.L. Functional magnetic resonance imaging (fMRI) 'brain reading': detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19, 261–270 (2003).

    Article  Google Scholar 

  9. Carlson, T.A., Schrater, P. & He, S. Patterns of activity in the categorical representations of objects. J. Cogn. Neurosci. 15, 704–717 (2003).

    Article  Google Scholar 

  10. Vanduffel, W., Tootell, R.B., Schoups, A.A. & Orban, G.A. The organization of orientation selectivity throughout macaque visual cortex. Cereb. Cortex 12, 647–662 (2002).

    Article  Google Scholar 

  11. Kim, D.S., Duong, T.Q. & Kim, S.G. High-resolution mapping of iso-orientation columns by fMRI. Nat. Neurosci. 3, 164–169 (2000).

    Article  CAS  Google Scholar 

  12. Engel, S.A., Glover, G.H. & Wandell, B.A. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb. Cortex 7, 181–192 (1997).

    Article  CAS  Google Scholar 

  13. Malonek, D. & Grinvald, A. Interactions between electrical activity and cortical microcirculation revealed by imaging spectroscopy: implications for functional brain mapping. Science 272, 551–554 (1996).

    Article  CAS  Google Scholar 

  14. Shtoyerman, E., Arieli, A., Slovin, H., Vanzetta, I. & Grinvald, A. Long-term optical imaging and spectroscopy reveal mechanisms underlying the intrinsic signal and stability of cortical maps in V1 of behaving monkeys. J. Neurosci. 20, 8111–8121 (2000).

    Article  CAS  Google Scholar 

  15. Vapnik, V.N. Statistical Learning Theory (Wiley, New York, 1998).

    Google Scholar 

  16. Minsky, L.M. & Papert, S.A. Perceptrons – Expanded Edition: An Introduction to Computational Geometry (MIT Press, Boston, 1987).

    Google Scholar 

  17. Furmanski, C.S. & Engel, S.A. An oblique effect in human primary visual cortex. Nat. Neurosci. 3, 535–536 (2000).

    Article  CAS  Google Scholar 

  18. Bauer, R. & Dow, B.M. Complementary global maps for orientation coding in upper and lower layers of the monkey's foveal striate cortex. Exp. Brain Res. 76, 503–509 (1989).

    Article  CAS  Google Scholar 

  19. Schall, J.D., Perry, V.H. & Leventhal, A.G. Retinal ganglion cell dendritic fields in old-world monkeys are oriented radially. Brain Res. 368, 18–23 (1986).

    Article  CAS  Google Scholar 

  20. Treue, S. & Maunsell, J.H. Attentional modulation of visual motion processing in cortical areas MT and MST. Nature 382, 539–541 (1996).

    Article  CAS  Google Scholar 

  21. Treue, S. & Martinez Trujillo, J.C. Feature-based attention influences motion processing gain in macaque visual cortex. Nature 399, 575–579 (1999).

    Article  CAS  Google Scholar 

  22. Treue, S. Visual attention: the where, what, how and why of saliency. Curr. Opin. Neurobiol. 13, 428–432 (2003).

    Article  CAS  Google Scholar 

  23. Roelfsema, P.R., Lamme, V.A. & Spekreijse, H. Object-based attention in the primary visual cortex of the macaque monkey. Nature 395, 376–381 (1998).

    Article  CAS  Google Scholar 

  24. Watanabe, T. et al. Task-dependent influences of attention on the activation of human primary visual cortex. Proc. Natl. Acad. Sci. USA 95, 11489–11492 (1998).

    Article  CAS  Google Scholar 

  25. Saenz, M., Buracas, G.T. & Boynton, G.M. Global effects of feature-based attention in human visual cortex. Nat. Neurosci. 5, 631–632 (2002).

    Article  CAS  Google Scholar 

  26. Wilson, H.R., Levi, D., Maffei, L., Rovamo, J. & DeValois, R. in Visual Perception: The Neurophysiological Foundations (eds. Spillman, L. & Werner, J.S.) 231–272 (Academic, San Diego, 1990).

    Book  Google Scholar 

  27. Tootell, R.B. et al. Functional analysis of primary visual cortex (V1) in humans. Proc. Natl. Acad. Sci. USA 95, 811–817 (1998).

    Article  CAS  Google Scholar 

  28. Boynton, G.M. & Finney, E.M. Orientation-specific adaptation in human visual cortex. J. Neurosci. 23, 8781–8787 (2003).

    Article  CAS  Google Scholar 

  29. Kamitani, Y. & Shimojo, S. Manifestation of scotomas created by transcranial magnetic stimulation of human visual cortex. Nat. Neurosci. 2, 767–771 (1999).

    Article  CAS  Google Scholar 

  30. Rees, G., Kreiman, G. & Koch, C. Neural correlates of consciousness in humans. Nat. Rev. Neurosci. 3, 261–270 (2002).

    Article  CAS  Google Scholar 

  31. Tong, F. Primary visual cortex and visual awareness. Nat. Rev. Neurosci. 4, 219–229 (2003).

    Article  CAS  Google Scholar 

  32. Koch, C. The Quest for Consciousness: A Neurobiological Approach (Roberts, Englewood, Colorado, USA, 2004).

    Google Scholar 

  33. Donoghue, J.P. Connecting cortex to machines: recent advances in brain interfaces. Nat. Neurosci. 5 (suppl.): 1085–1088 (2002).

    Article  CAS  Google Scholar 

  34. Wolpaw, J.R. & McFarland, D.J. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. USA 101, 17849–17854 (2004).

    Article  CAS  Google Scholar 

  35. Kahneman, D. & Wolman, R.E. Stroboscopic motion: Effects of duration and interval. Percept. Psychophys. 8, 161–164 (1970).

    Article  Google Scholar 

  36. Sereno, M.I. et al. Borders of multiple visual area in humans revealed by functional magnetic resonance imaging. Science 268, 889–893 (1995).

    Article  CAS  Google Scholar 

  37. Zeki, S. et al. A direct demonstration of functional specialization in human visual cortex. J. Neurosci. 11, 641–649 (1991).

    Article  CAS  Google Scholar 

  38. Watson, J.D. et al. Area V5 of the human brain: evidence from a combined study using positron emission tomography and magnetic resonance imaging. Cereb. Cortex 3, 79–94 (1993).

    Article  CAS  Google Scholar 

  39. Tootell, R.B. et al. Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging. J. Neurosci. 15, 3215–3230 (1995).

    Article  CAS  Google Scholar 

  40. Woods, R.P., Grafton, S.T., Holmes, C.J., Cherry, S.R. & Mazziotta, J.C. Automated image registration: I. General methods and intrasubject, intramodality validation. J. Comput. Assist. Tomogr. 22, 139–152 (1998).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank D. Remus and J. Kerlin for technical support, the Princeton Center for Brain, Mind and Behavior for MRI support and J. Haxby, D. Heeger and A. Seiffert for comments. This work was supported by grants from Japan Society for the Promotion of Science and National Institute of Information and Communications Technology to Y.K., and grants R01-EY14202 and P50-MH62196 from the US National Institutes of Health to F.T.

Author information

Author notes

  1. Frank Tong

    Present address: Psychology Department, Vanderbilt University, 301 Wilson Hall, 111 21st Avenue South, Nashville, Tennessee, 37203, USA

Authors and Affiliations

  1. ATR Computational Neuroscience Laboratories, 2-2-2 Hikaridai, Keihanna Science City, Kyoto, 619-0288, Japan

    Yukiyasu Kamitani

  2. Psychology Department, Princeton University, Green Hall, Princeton, 08544, New Jersey, USA

    Frank Tong

Authors
  1. Yukiyasu Kamitani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frank Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yukiyasu Kamitani.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Fig. 1

Average fMRI responses to orientations for each visual area. (PDF 80 kb)

Supplementary Fig. 2

Average fMRI responses for voxels with the same orientation preference, shown by visual area. (PDF 50 kb)

Supplementary Fig. 3

Map of differential responses to eight orientations. (PDF 507 kb)

Supplementary Fig. 4

Split-display experiments. (PDF 273 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kamitani, Y., Tong, F. Decoding the visual and subjective contents of the human brain. Nat Neurosci 8, 679–685 (2005). https://doi.org/10.1038/nn1444

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn1444

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing