Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the ‘how’ of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
Churchland, P. S. & Sejnowski, T. J. Perspectives on cognitive neuroscience. Science 242, 741–745 (1988).
Coltheart, M. How can functional neuroimaging inform cognitive theories? Perspect. Psychol. Sci. 8, 98–103 (2013).
Downing, P., Liu, J. & Kanwisher, N. Testing cognitive models of visual attention with fMRI and MEG. Neuropsychologia 39, 1329–1342 (2001).
Henson, R. What can functional neuroimaging tell the experimental psychologist? Q. J. Exp. Psychol. A 58, 193–233 (2005).
Mather, M., Cacioppo, J. T. & Kanwisher, N. How fMRI can inform cognitive theories. Perspect. Psychol. Sci. 8, 108–113 (2013).
Page, M. P. A. What can’t functional neuroimaging tell the cognitive psychologist? Cortex 42, 428–443 (2006).
Norman, K. A., Polyn, S. M., Detre, G. J. & Haxby, J. V. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10, 424–430 (2006).
Haynes, J.-D. A primer on pattern-based approaches to fMRI: principles, pitfalls, and perspectives. Neuron 87, 257–270 (2015).
Poldrack, R. A. & Farah, M. J. Progress and challenges in probing the human brain. Nature 526, 371–379 (2015).
Rissman, J. & Wagner, A. D. Distributed representations in memory: insights from functional brain imaging. Annu. Rev. Psychol. 63, 101–128 (2012).
Haxby, J. V., Connolly, A. C. & Guntupalli, J. S. Decoding neural representational spaces using multivariate pattern analysis. Annu. Rev. Neurosci. 37, 435–456 (2014).
Hebart, M. N. & Baker, C. I. Deconstructing multivariate decoding for the study of brain function. NeuroImage 180, 4–18 (2018).
Grootswagers, T., Wardle, S. G. & Carlson, T. A. Decoding dynamic brain patterns from evoked responses: a tutorial on multivariate pattern analysis applied to time series neuroimaging data. J. Cogn. Neurosci. 29, 677–697 (2017).
Carlson, T., Goddard, E., Kaplan, D. M., Klein, C. & Ritchie, J. B. Ghosts in machine learning for cognitive neuroscience: moving from data to theory. NeuroImage 180, 88–100 (2018).
de-Wit, L., Alexander, D., Ekroll, V. & Wagemans, J. Is neuroimaging measuring information in the brain? Psychon. Bull. Rev. 23, 1415–1428 (2016).
Poeppel, D. The maps problem and the mapping problem: two challenges for a cognitive neuroscience of speech and language. Cogn. Neuropsychol. 29, 34–55 (2012).
Lebedev, M. A. & Nicolelis, M. A. L. Brain–machine interfaces: from basic science to neuroprostheses and neurorehabilitation. Physiol. Rev. 97, 767–837 (2017).
Haxby, J. V. Multivariate pattern analysis of fMRI: the early beginnings. NeuroImage 62, 852–855 (2012).
Haynes, J.-D. & Rees, G. Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7, 523–534 (2006).
Tong, F. & Pratte, M. S. Decoding patterns of human brain activity. Annu. Rev. Psychol. 63, 483–509 (2012).
Cohen, J. D. et al. Computational approaches to fMRI analysis. Nat. Neurosci. 20, 304–313 (2017).
Kriegeskorte, N. & Douglas, P. K. Cognitive computational neuroscience. Nat. Neurosci. 21, 1148–1160 (2018).
Guest, O. & Love, B. C. What the success of brain imaging implies about the neural code. eLife 6, e21397 (2017).
Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).
Ritchie, J. B., Kaplan, D. M. & Klein, C. Decoding the brain: neural representation and the limits of multivariate pattern analysis in cognitive neuroscience. Br. J. Phil. Sci. 70, 581–607 (2019).
Kragel, P. A., Koban, L., Barrett, L. F. & Wager, T. D. Representation, pattern information, and brain signatures: from neurons to neuroimaging. Neuron 99, 257–273 (2018).
Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fMRI. NeuroImage 56, 400–410 (2011).
Sprague, T. C. & Serences, J. T. in An Introduction to Model-Based Cognitive Neuroscience (eds Forstmann, B. & Wagenmakers, E.-J.) 245–274 (Springer, 2015).
Kriegeskorte, N. & Douglas, P. K. Interpreting encoding and decoding models. Curr. Opin. Neurobiol. 55, 167–179 (2019).
Op de Beeck, H. P. Against hyperacuity in brain reading: spatial smoothing does not hurt multivariate fMRI analyses? NeuroImage 49, 1943–1948 (2010).
Swisher, J. D. et al. Multiscale pattern analysis of orientation-selective activity in the primary visual cortex. J. Neurosci. 30, 325–330 (2010).
Freeman, J., Brouwer, G. J., Heeger, D. J. & Merriam, E. P. Orientation decoding depends on maps, not columns. J. Neurosci. 31, 4792–4804 (2011).
Alink, A., Krugliak, A., Walther, A. & Kriegeskorte, N. fMRI orientation decoding in V1 does not require global maps or globally coherent orientation stimuli. Front. Psychol. 4, 493 (2013).
Carlson, T. A. Orientation decoding in human visual cortex: new insights from an unbiased perspective. J. Neurosci. 34, 8373–8383 (2014).
Roth, Z. N., Heeger, D. J. & Merriam, E. P. Stimulus vignetting and orientation selectivity in human visual cortex. eLife 7, e37241 (2018).
Marr, D. Vision: A Computational Approach (Freeman, 1982).
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A. & Poeppel, D. Neuroscience needs behavior: correcting a reductionist bias. Neuron 93, 480–490 (2017).
Neisser, U. Cognitive Psychology (Appleton-Century-Crofts, 1967).
Anderson, J. R. Cognitive Psychology and Its Implications (Worth, 2020).
Garner, W. R. The Processing of Information and Structure (Psychology Press, 1974).
Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl Acad. Sci. USA 111, 8619–8624 (2014).
Miyake, A. & Friedman, N. P. The nature and organization of individual differences in executive functions: four general conclusions. Curr. Dir. Psychol. Sci. 21, 8–14 (2012).
Cohen, J. D., Barch, D. M., Carter, C. & Servan-Schreiber, D. Context-processing deficits in schizophrenia: converging evidence from three theoretically motivated cognitive tasks. J. Abnorm. Psychol. 108, 120–133 (1999).
Markus, H. R. & Kitayama, S. Culture and the self: implications for cognition, emotion, and motivation. Psychol. Rev. 98, 224–253 (1991).
Kriegeskorte, N. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).
Mur, M., Bandettini, P. A. & Kriegeskorte, N. Revealing representational content with pattern-information fMRI—an introductory guide. Soc. Cogn. Affect. Neurosci. 4, 101–109 (2009).
Cichy, R. M., Pantazis, D. & Oliva, A. Resolving human object recognition in space and time. Nat. Neurosci. 17, 455–462 (2014).
Proklova, D., Kaiser, D. & Peelen, M. V. Disentangling representations of object shape and object category in human visual cortex: the animate–inanimate distinction. J. Cogn. Neurosci. 28, 680–692 (2016).
Cohen, M. A., Alvarez, G. A., Nakayama, K. & Konkle, T. Visual search for object categories is predicted by the representational architecture of high-level visual cortex. J. Neurophysiol. 117, 388–402 (2017).
Groen, I. I. et al. Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior. eLife 7, e32962 (2018).
Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A. & Oliva, A. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Sci. Rep. 6, 27755 (2016).
Smith, P. L. & Little, D. R. Small is beautiful: in defense of the small-N design. Psychon. Bull. Rev. 25, 2083–2101 (2018).
Meehl, P. E. Theory-testing in psychology and physics: a methodological paradox. Phil. Sci. 34, 103–115 (1967).
Kanwisher, N. & Wojciulik, E. Visual attention: insights from brain imaging. Nat. Rev. Neurosci. 1, 91–100 (2000).
Wagner, A. D. et al. Building memories: remembering and forgetting of verbal experiences as predicted by brain activity. Science 281, 1188–1191 (1998).
Curtis, C. E. & D’Esposito, M. Persistent activity in the prefrontal cortex during working memory. Trends Cogn. Sci. 7, 415–423 (2003).
Kastner, S., Pinsk, M. A., De Weerd, P., Desimone, R. & Ungerleider, L. G. Increased activity in human visual cortex during directed attention in the absence of visual stimulation. Neuron 22, 751–761 (1999).
Jiang, J., Summerfield, C. & Egner, T. Attention sharpens the distinction between expected and unexpected percepts in the visual brain. J. Neurosci. 33, 18438–18447 (2013).
Seidl, K. N., Peelen, M. V. & Kastner, S. Neural evidence for distracter suppression during visual search in real-world scenes. J. Neurosci. 32, 11812–11819 (2012).
Serences, J. T., Ester, E. F., Vogel, E. K. & Awh, E. Stimulus-specific delay activity in human primary visual cortex. Psychol. Sci. 20, 207–214 (2009).
Harrison, S. A. & Tong, F. Decoding reveals the contents of visual working memory in early visual areas. Nature 458, 632–635 (2009).
Polyn, S. M., Natu, V. S., Cohen, J. D. & Norman, K. A. Category-specific cortical activity precedes retrieval during memory search. Science 310, 1963–1966 (2005).
Kornysheva, K. et al. Neural competitive queuing of ordinal structure underlies skilled sequential action. Neuron 101, 1166–1180.e3 (2019).
Ariani, G., Pruszynski, J. A. & Diedrichsen, J. Motor planning brings human primary somatosensory cortex into action-specific preparatory states. eLife 11, e69517 (2022).
Oosterhof, N. N., Tipper, S. P. & Downing, P. E. Crossmodal and action-specific: neuroimaging the human mirror neuron system. Trends Cogn. Sci. 17, 311–318 (2013).
Kaplan, J. T., Man, K. & Greening, S. G. Multivariate cross-classification: applying machine learning techniques to characterize abstraction in neural representations. Front. Hum. Neurosci. 9, 151 (2015).
King, J.-R. & Dehaene, S. Characterizing the dynamics of mental representations: the temporal generalization method. Trends Cogn. Sci. 18, 203–210 (2014).
Haxby, J. V., Guntupalli, J. S., Nastase, S. A. & Feilong, M. Hyperalignment: modeling shared information encoded in idiosyncratic cortical topographies. eLife 9, e56601 (2020).
Tolman, E. C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).
Whittington, J. C. R., McCaffary, D., Bakermans, J. J. W. & Behrens, T. E. J. How to build a cognitive map. Nat. Neurosci. 25, 1257–1272 (2022).
Vass, L. K. & Epstein, R. A. Abstract representations of location and facing direction in the human brain. J. Neurosci. 33, 6133–6142 (2013).
Nielson, D. M., Smith, T. A., Sreekumar, V., Dennis, S. & Sederberg, P. B. Human hippocampus represents space and time during retrieval of real-world memories. Proc. Natl Acad. Sci. USA 112, 11078–11083 (2015).
Biederman, I., Mezzanotte, R. J. & Rabinowitz, J. C. Scene perception: detecting and judging objects undergoing relational violations. Cogn. Psychol. 14, 143–177 (1982).
Davenport, J. L. & Potter, M. C. Scene consistency in object and background perception. Psychol. Sci. 15, 559–564 (2004).
Bar, M. Visual objects in context. Nat. Rev. Neurosci. 5, 617–629 (2004).
Henderson, J. M. & Hollingworth, A. High-level scene perception. Annu. Rev. Psychol. 50, 243–271 (1999).
Brandman, T. & Peelen, M. V. Interaction between scene and object processing revealed by human fMRI and MEG decoding. J. Neurosci. 37, 7700–7710 (2017).
Chun, M. M., Golomb, J. D. & Turk-Browne, N. B. A taxonomy of external and internal attention. Annu. Rev. Psychol. 62, 73–101 (2011).
Desimone, R. & Duncan, J. Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222 (1995).
Duncan, J. & Humphreys, G. W. Visual search and stimulus similarity. Psychol. Rev. 96, 433–458 (1989).
Chelazzi, L., Miller, E. K., Duncan, J. & Desimone, R. A neural basis for visual search in inferior temporal cortex. Nature 363, 345–347 (1993).
Battistoni, E., Stein, T. & Peelen, M. V. Preparatory attention in visual cortex: preparatory attention in visual cortex. Ann. N. Y. Acad. Sci. 1396, 92–107 (2017).
Stokes, M., Thompson, R., Nobre, A. C. & Duncan, J. Shape-specific preparatory activity mediates attention to targets in human visual cortex. Proc. Natl Acad. Sci. USA 106, 19569–19574 (2009).
Peelen, M. V. & Kastner, S. A neural basis for real-world visual search in human occipitotemporal cortex. Proc. Natl Acad. Sci. USA 108, 12125–12130 (2011).
Gayet, S. & Peelen, M. V. Preparatory attention incorporates contextual expectations. Curr. Biol. 32, 687–692.e6 (2022).
Tulving, E. & Thomson, D. M. Encoding specificity and retrieval processes in episodic memory. Psychol. Rev. 80, 352–373 (1973).
Jafarpour, A., Fuentemilla, L., Horner, A. J., Penny, W. & Duzel, E. Replay of very early encoding representations during recollection. J. Neurosci. 34, 242–248 (2014).
Mack, M. L., Preston, A. R. & Love, B. C. Decoding the brain’s algorithm for categorization from its neural implementation. Curr. Biol. 23, 2023–2027 (2013).
Posner, M. I. & Keele, S. W. On the genesis of abstract ideas. J. Exp. Psychol. 77, 353–363 (1968).
Medin, D. L. & Schaffer, M. M. Context theory of classification learning. Psychol. Rev. 85, 207–238 (1978).
Barsalou, L. W. Perceptual symbol systems. Behav. Brain Sci. 22, 577–660 (1999).
Glenberg, A. M. Few believe the world is flat: how embodiment is changing the scientific understanding of cognition. Can. J. Exp. Psychol. 69, 165–171 (2015).
Fernandino, L., Tong, J.-Q., Conant, L. L., Humphries, C. J. & Binder, J. R. Decoding the information structure underlying the neural representation of concepts. Proc. Natl Acad. Sci. USA 119, e2108091119 (2022).
Levenson, R. W. Basic emotion questions. Emot. Rev. 3, 379–386 (2011).
Gendron, M. & Feldman Barrett, L. Reconstructing the past: a century of ideas about emotion in psychology. Emot. Rev. 1, 316–339 (2009).
Ekman, P. Universal and cultural differences in facial expressions of emotions. In Nebraska Symposium on Motivation, 1971 (ed. Cole, J.) 207–283 (University of Nebraska Press, 1972).
Saarimäki, H. et al. Discrete neural signatures of basic emotions. Cereb. Cortex 26, 2563–2573 (2016).
Skerry, A. E. & Saxe, R. Neural representations of emotion are organized around abstract event features. Curr. Biol. 25, 1945–1954 (2015).
Peelen, M. V., Atkinson, A. P. & Vuilleumier, P. Supramodal representations of perceived emotions in the human brain. J. Neurosci. 30, 10127–10134 (2010).
Ellsworth, P. C. Appraisal theory: old and new questions. Emot. Rev. 5, 125–131 (2013).
Scherer, K. R. The nature and dynamics of relevance and valence appraisals: theoretical advances and recent evidence. Emot. Rev. 5, 150–162 (2013).
Gallese, V. & Goldman, A. Mirror neurons and the simulation theory of mind-reading. Trends Cogn. Sci. 2, 493–501 (1998).
Saxe, R. Against simulation: the argument from error. Trends Cogn. Sci. 9, 174–179 (2005).
Corradi-Dell’Acqua, C., Tusche, A., Vuilleumier, P. & Singer, T. Cross-modal representations of first-hand and vicarious pain, disgust and fairness in insular and cingulate cortex. Nat. Commun. 7, 10904 (2016).
Koster-Hale, J., Bedny, M. & Saxe, R. Thinking about seeing: perceptual sources of knowledge are encoded in the theory of mind brain regions of sighted and blind adults. Cognition 133, 65–78 (2014).
von Holst, E. & Mittelstaedt, H. The reafference principle: interaction between the central nervous system and the periphery. Naturwissenschaften 37, 464–476 (1950).
Wolpert, D. M. & Flanagan, J. R. Motor prediction. Curr. Biol. 11, R729–R732 (2001).
Umeda, T., Isa, T. & Nishimura, Y. The somatosensory cortex receives information about motor output. Sci. Adv. 5, eaaw5388 (2019).
Gale, D. J., Flanagan, J. R. & Gallivan, J. P. Human somatosensory cortex is modulated during motor planning. J. Neurosci. 41, 5909–5922 (2021).
Terrace, H. S. The simultaneous chain: a new approach to serial learning. Trends Cogn. Sci. 9, 202–210 (2005).
Houghton, G. & Hartley, T. Parallel models of serial behaviour: Lashley revisited. Psyche 2, 25 (1995).
Williams, M. A., Dang, S. & Kanwisher, N. G. Only some spatial patterns of fMRI response are read out in task performance. Nat. Neurosci. 10, 685–686 (2007).
Ritchie, J. B. & Carlson, T. A. Neural decoding and ‘inner’ psychophysics: a distance-to-bound approach for linking mind, brain, and behavior. Front. Neurosci. 10, 190 (2016).
Wischnewski, M. & Peelen, M. V. Causal neural mechanisms of context-based object recognition. eLife 10, e69736 (2021).
Thielen, J., Bosch, S. E., van Leeuwen, T. M., van Gerven, M. A. J. & van Lier, R. Evidence for confounding eye movements under attempted fixation and active viewing in cognitive neuroscience. Sci. Rep. 9, 17456 (2019).
Proklova, D., Kaiser, D. & Peelen, M. V. MEG sensor patterns reflect perceptual but not categorical similarity of animate and inanimate objects. NeuroImage 193, 167–177 (2019).
Mole, C. & Klein, C. in Foundational Issues of Human Brain Mapping (eds Hanson, S. H. & Bunzl, M.) 99–112 (MIT Press, 2010).
Charest, I., Kievit, R. A., Schmitz, T. W., Deca, D. & Kriegeskorte, N. Unique semantic space in the brain of each beholder predicts perceived similarity. Proc. Natl Acad. Sci. USA 111, 14565–14570 (2014).
Anderson, A. J. et al. Decoding individual identity from brain activity elicited in imagining common experiences. Nat. Commun. 11, 5916 (2020).
Feilong, M., Guntupalli, J. S. & Haxby, J. V. The neural basis of intelligence in fine-grained cortical topographies. eLife 10, e64058 (2021).
Braunlich, K. & Love, B. C. Occipitotemporal representations reflect individual differences in conceptual knowledge. J. Exp. Psychol. Gen. 148, 1192–1203 (2019).
Anzellotti, S. & Coutanche, M. N. Beyond functional connectivity: investigating networks of multivariate representations. Trends Cogn. Sci. 22, 258–269 (2018).
Ju, H. & Bassett, D. S. Dynamic representations in networked neural systems. Nat. Neurosci. 23, 908–917 (2020).
van Gerven, M. A. J., Seeliger, K., Güçlü, U. & Güçlütürk, Y. in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (eds Samek, W. et al.) 379–394 (Springer International, 2019).
Lawrence, S. J. D., Formisano, E., Muckli, L. & de Lange, F. P. Laminar fMRI: applications for cognitive neuroscience. NeuroImage 197, 785–791 (2019).
De Vos, M. & Debener, S. Mobile EEG: towards brain activity monitoring during natural action and cognition. Int. J. Psychophysiol. 91, 1–2 (2014).
Snow, J. C. & Culham, J. C. The treachery of images: how realism influences brain and behavior. Trends Cogn. Sci. 25, 506–519 (2021).
Willems, R. M. & Peelen, M. V. How context changes the neural basis of perception and language. iScience 24, 102392 (2021).
van Rooij, I. & Baggio, G. Theory before the test: how to build high-verisimilitude explanatory theories in psychological science. Perspect. Psychol. Sci. 16, 682–697 (2021).
Guest, O. & Martin, A. E. How computational modeling can force theory building in psychological science. Perspect. Psychol. Sci. 16, 789–802 (2021).
Muthukrishna, M. & Henrich, J. A problem in theory. Nat. Hum. Behav. 3, 221–229 (2019).
Walther, A. et al. Reliability of dissimilarity measures for multi-voxel pattern analysis. NeuroImage 137, 188–200 (2016).
This project has received funding from the European Research Council under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 725970). The funder had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank F. de Lange, R. Willems and P. Medendorp for helpful comments on an earlier version of the manuscript.
The authors declare no competing interests.
Peer review information
Nature Human Behaviour thanks Adina Roskies, Thomas Carlson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Peelen, M.V., Downing, P.E. Testing cognitive theories with multivariate pattern analysis of neuroimaging data. Nat Hum Behav 7, 1430–1441 (2023). https://doi.org/10.1038/s41562-023-01680-z