Abstract
A framework to pinpoint the scope of unconscious processing is critical to improve models of visual consciousness. Previous research observed brain signatures of unconscious processing in visual cortex, but these were not reliably identified. Further, whether unconscious contents are represented in high-level stages of the ventral visual stream and linked parieto-frontal areas remains unknown. Using a within-subject, high-precision functional magnetic resonance imaging approach, we show that unconscious contents can be decoded from multi-voxel patterns that are highly distributed alongside the ventral visual pathway and also involving parieto-frontal substrates. Classifiers trained with multi-voxel patterns of conscious items generalized to predict the unconscious counterparts, indicating that their neural representations overlap. These findings suggest revisions to models of consciousness such as the neuronal global workspace. We then provide a computational simulation of visual processing/representation without perceptual sensitivity by using deep neural networks performing a similar visual task. The work provides a framework for pinpointing the representation of unconscious knowledge across different task domains.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The fMRI data can be found at https://openneuro.org/datasets/ds003927.
Code availability
Full scripts for the fMRI analyses and deep neural network simulations are available at https://github.com/nmningmei/unconfeats.
Change history
28 April 2022
A Correction to this paper has been published: https://doi.org/10.1038/s41562-022-01362-2
References
Dehaene, S. Consciousness and the Brain: Deciphering How the Brain Codes Our Thoughts (Penguin, 2014).
Lamme, V. A. F. Visual functions generating conscious seeing. Front. Psychol. https://doi.org/10.3389/fpsyg.2020.00083 (2020).
Van Gaal, S. & Lamme, V. A. F. Unconscious high-level information processing: implication for neurobiological theories of consciousness. Neuroscientist 18, 287–301 (2012).
Soto, D., Mäntylä, T. & Silvanto, J. Working memory without consciousness. Curr. Biol. 21, 912–913 (2011).
Trübutschek, D. et al. A theory of working memory without consciousness or sustained activity. eLife 6, e23871 (2017).
Rosenthal, C. R., Andrews, S. K., Antoniades, C. A., Kennard, C. & Soto, D. Learning and recognition of a non-conscious sequence of events in human primary visual cortex. Curr. Biol. 26, 834–841 (2016).
Chong, T. T.-J., Husain, M. & Rosenthal, C. R. Recognizing the unconscious. Curr. Biol. 24, 1033–1035 (2014).
Wuethrich, S., Hannula, D. E., Mast, F. W. & Henke, K. Subliminal encoding and flexible retrieval of objects in scenes. Hippocampus 28, 633–643 (2018).
Hassin, R. R. Yes it can. Perspect. Psychol. Sci. 8, 195–207 (2013).
Rabagliati, H., Robertson, A. & Carmel, D. The importance of awareness for understanding language. J. Exp. Psychol. Gen. 147, 190 (2018).
Kouider, S. & Dehaene, S. Levels of processing during non-conscious perception: a critical review of visual masking. Philos. Trans. R. Soc. B 362, 857–875 (2007).
Stein, T., Utz, V. & van Opstal, F. Unconscious semantic priming from pictures under backward masking and continuous flash suppression. Conscious. Cogn. 78, 102864 (2020).
Soto, D., Sheikh, U. A. & Rosenthal, C. R. A novel framework for unconscious processing. Trends Cogn. Sci. 23, 372–376 (2019).
Overgaard, M., Timmermans, B., Sandberg, K. & Cleeremans, A. Optimizing subjective measures of consciousness. Conscious. Cogn. 19, 682–684 (2010).
Peters, M.A.K. & Lau, H. Human observers have optimal introspective access to perceptual processes even for visually masked stimuli. eLife https://doi.org/10.7554/elife.09651 (2015).
Sterzer, P., Haynes, J. D. & Rees, G. Fine-scale activity patterns in high-level visual areas encode the category of invisible objects. J. Vis. 8, 10–10 (2008).
Haynes, J.-D. & Rees, G. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat. Neurosci. 8, 686–691 (2005).
Jiang, Y., Zhou, K. & He, S. Human visual cortex responds to invisible chromatic flicker. Nat. Neurosci. 10, 657–662 (2007).
Dehaene, S. et al. Cerebral mechanisms of word masking and unconscious repetition priming. Nat. Neurosci. 4, 752–758 (2001a).
Fang, F. & He, S. Cortical responses to invisible objects in the human dorsal and ventral pathways. Nat. Neurosci. 8, 1380–1385 (2005).
Hesselmann, G., Hebart, M. & Malach, R. Differential BOLD activity associated with subjective and objective reports during ‘blindsight’ in normal observers. J. Neurosci. 31, 12936–12944 (2011).
Ludwig, K. & Hesselmann, G. Weighing the evidence for a dorsal processing bias under continuous flash suppression. Conscious. Cogn. 35, 251–259 (2015).
Ludwig, K., Kathmann, N., Sterzer, P. & Hesselmann, G. Investigating category-and shape-selective neural processing in ventral and dorsal visual stream under interocular suppression. Human Brain Mapp. 36, 137–149 (2015).
Macmillan, N. A. The psychophysics of subliminal perception. Behav. Brain Sci. 9, 38–39 (1986).
Newell, B. R. & Shanks, D. R. Unconscious influences on decision making: a critical review. Behav. Brain Sci. 37, 1–19 (2014).
Kravitz, D. J., Saleem, K. S., Baker, C. I., Ungerleider, L. G. & Mishkin, M. The ventral visual pathway: an expanded neural framework for the processing of object quality. Trends Cogn. Sci. 17, 26–49 (2013).
Haxby, J. V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).
Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fMRI. NeuroImage 56, 400–410 (2011).
Kriegeskorte, N. Pattern-information analysis: from stimulus decoding to computational-model testing. NeuroImage 56, 411–421 (2011).
Ester, E. F., Sprague, T. C. & Serences, J. T. Parietal and frontal cortex encode stimulus-specific mnemonic representations during visual working memory. Neuron 87, 893–905 (2015).
Christophel, T. B., Hebart, M. N. & Haynes, J.-D. Decoding the contents of visual short-term memory from human visual and parietal cortex. J. Neurosci. 32, 12983–12989 (2012).
Kapoor, V. et al. Decoding the contents of consciousness from prefrontal ensembles. Preprint at bioRxiv https://doi.org/10.1101/2020.01.28.921841 (2020).
Schurger, A., Pereira, F., Treisman, A. & Cohen, J. D. Reproducibility distinguishes conscious from nonconscious neural representations. Science 327, 97–99 (2010).
Dehaene, S., Changeux, J.-P., Naccache, L., Sackur, J. & Sergent, C. Conscious, preconscious, and subliminal processing: a testable taxonomy. Trends Cogn. Sci. 10, 204–211 (2006).
G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. Preprint at https://arxiv.org/abs/1503.02531 (2015).
LeCun, Y. & Bengio, Y. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) (MIT Press, 1995).
McFee, B., Salamon, J. & Bello, J. P. Adaptive pooling operators for weakly labeled sound event detection. IEEE/ACM Trans. Audio Speech Lang. Process. 26, 2180–2193 (2018).
Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (MIT Press, 1982).
Kriegeskorte, N. & Douglas, P. K. Cognitive computational neuroscience. Nat. Neurosci. 21, 1148–1160 (2018).
Kietzmann, T. C., McClure, P., & Kriegeskorte, N. in Oxford Research Encyclopaedia of Neuroscience https://doi.org/10.1093/acrefore/9780190264086.013.46 (2019a).
Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356 (2016).
Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1, 417–446 (2015).
Moreno-Martínez, F. J. & Montoro, P. R. An ecological alternative to Snodgrass & Vanderwart: 360 high quality colour images with norms for seven psycholinguistic variables. PloS ONE 7, e37527 (2012).
Aru, J., Bachmann, T., Singer, W. & Melloni, L. Distilling the neural correlates of consciousness. Neurosci. Biobehav. Rev. 36, 737–746 (2012).
Zhang, J. & Mueller, S. T. A note on ROC analysis and non-parametric estimate of sensitivity. Psychometrika 70, 203–212 (2005).
Stein, T., Kaiser, D., Fahrenfort, J. J. & Van Gaal, S. The human visual system differentially represents subjectively and objectively invisible stimuli. PLoS Biol. 19, e3001241 (2021).
Jamalabadi, H., Alizadeh, S., Schönauer, M., Leibold, C. & Gais, S. Classification based hypothesis testing in neuroscience: below-chance level classification rates and overlooked statistical properties of linear parametric classifiers. Human Brain Mapp. 37, 1842–1855 (2016).
Snoek, L., Miletić, S. & Scholte, H. S. How to control for confounds in decoding analyses of neuroimaging data. NeuroImage 184, 741–760 (2019).
Ben-David, S. et al. A theory of learning from different domains. Mach. Learn. 79, 151–175 (2010).
Fukushima, K. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36, 193–202 (1980).
Kubilius, J., Bracci, S. & Op de Beeck, H. P. Deep neural networks as a computational model for human shape sensitivity. PLoS Comput. Biol. 12, e1004896 (2016).
Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).
Güçlü, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).
Kriegeskorte, N. et al. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60, 1126–1141 (2008a).
Fisher, A., Rudin, C. & Dominici, F. All models are wrong but many are useful: variable importance for black-box, proprietary, or misspecified prediction models, using model class reliance. Preprint at https://arxiv.org/abs/1801.01489 (2018).
Altmann, A., Toloşi, L., Sander, O. & Lengauer, T. Permutation importance: a corrected feature importance measure. Bioinformatics 26, 1340–1347 (2010).
Kriegeskorte, N., Mur, M. & Bandettini, P. A. Representational similarity analysis-connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008b).
Michel, M. Consciousness science underdetermined: a short history of endless debates. Ergo https://doi.org/10.3998/ergo.12405314.0006.028 (2019)
Gayet, S. et al. No evidence for mnemonic modulation of interocularly suppressed visual input. NeuroImage 215, 116801 (2020).
Kriegeskorte, N., Goebel, R. & Bandettini, P. Information-based functional brain mapping. Proc. Natl Acad. Sci. USA 103, 3863–3868 (2006).
Dehaene, S. & Naccache, L. Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework. Cognition 79, 1–37 (2001).
Dehaene, S. & Changeux, J.-P. Neural mechanisms for access to consciousness. Cogn. Neurosci. 3, 1145–58 (2004).
Dehaene, S. et al. Cerebral mechanisms of word masking and unconscious repetition priming. Nat. Neurosci. 4, 752–758 (2001b).
Haynes, J.-D., Driver, J. & Rees, G. Visibility reflects dynamic changes of effective connectivity between v1 and fusiform cortex. Neuron 46, 811–821 (2005).
Beck, D. M., Rees, G., Frith, C. D. & Lavie, N. Neural correlates of change detection and change blindness. Nat. Neurosci. 4, 645–650 (2001).
Pessoa, L. & Ungerleider, L. G. Neural correlates of change detection and change blindness in a working memory task. Cereb. Cortex 14, 511–520 (2004).
Kranczioch, C., Debener, S., Schwarzbach, J., Goebel, R. & Engel, A. K. Neural correlates of conscious perception in the attentional blink. NeuroImage 24, 704–714 (2005).
Yamins, D. L., Hong, H., Cadieu, C. & DiCarlo, J. J. Hierarchical modular optimization of convolutional networks achieves representations similar to macaque IT and human ventral stream. In Advances in Neural Information Processing Systems 26 (eds Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z. & Weinberger, K. Q.) 3093–3101 (Curran Associates, Inc., 2013).
Geirhos, R. et al. Comparing deep neural networks against humans: object recognition when the signal gets weaker. Preprint at https://arxiv.org/abs/1706.06969 (2017).
Wichmann, F. A. et al. Methods and measurements to compare men against machines. Electron. Imaging 2017, 36–45 (2017).
Geirhos, R. et al. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. Preprint at https://arxiv.org/abs/1811.12231 (2018).
Ghodrati, M., Farzmahdi, A., Rajaei, K., Ebrahimpour, R. & Khaligh-Razavi, S.-M. Feedforward object-vision models only tolerate small image variations compared to human. Front. Comput. Neurosci. 8, 74 (2014).
Kriegeskorte, N. & Diedrichsen, J. Peeling the onion of brain representations. Annu. Rev. Neurosci. 42, 407–432 (2019).
Shea, N. Representation in Cognitive Science (Oxford Univ. Press, 2018).
Lamme, V. A. F. & Roelfsema, P. R. The distinct modes of vision offered by feedforward and recurrent processing. Trends Neurosci. 23, 571–579 (2000).
Fahrenfort, J. J., Scholte, H. S. & Lamme, V. A. F. Masking disrupts reentrant processing in human visual cortex. J. Cogn. Neurosci. 19, 1488–1497 (2007).
DiCarlo, J. J., Zoccolan, D. & Rust, N. C. How does the brain solve visual object recognition? Neuron 73, 415–434 (2012).
Nayebi, A. et al. Task-driven convolutional recurrent models of the visual system. Preprint at https://arxiv.org/abs/1807.00053 (2018).
Bullier, J. Feedback connections and conscious vision. Trends Cogn. Sci. 5, 369–370 (2001).
Pascual-Leone, A. & Walsh, V. Fast backprojections from the motion to the primary visual area necessary for visual awareness. Science 292, 510–512 (2001).
Huang, L. et al. A source for awareness-dependent figure–ground segregation in human prefrontal cortex. Proc. Natl Acad. Sci. USA. 117, 30836–30847 (2020).
Cohen, M. A. & Dennett, D. C. Consciousness cannot be separated from function. Trends Cogn. Sci. 15, 358–364 (2011).
Soto, D. & Silvanto, J. Reappraising the relationship between working memory and conscious awareness. Trends Cogn. Sci. 18, 520–525 (2014).
Melloni, L. et al. Synchronization of neural activity across cortical areas correlates with conscious perception. J. Neurosci. 27, 2858–2865 (2007).
Koivisto, M., Mäntylä, T. & Silvanto, J. The role of early visual cortex (v1/v2) in conscious and unconscious visual perception. NeuroImage 51, 828–834 (2010).
Cohen, M. X., Van Gaal, S., Ridderinkhof, K. R. & Lamme, V. Unconscious errors enhance prefrontal-occipital oscillatory synchrony. Front. Human Neurosci. 3, 54 (2009).
Zwickel, T., Wachtler, T. & Eckhorn, R. Coding the presence of visual objects in a recurrent neural network of visual cortex. Biosystems 89, 216–226 (2007).
Spoerer, C. J., McClure, P. & Kriegeskorte, N. Recurrent convolutional neural networks: a better model of biological object recognition. Front. Psychol. 8, 1551 (2017).
Shi, J., Wen, H., Zhang, Y., Han, K. & Liu, Z. Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision. Human Brain Mapp. 39, 2269–2282 (2018).
Kietzmann, T. C. et al. Recurrence is required to capture the representational dynamics of the human visual system. Proc. Natl Acad. Sci. USA 116, 21854–21863 (2019b).
Zamir, A. R. et al. Feedback networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 1308–1317 (IEEE, 2017).
Liao, Q. & Poggio, T. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. Preprint at https://arxiv.org/abs/1604.03640 (2016).
Grodan, E. M. et al. Precision functional mapping of individual human brains. Neuron 95, 791–807 (2017).
Gratton, C. et al. Defining individual-specific functional neuroanatomy for precision psychiatry. Biol. Psychiatry 88, 28–39 (2020).
Chollet, F. et al. Keras: the Python deep learning library. Astrophysics Source Code Library, ascl-1806.022 (2018).
Peirce, J. W. Psychopy—Psychophysics software in Python. J. Neurosci. Methods 162, 8–13 (2007).
Horowitz, J. L. in Handbook of Econometrics (eds Heckman, J. J. & Leamer, E.) vol. 5, pp. 3159–3228 (Elsevier, 2001).
Ojala, M. & Garriga, G. C. Permutation tests for studying classifier performance. J. Mach. Learn. Res. 11, 1833–1863 (2010).
Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Abraham, A. et al. Machine learning for neuroimaging with scikit-learn. Front. Neuroinform. 8, 14 (2014).
Pereira, F. & Botvinick, M. Information mapping with pattern classifiers: a comparative study. NeuroImage 56, 476–496 (2011).
Lewis-Peacock, J. A. & Norman, K. A. Multi-voxel pattern analysis of fmri data. Cogn. Neurosci. 512, 911–920 (2014).
Bruha, I. From machine learning to knowledge discovery: survey of preprocessing and postprocessing. Intell. Data Anal. 4, 363–374 (2000).
King, J.-R. & Dehaene, S. Characterizing the dynamics of mental representations: the temporal generalization method. Trends Cogn. Sci. 18, 203–210 (2014).
Paszke, A. et al. Automatic Differentiation in pytorch. Neural Information Processing Systems Workshop Autodiff, https://openreview.net/forum?id=BJJsrmfCZ (2017).
Krizhevsky, A., Sutskever, I. & Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (eds Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q.) 25, 1097–1105 (Curran Associates, Inc., 2012).
Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at https://arxiv.org/abs/1409.1556 (2015).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (IEEE, 2016).
Howard, A.G. et al. Mobilenets: efficient convolutional neural networks for mobile vision applications. Preprint at https://arxiv.org/abs/1704.04861 (2017).
Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 4700–4708 (IEEE, 2017).
Deng, J. et al. ImageNet: A large-scale hierarchical image database. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 248–255 (IEEE, 2009).
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems 27 (eds Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. & Weinberger, K. Q.) 3320–3328 (Curran Associates, Inc., 2014).
Specht, D. F. Probabilistic neural networks and the polynomial adaline as complementary techniques for classification. IEEE Trans. Neural Netw. 1, 111–121 (1990).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).
Smith, S. M. Fast robust automated brain extraction. Human Brain Mapp. 17, 143–155 (2002).
Jenkinson, M., Bannister, P., Brady, M. & Smith, S. M. Improved optimization for the robust and accurate linear registration and motion correction of brain images. NeuroImage 17, 825–841 (2002).
Pruim, R. H. R. et al. Ica-aroma: A robust ica-based strategy for removing motion artifacts from fmri data. NeuroImage 112, 267–277 (2015).
Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W. & Smith, S. M. FSL. NeuroImage 62, 782–790 (2012).
Gorgolewski, K. et al. Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Front. Neuroinform. 5, 13 (2011).
Acknowledgements
D.S. acknowledges support from the Basque Government through the BERC 2018-2021 programme, from the Spanish Ministry of Economy and Competitiveness, through the ‘Severo Ochoa’ Programme for Centres/Units of Excellence in R & D (CEX2020-001010-S) and also from project grants PSI2016-76443-P and PID2019-105494GB-I00 from MINECO. R.S. acknowledges support by the Basque Government (IT1244-19 and ELKARTEK programmes), and the Spanish Ministry of Economy and Competitiveness MINECO (project TIN2016-78365-R). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
N.M. and D.S. designed the study. N.M. analysed the data under the guidance of R.S. and D.S. N.M. prepared a first draft of the paper. All authors discussed the results and contributed towards the writing of the final version of the manuscript. D.S. supervised the project.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Peer review
Peer review information
Nature Human Behaviour thanks Steven Scholte and Axel Cleeremans for their contribution to the peer review of this work. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary information
Supplementary Tables 1–5 and Figs. 1–18.
Rights and permissions
About this article
Cite this article
Mei, N., Santana, R. & Soto, D. Informative neural representations of unseen contents during higher-order processing in human brains and deep artificial networks. Nat Hum Behav 6, 720–731 (2022). https://doi.org/10.1038/s41562-021-01274-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-021-01274-7
This article is cited by
-
A behavioral advantage for the face pareidolia illusion in peripheral vision
Scientific Reports (2024)