Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Predictive processing of scenes and objects

Nature Reviews Psychology volume 3, pages 13–26 (2024)Cite this article

Subjects

Abstract

Real-world visual input consists of rich scenes that are meaningfully composed of multiple objects that interact in complex but predictable ways. Despite this complexity, humans can recognize scenes and objects within scenes from a brief glance at an image. In this Review, we synthesize behavioural and neural findings that elucidate the mechanisms underlying this impressive ability. First, we review evidence that visual object and scene processing is partly implemented in parallel, enabling rapid computation of an initial gist of objects and scenes concurrently. Next, we discuss bidirectional interactions between object and scene processing, with scene information modulating the visual processing of objects and object information modulating the visual processing of scenes. Finally, we review evidence that objects also combine with each other to form object constellations, modulating the processing of individual objects within the object pathway. Altogether, these findings can be understood by conceptualizing object and scene perception as the outcome of a joint probabilistic inference in which best guesses about objects act as priors for scene perception and vice versa.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Scene-selective and object-selective regions in human visual cortex and their relation to the centre–periphery organization.
Fig. 2: Bidirectional interactions between objects and scenes.
Fig. 3: Scene context can facilitate or impair object perception.
Fig. 4: Neural evidence for bidirectional interactions between object and scene processing.
Fig. 5: Object constellations.
Fig. 6: An integrated model of object–scene and object–object interactions.

Similar content being viewed by others

References

  1. Barlow, H. B. The knowledge used in vision and where it comes from. Phil. Trans. R. Soc. Lond. B 352, 1141–1147 (1997).

    Article  Google Scholar 

  2. Oliva, A. & Torralba, A. The role of context in object recognition. Trends Cogn. Sci. 11, 520–527 (2007).

    Article  PubMed  Google Scholar 

  3. Bar, M. Visual objects in context. Nat. Rev. Neurosci. 5, 617–629 (2004). This comprehensive review of contextual facilitation of object processing links these effects to expectations derived from ‘context frames’.

    Article  PubMed  Google Scholar 

  4. Purves, D., Wojtach, W. T. & Lotto, R. B. Understanding vision in wholly empirical terms. Proc. Natl Acad. Sci. USA 108, 15588–15595 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Simoncelli, E. P. & Olshausen, B. A. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 1193–1216 (2001).

    Article  PubMed  Google Scholar 

  6. Kaiser, D., Quek, G. L., Cichy, R. M. & Peelen, M. V. Object vision in a structured world. Trends Cogn. Sci. 23, 672–685 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Malcolm, G. L., Groen, I. I. A. & Baker, C. I. Making sense of real-world scenes. Trends Cogn. Sci. 20, 843–856 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Epstein, R. A. Neural systems for visual scene recognition. In Scene Vision (eds Kveraga, K. & Bar, M.) 105–134 (The MIT Press, 2014).

  9. Henderson, J. M. & Hollingworth, A. High-level scene perception. Annu. Rev. Psychol. 50, 243–271 (1999).

    Article  PubMed  Google Scholar 

  10. Biederman, I., Mezzanotte, R. J. & Rabinowitz, J. C. Scene perception: detecting and judging objects undergoing relational violations. Cogn. Psychol. 14, 143–177 (1982).

    Article  PubMed  Google Scholar 

  11. Palmer, S. E. Visual perception and world knowledge: notes on a model of sensory-cognitive interaction. In Explorations in Cognition 279–307 (Freeman, 1975).

  12. Hollingworth, A. & Henderson, J. M. Does consistent scene context facilitate object perception? J. Exp. Psychol. Gen. 127, 398–415 (1998).

    Article  PubMed  Google Scholar 

  13. Clark, A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 36, 181–204 (2013).

    Article  PubMed  Google Scholar 

  14. Spratling, M. W. A review of predictive coding algorithms. Brain Cogn. 112, 92–97 (2017).

    Article  PubMed  Google Scholar 

  15. Grill-Spector, K. & Weiner, K. S. The functional architecture of the ventral temporal cortex and its role in categorization. Nat. Rev. Neurosci. 15, 536–548 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Levy, I., Hasson, U., Avidan, G., Hendler, T. & Malach, R. Center–periphery organization of human object areas. Nat. Neurosci. 4, 533–539 (2001).

    Article  PubMed  Google Scholar 

  17. Larson, A. M. & Loschky, L. C. The contributions of central versus peripheral vision to scene gist recognition. J. Vis. 9, 6 (2009).

    Article  Google Scholar 

  18. Trouilloud, A. et al. Rapid scene categorization: from coarse peripheral vision to fine central vision. Vis. Res. 170, 60–72 (2020).

    Article  PubMed  Google Scholar 

  19. Grill-Spector, K. & Malach, R. The human visual cortex. Annu. Rev. Neurosci. 27, 649–677 (2004).

    Article  PubMed  Google Scholar 

  20. Epstein, R. & Kanwisher, N. A cortical representation of the local visual environment. Nature 392, 598–601 (1998).

    Article  PubMed  Google Scholar 

  21. Epstein, R. A. & Baker, C. I. Scene perception in the human brain. Annu. Rev. Vis. Sci. 5, 373–397 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  22. Bar, M. & Aminoff, E. Cortical analysis of visual context. Neuron 38, 347–358 (2003).

    Article  PubMed  Google Scholar 

  23. Hasson, U., Levy, I., Behrmann, M., Hendler, T. & Malach, R. Eccentricity bias as an organizing principle for human high-order object areas. Neuron 34, 479–490 (2002).

    Article  PubMed  Google Scholar 

  24. Kravitz, D. J., Peng, C. S. & Baker, C. I. Real-world scene representations in high-level visual cortex: it’s the spaces more than the places. J. Neurosci. 31, 7322–7333 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  25. Park, S., Brady, T. F., Greene, M. R. & Oliva, A. Disentangling scene content from spatial boundary: complementary roles for the parahippocampal place area and lateral occipital complex in representing real-world scenes. J. Neurosci. 31, 1333–1340 (2011). This paper used a factorial design to show that scene-selective cortex represents a scene’s spatial boundaries whereas object-selective cortex represents a scene’s object content.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Harel, A., Kravitz, D. J. & Baker, C. I. Deconstructing visual scenes in cortex: gradients of object and spatial layout information. Cereb. Cortex 23, 947–957 (2013).

    Article  PubMed  Google Scholar 

  27. Walther, D. B., Caddigan, E., Fei-Fei, L. & Beck, D. M. Natural scene categories revealed in distributed patterns of activity in the human brain. J. Neurosci. 29, 10573–10581 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  28. Dilks, D. D., Julian, J. B., Paunov, A. M. & Kanwisher, N. The occipital place area is causally and selectively involved in scene perception. J. Neurosci. 33, 1331–1336 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  29. Wischnewski, M. & Peelen, M. V. Causal evidence for a double dissociation between object- and scene-selective regions of visual cortex: a preregistered TMS replication study. J. Neurosci. 41, 751–756 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  30. Ganaden, R. E., Mullin, C. R. & Steeves, J. K. E. Transcranial magnetic stimulation to the transverse occipital sulcus affects scene but not object processing. J. Cogn. Neurosci. 25, 961–968 (2013).

    Article  PubMed  Google Scholar 

  31. Mullin, C. R. & Steeves, J. K. E. TMS to the lateral occipital cortex disrupts object processing but facilitates scene processing. J. Cogn. Neurosci. 23, 4174–4184 (2011).

    Article  PubMed  Google Scholar 

  32. Troiani, V., Stigliani, A., Smith, M. E. & Epstein, R. A. Multiple object properties drive scene-selective regions. Cereb. Cortex 24, 883–897 (2014).

    Article  PubMed  Google Scholar 

  33. Aminoff, E. M. & Durham, T. Scene-selective brain regions respond to embedded objects of a scene. Cereb. Cortex 33, 5066–5074 (2022).

    Article  Google Scholar 

  34. Brandman, T. & Peelen, M. V. Interaction between scene and object processing revealed by human fMRI and MEG decoding. J. Neurosci. 37, 7700–7710 (2017). This study used functional magnetic resonance imaging and magnetoencephalography to show that scenes sharpen object representations in object-selective cortex at 300 ms after scene onset.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Kravitz, D. J., Saleem, K. S., Baker, C. I., Ungerleider, L. G. & Mishkin, M. The ventral visual pathway: an expanded neural framework for the processing of object quality. Trends Cogn. Sci. 17, 26–49 (2013).

    Article  PubMed  Google Scholar 

  36. Andrews, T. J., Clarke, A., Pell, P. & Hartley, T. Selectivity for low-level features of objects in the human ventral stream. NeuroImage 49, 703–711 (2010).

    Article  PubMed  Google Scholar 

  37. Groen, I. I. A., Silson, E. H. & Baker, C. I. Contributions of low- and high-level properties to neural processing of visual scenes in the human brain. Phil. Trans. R. Soc. B 372, 20160102 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  38. Op de Beeck, H. P., Haushofer, J. & Kanwisher, N. G. Interpreting fMRI data: maps, modules and dimensions. Nat. Rev. Neurosci. 9, 123–135 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  39. Biederman, I. Perceiving real-world scenes. Science 177, 77–80 (1972).

    Article  PubMed  Google Scholar 

  40. Rossel, P., Peyrin, C., Roux-Sibilon, A. & Kauffmann, L. It makes sense, so I see it better! Contextual information about the visual environment increases its perceived sharpness. J. Exp. Psychol. Hum. Percept. Perform. 48, 331–350 (2022). This paper presents behavioural evidence that scene-based expectations make objects look sharper.

    Article  PubMed  Google Scholar 

  41. Wischnewski, M. & Peelen, M. V. Causal neural mechanisms of context-based object recognition. eLife 10, e69736 (2021). This paper reports a pre-registered transcranial magnetic stimulation study that shows that feedback to object-selective cortex is necessary for accurate context-based object recognition.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Wolfe, J. M., Võ, M. L.-H., Evans, K. K. & Greene, M. R. Visual search in scenes involves selective and nonselective pathways. Trends Cogn. Sci. 15, 77–84 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  43. Peelen, M. V. & Kastner, S. Attention in the real world: toward understanding its neural basis. Trends Cogn. Sci. 18, 242–250 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  44. Castelhano, M. S. & Krzyś, K. Rethinking space: a review of perception, attention, and memory in scene processing. Annu. Rev. Vis. Sci. 6, 563–586 (2020).

    Article  PubMed  Google Scholar 

  45. Võ, M. L.-H., Boettcher, S. E. & Draschkow, D. Reading scenes: how scene grammar guides attention and aids perception in real-world environments. Curr. Opin. Psychol. 29, 205–210 (2019).

    Article  PubMed  Google Scholar 

  46. Friedman, A. Framing pictures: the role of knowledge in automatized encoding and memory for gist. J. Exp. Psychol. Gen. 108, 316–355 (1979).

    Article  PubMed  Google Scholar 

  47. Henderson, J. M., Weeks, P. A. Jr & Hollingworth, A. The effects of semantic consistency on eye movements during complex scene viewing. J. Exp. Psychol. Hum. Percept. Perform. 25, 210–228 (1999).

    Article  Google Scholar 

  48. Cornelissen, T. H. W. & Võ, M. L.-H. Stuck on semantics: processing of irrelevant object-scene inconsistencies modulates ongoing gaze behavior. Attent. Percept. Psychophys. 79, 154–168 (2017).

    Article  Google Scholar 

  49. Underwood, G., Templeman, E., Lamming, L. & Foulsham, T. Is attention necessary for object identification? Evidence from eye movements during the inspection of real-world scenes. Conscious. Cogn. 17, 159–170 (2008).

    Article  PubMed  Google Scholar 

  50. LaPointe, M. R. P., Lupianez, J. & Milliken, B. Context congruency effects in change detection: opposing effects on detection and identification. Vis. Cogn. 21, 99–122 (2013).

    Article  Google Scholar 

  51. Ortiz-Tudela, J., Jiménez, L. & Lupiáñez, J. Scene-object semantic incongruity across stages of processing: from detection to identification and episodic encoding. Front. Cogn. 2, 1125145 (2023).

    Article  Google Scholar 

  52. Loftus, G. R. & Mackworth, N. H. Cognitive determinants of fixation location during picture viewing. J. Exp. Psychol. Hum. Percept. Perform. 4, 565–572 (1978).

    Article  PubMed  Google Scholar 

  53. Hollingworth, A. & Henderson, J. M. Semantic informativeness mediates the detection of changes in natural scenes. Vis. Cogn. 7, 213–235 (2000).

    Article  Google Scholar 

  54. Rensink, R. A., O’Regan, J. K. & Clark, J. J. To see or not to see: the need for attention to perceive changes in scenes. Psychol. Sci. 8, 368–373 (1997).

    Article  Google Scholar 

  55. Spaak, E., Peelen, M. V. & de Lange, F. P. Scene context impairs perception of semantically congruent objects. Psychol. Sci. 33, 299–313 (2022). This paper shows that scene-incongruent objects are processed better than scene-congruent objects when sensory input is unambiguous.

    Article  PubMed  Google Scholar 

  56. Öhlschläger, S. & Võ, M. L.-H. SCEGRAM: an image database for semantic and syntactic inconsistencies in scenes. Behav. Res. 49, 1780–1791 (2017).

    Article  Google Scholar 

  57. Wiesmann, S. L. & Võ, M. L.-H. What makes a scene? Fast scene categorization as a function of global scene information at different resolutions. J. Exp. Psychol. Hum. Percept. Perform. 48, 871–888 (2022).

    Article  PubMed  Google Scholar 

  58. Davenport, J. L. & Potter, M. C. Scene consistency in object and background perception. Psychol. Sci. 15, 559–564 (2004).

    Article  PubMed  Google Scholar 

  59. Davenport, J. L. Consistency effects between objects in scenes. Mem. Cogn. 35, 393–401 (2007).

    Article  Google Scholar 

  60. Leroy, A., Faure, S. & Spotorno, S. Reciprocal semantic predictions drive categorization of scene contexts and objects even when they are separate. Sci. Rep. 10, 8447 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  61. Joubert, O. R., Rousselet, G. A., Fize, D. & Fabre-Thorpe, M. Processing scene context: fast categorization and object interference. Vis. Res. 47, 3286–3297 (2007).

    Article  PubMed  Google Scholar 

  62. Furtak, M., Mudrik, L. & Bola, M. The forest, the trees, or both? Hierarchy and interactions between gist and object processing during perception of real-world scenes. Cognition 221, 104983 (2022).

    Article  PubMed  Google Scholar 

  63. Linsley, D. & MacEvoy, S. P. Encoding-stage crosstalk between object- and spatial property-based scene processing pathways. Cereb. Cortex 25, 2267–2281 (2015). This functional magnetic resonance imaging study shows that objects can bias the encoding of spatial scene properties in scene-selective cortex.

    Article  PubMed  Google Scholar 

  64. Brandman, T. & Peelen, M. V. Signposts in the fog: objects facilitate scene representations in left scene-selective cortex. J. Cogn. Neurosci. 31, 390–400 (2019).

    Article  PubMed  Google Scholar 

  65. Mullally, S. L. & Maguire, E. A. A new role for the parahippocampal cortex in representing space. J. Neurosci. 31, 7441–7449 (2011). This paper shows that objects that evoke a sense of space activate the scene-selective parahippocampal place area.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Bonner, M. F. & Epstein, R. A. Object representations in the human brain reflect the co-occurrence statistics of vision and language. Nat. Commun. 12, 4081 (2021). This paper shows that responses in object-selective cortex reflect the semantic similarity of objects whereas responses in scene-selective cortex reflect the contextual similarity of objects.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Schyns, P. G. & Oliva, A. From blobs to boundary edges: evidence for time- and spatial-scale-dependent scene recognition. Psychol. Sci. 5, 195–200 (1994).

    Article  Google Scholar 

  68. Oliva, A. & Torralba, A. Building the gist of a scene: the role of global image features in recognition. Prog. Brain Res. 155, 23–36 (2006).

    Article  PubMed  Google Scholar 

  69. Hochstein, S. & Ahissar, M. View from the top: review hierarchies and reverse hierarchies in the visual system. Neuron 36, 791–804 (2002).

    Article  PubMed  Google Scholar 

  70. Kauffmann, L., Ramanoël, S. & Peyrin, C. The neural bases of spatial frequency processing during scene perception. Front. Integr. Neurosci. 8, 37 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  71. Bullier, J. Integrated model of visual processing. Brain Res. Rev. 36, 96–107 (2001).

    Article  PubMed  Google Scholar 

  72. Bar, M. et al. Top-down facilitation of visual recognition. Proc. Natl Acad. Sci. USA 103, 449–454 (2006).

    Article  PubMed  PubMed Central  Google Scholar 

  73. Liu, H., Agam, Y., Madsen, J. R. & Kreiman, G. Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron 62, 281–290 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  74. Thorpe, S., Fize, D. & Marlot, C. Speed of processing in the human visual system. Nature 381, 520–522 (1996).

    Article  PubMed  Google Scholar 

  75. Crouzet, S. M., Joubert, O. R., Thorpe, S. J. & Fabre-Thorpe, M. Animal detection precedes access to scene category. PLoS ONE 7, e51471 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  76. MacEvoy, S. P. & Epstein, R. A. Constructing scenes from objects in human occipitotemporal cortex. Nat. Neurosci. 14, 1323–1329 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  77. Fei-Fei, L., Iyer, A., Koch, C. & Perona, P. What do we perceive in a glance of a real-world scene? J. Vis. 7, 10 (2007). This paper shows that participants are equally likely to report object and scene features, even at very short presentation durations.

    Article  PubMed  Google Scholar 

  78. Chuyin, Z., Koh, Z. H., Gallagher, R., Nishimoto, S. & Tsuchiya, N. What can we experience and report on a rapidly presented image? Intersubjective measures of specificity of freely reported contents of consciousness. F1000Res 11, 69 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  79. Carlson, T., Tovar, D. A., Alink, A. & Kriegeskorte, N. Representational dynamics of object vision: the first 1000 ms. J. Vis. 13, 1 (2013).

    Article  PubMed  Google Scholar 

  80. Cichy, R. M., Pantazis, D. & Oliva, A. Resolving human object recognition in space and time. Nat. Neurosci. 17, 455–462 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  81. Kaiser, D., Azzalini, D. C. & Peelen, M. V. Shape-independent object category responses revealed by MEG and fMRI decoding. J. Neurophysiol. 115, 2246–2250 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  82. Brandman, T. & Peelen, M. V. Objects sharpen visual scene representations: evidence from MEG decoding. Cereb. Cortex 33, 9524–9531 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  83. Rao, R. P. N. & Ballard, D. H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79–87 (1999).

    Article  PubMed  Google Scholar 

  84. Friston, K. A theory of cortical responses. Phil. Trans. R. Soc. B 360, 815–836 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  85. Altmann, C. F., Bülthoff, H. H. & Kourtzi, Z. Perceptual organization of local elements into global shapes in the human visual cortex. Curr. Biol. 13, 342–349 (2003).

    Article  PubMed  Google Scholar 

  86. Teufel, C., Dakin, S. C. & Fletcher, P. C. Prior object-knowledge sharpens properties of early visual feature-detectors. Sci. Rep. 8, 10853 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  87. Neri, P. Global properties of natural scenes shape local properties of human edge detectors. Front. Psychol. 2, 172 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  88. Smith, F. W. & Muckli, L. Nonstimulated early visual areas carry information about surrounding context. Proc. Natl Acad. Sci. USA 107, 20099–20103 (2010). This paper provides functional magnetic resonance imaging evidence for context-based feedback to early visual cortex.

    Article  PubMed  PubMed Central  Google Scholar 

  89. Doya, K., Ishii, S., Pouget, A. & Rao, R. P. N. Bayesian Brain: Probabilistic Approaches to Neural Coding (MIT Press, 2006).

  90. Ma, W. J. Organizing probabilistic models of perception. Trends Cogn. Sci. 16, 511–518 (2012).

    Article  PubMed  Google Scholar 

  91. Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    Article  PubMed  Google Scholar 

  92. Werner, S. & Noppeney, U. Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J. Neurosci. 30, 2662–2675 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  93. Brandman, T., Avancini, C., Leticevscaia, O. & Peelen, M. V. Auditory and semantic cues facilitate decoding of visual object category in MEG. Cereb. Cortex 30, 597–606 (2020).

    PubMed  Google Scholar 

  94. Rossel, P., Peyrin, C. & Kauffmann, L. Subjective perception of objects depends on the interaction between the validity of context-based expectations and signal reliability. Vis. Res. 206, 108191 (2023).

    Article  PubMed  Google Scholar 

  95. Press, C., Kok, P. & Yon, D. The perceptual prediction paradox. Trends Cogn. Sci. 24, 13–24 (2020).

    Article  PubMed  Google Scholar 

  96. Brockmole, J. R. & Henderson, J. M. Recognition and attention guidance during contextual cueing in real-world scenes: evidence from eye movements. Q. J. Exp. Psychol. 59, 1177–1187 (2006).

    Article  Google Scholar 

  97. Boettcher, S. E. P., Draschkow, D., Dienhart, E. & Võ, M. L.-H. Anchoring visual search in scenes: assessing the role of anchor objects on eye movements during visual search. J. Vis. 18, 11 (2018).

    Article  PubMed  Google Scholar 

  98. de Graef, P., de Troy, A. & d’Ydewalle, G. Local and global contextual constraints on the identification of objects in scenes. Can. J. Psychol. 46, 489–508 (1992).

    Article  PubMed  Google Scholar 

  99. Henderson, J. M., Pollatsek, A. & Rayner, K. Effects of foveal priming and extrafoveal preview on object identification. J. Exp. Psychol. Hum. Percept. Perform. 13, 449–463 (1987).

    Article  PubMed  Google Scholar 

  100. Auckland, M. E., Cave, K. R. & Donnelly, N. Nontarget objects can influence perceptual processes during object recognition. Psychon. Bull. Rev. 14, 332–337 (2007).

    Article  PubMed  Google Scholar 

  101. Meyer, T. & Olson, C. R. Statistical learning of visual transitions in monkey inferotemporal cortex. Proc. Natl Acad. Sci. USA 108, 19401–19406 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  102. He, T., Richter, D., Wang, Z. & de Lange, F. P. Spatial and temporal context jointly modulate the sensory response within the ventral visual stream. J. Cogn. Neurosci. 34, 332–347 (2022).

    Article  PubMed  Google Scholar 

  103. Bar, M. & Ullman, S. Spatial context in recognition. Perception 25, 343–352 (1996).

    Article  PubMed  Google Scholar 

  104. Green, C. & Hummel, J. E. Familiar interacting object pairs are perceptually grouped. J. Exp. Psychol. Hum. Percept. Perform. 32, 1107–1119 (2006).

    Article  PubMed  Google Scholar 

  105. Riddoch, M. J., Humphreys, G. W., Edwards, S., Baker, T. & Willson, K. Seeing the action: neuropsychological evidence for action-based effects on object selection. Nat. Neurosci. 6, 82–89 (2003).

    Article  PubMed  Google Scholar 

  106. Stein, T., Kaiser, D. & Peelen, M. V. Interobject grouping facilitates visual awareness. J. Vis. 15, 10 (2015).

    Article  PubMed  Google Scholar 

  107. Wang, L., Weng, X. & He, S. Perceptual grouping without awareness: superiority of Kanizsa triangle in breaking interocular suppression. PLoS ONE 7, e40106 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  108. Kaiser, D., Stein, T. & Peelen, M. V. Object grouping based on real-world regularities facilitates perception by reducing competitive interactions in visual cortex. Proc. Natl Acad. Sci. USA 111, 11217–11222 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  109. Thorat, S., Quek, G. L. & Peelen, M. V. Statistical learning of distractor co-occurrences facilitates visual search. J. Vis. 22, 2 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  110. Gronau, N., Neta, M. & Bar, M. Integrated contextual representation for objects’ identities and their locations. J. Cogn. Neurosci. 20, 371–388 (2008).

    Article  PubMed  Google Scholar 

  111. Kim, J. G. & Biederman, I. Where do objects become scenes? Cereb. Cortex 21, 1738–1746 (2011).

    Article  PubMed  Google Scholar 

  112. Roberts, K. L. & Humphreys, G. W. Action relationships concatenate representations of separate objects in the ventral visual system. NeuroImage 52, 1541–1548 (2010).

    Article  PubMed  Google Scholar 

  113. Baldassano, C., Beck, D. M. & Fei-Fei, L. Human–object interactions are more than the sum of their parts. Cereb. Cortex 27, 2276–2288 (2016).

    PubMed Central  Google Scholar 

  114. Kaiser, D. & Peelen, M. V. Transformation from independent to integrative coding of multi-object arrangements in human visual cortex. NeuroImage 169, 334–341 (2018). This paper shows that functional magnetic resonance imaging response patterns to multiple-object displays in object-selective cortex do not resemble the average of individual-object response patterns when the objects are arranged in familiar positions.

    Article  PubMed  Google Scholar 

  115. Quek, G. L. & Peelen, M. V. Contextual and spatial associations between objects interactively modulate visual processing. Cereb. Cortex 30, 6391–6404 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  116. MacEvoy, S. P. & Epstein, R. A. Decoding the representation of multiple simultaneous objects in human occipitotemporal cortex. Curr. Biol. 19, 943–947 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  117. Reicher, G. M. Perceptual recognition as a function of meaningfulness of stimulus material. J. Exp. Psychol. 81, 275–280 (1969).

    Article  PubMed  Google Scholar 

  118. Lupyan, G. Objective effects of knowledge on visual perception. J. Exp. Psychol. Hum. Percept. Perform. 43, 794–806 (2017).

    Article  PubMed  Google Scholar 

  119. Heilbron, M., Richter, D., Ekman, M., Hagoort, P. & de Lange, F. P. Word contexts enhance the neural representation of individual letters in early visual cortex. Nat. Commun. 11, 321 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  120. Lee, T. S. The visual system’s internal model of the world. Proc. IEEE 103, 1359–1378 (2015).

    Article  Google Scholar 

  121. Munneke, J., Brentari, V. & Peelen, M. V. The influence of scene context on object recognition is independent of attentional focus. Front. Psychol. 4, 552 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  122. Mudrik, L., Breska, A., Lamy, D. & Deouell, L. Y. Integration without awareness: expanding the limits of unconscious processing. Psychol. Sci. 22, 764–770 (2011).

    Article  PubMed  Google Scholar 

  123. Faivre, N., Dubois, J., Schwartz, N. & Mudrik, L. Imaging object–scene relations processing in visible and invisible natural scenes. Sci. Rep. 9, 4567 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  124. Kim, M. et al. Anatomical correlates of the functional organization in the human occipitotemporal cortex. Magn. Reson. Imag. 24, 583–590 (2006).

    Article  Google Scholar 

  125. van Kesteren, M. T. R., Ruiter, D. J., Fernández, G. & Henson, R. N. How schema and novelty augment memory formation. Trends Neurosci. 35, 211–219 (2012).

    Article  PubMed  Google Scholar 

  126. Dehaene, S., Cohen, L., Sigman, M. & Vinckier, F. The neural code for written words: a proposal. Trends Cogn. Sci. 9, 335–341 (2005).

    Article  PubMed  Google Scholar 

  127. Josephs, E. L. & Konkle, T. Large-scale dissociations between views of objects, scenes, and reachable-scale environments in visual cortex. Proc. Natl Acad. Sci. USA 117, 29354–29362 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  128. De Vos, M. & Debener, S. Mobile EEG: towards brain activity monitoring during natural action and cognition. Int. J. Psychophysiol. 91, 1–2 (2014).

    Article  PubMed  Google Scholar 

  129. Helbing, J., Draschkow, D. & Võ, M. L -H. Auxiliary scene-context information provided by anchor objects guides attention and locomotion in natural search behavior. Psychol. Sci. 33, 1463–1476 (2022).

    Article  PubMed  Google Scholar 

  130. Shamay-Tsoory, S. G. & Mendelsohn, A. Real-life neuroscience: an ecological approach to brain and behavior research. Perspect. Psychol. Sci. 14, 841–859 (2019).

    Article  PubMed  Google Scholar 

  131. Matusz, P. J., Dikker, S., Huth, A. G. & Perrodin, C. Are we ready for real-world neuroscience? J. Cogn. Neurosci. 31, 327–338 (2019).

    Article  PubMed  Google Scholar 

  132. Willems, R. M. & Peelen, M. V. How context changes the neural basis of perception and language. iScience 24, 102392 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  133. Bainbridge, W. A., Hall, E. H. & Baker, C. I. Distinct representational structure and localization for visual encoding and recall during visual imagery. Cereb. Cortex 31, 1898–1913 (2021).

    Article  PubMed  Google Scholar 

  134. Malach, R., Levy, I. & Hasson, U. The topography of high-order human object areas. Trends Cogn. Sci. 6, 176–184 (2002).

    Article  PubMed  Google Scholar 

  135. Keller, G. B. & Mrsic-Flogel, T. D. Predictive processing: a canonical cortical computation. Neuron 100, 424–435 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  136. Lawrence, S. J. D., Formisano, E., Muckli, L. & de Lange, F. P. Laminar fMRI: applications for cognitive neuroscience. NeuroImage 197, 785–791 (2019).

    Article  PubMed  Google Scholar 

  137. Lawrence, S. J. D. et al. Laminar organization of working memory signals in human visual cortex. Curr. Biol. 28, 3435–3440.e4 (2018).

    Article  PubMed  Google Scholar 

  138. Muckli, L. et al. Contextual feedback to superficial layers of V1. Curr. Biol. 25, 2690–2695 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  139. Aitken, F. et al. Prior expectations evoke stimulus-specific activity in the deep layers of the primary visual cortex. PLoS Biol. 18, e3001023 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  140. Kok, P., Bains, L. J., van Mourik, T., Norris, D. G. & de Lange, F. P. Selective activation of the deep layers of the human primary visual cortex by top-down feedback. Curr. Biol. 26, 371–376 (2016).

    Article  PubMed  Google Scholar 

  141. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2012).

    Article  Google Scholar 

  142. VanRullen, R. Perception science in the age of deep neural networks. Front. Psychol. 8, 142 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  143. Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  144. Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A. & Oliva, A. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Sci. Rep. 6, 27755 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  145. Guclu, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  146. Eckstein, M. P., Koehler, K., Welbourne, L. E. & Akbas, E. Humans, but not deep neural networks, often miss giant targets in scenes. Curr. Biol. 27, 2827–2832.e3 (2017). This paper shows that deep neural networks were equally accurate at detecting consistently and inconsistently sized targets in scenes, unlike human participants.

    Article  PubMed  Google Scholar 

  147. Gayet, S. & Peelen, M. V. Preparatory attention incorporates contextual expectations. Curr. Biol. 32, 687–692.e6 (2022).

    Article  PubMed  Google Scholar 

  148. Katti, H., Peelen, M. V. & Arun, S. P. Machine vision benefits from human contextual expectations. Sci. Rep. 9, 2112 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  149. Zhu, Z., Xie, L. & Yuille, A. Object recognition with and without objects. In Proc 26th Int. Joint Conf. on AI (IJCAI) https://doi.org/10.24963/ijcai.2017/505 (IJCAI, 2017).

  150. Daucé, E., Albiges, P. & Perrinet, L. U. A dual foveal-peripheral visual processing model implements efficient saccade selection. J. Vis. 20, 22 (2020).

    Article  PubMed Central  Google Scholar 

  151. Akbas, E. & Eckstein, M. P. Object detection through search with a foveated visual system. PLoS Comput. Biol. 13, e1005743 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  152. Pramod, R. T., Katti, H. & Arun, S. P. Human peripheral blur is optimal for object recognition. Vis. Res. 200, 108083 (2022).

    Article  PubMed  Google Scholar 

  153. Chen, G. et al. A survey of the four pillars for small object detection: multiscale representation, contextual information, super-resolution, and region proposal. IEEE Trans. Syst. Man. Cybern. Syst. 52, 936–953 (2022).

    Article  Google Scholar 

  154. Xiang, W., Zhang, D.-Q., Yu, H. & Athitsos, V. Context-aware single-shot detector. In 2018 IEEE Winter Conf. on Applications of Computer Vision (WACV) 1784–1793 (IEEE, 2018).

  155. Wang, A. Y., Kay, K., Naselaris, T., Tarr, M. J. & Wehbe, L. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. Preprint at bioRxiv https://www.biorxiv.org/content/10.1101/2022.09.27.508760v1 (2022).

Download references

Acknowledgements

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements 725970 to M.V.P. and 101000942 to F.P.d.L.). E.B. is supported by The EMBO Postdoctoral Fellowship (award number ALTF 579-2021).

Author information

Authors and Affiliations

  1. Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands

    Marius V. Peelen, Eva Berlot & Floris P. de Lange

Authors
  1. Marius V. Peelen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eva Berlot
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Floris P. de Lange
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed to all aspects of the article.

Corresponding author

Correspondence to Marius V. Peelen.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Psychology thanks Chris Baker and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peelen, M.V., Berlot, E. & de Lange, F.P. Predictive processing of scenes and objects. Nat Rev Psychol 3, 13–26 (2024). https://doi.org/10.1038/s44159-023-00254-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s44159-023-00254-0

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing