Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Behavior and neural basis of near-optimal visual search

Nature Neuroscience volume 14, pages 783–790 (2011)Cite this article

Subjects

Abstract

The ability to search efficiently for a target in a cluttered environment is one of the most remarkable functions of the nervous system. This task is difficult under natural circumstances, as the reliability of sensory information can vary greatly across space and time and is typically a priori unknown to the observer. In contrast, visual-search experiments commonly use stimuli of equal and known reliability. In a target detection task, we randomly assigned high or low reliability to each item on a trial-by-trial basis. An optimal observer would weight the observations by their trial-to-trial reliability and combine them using a specific nonlinear integration rule. We found that humans were near-optimal, regardless of whether distractors were homogeneous or heterogeneous and whether reliability was manipulated through contrast or shape. We present a neural-network implementation of near-optimal visual search based on probabilistic population coding. The network matched human performance.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Reliability and inference in visual search.
Figure 2: Optimal search under unequal reliabilities.
Figure 3: Experimental procedure.
Figure 4: Model predictions for individual-subject receiver operating characteristics.
Figure 5: Model predictions for area under the ROC curve (AUC) in the mixed-reliability condition.
Figure 6: Log likelihood of nonoptimal models relative to the optimal model for individual subjects.
Figure 7: Neural implementation of near-optimal visual search.
Figure 8: Neural network reproduces human search performance.

Similar content being viewed by others

References

  1. Palmer, J., Ames, C.T. & Lindsey, D.T. Measuring the effect of attention on simple visual search. J. Exp. Psychol. Hum. Percept. Perform. 19, 108–130 (1993).

    Article  CAS  Google Scholar 

  2. Treisman, A.M. & Gelade, G. A feature-integration theory of attention. Cognit. Psychol. 12, 97–136 (1980).

    Article  CAS  Google Scholar 

  3. Estes, W.D. & Taylor, R.M. A detection method and probabilistic models for assessing information processing from brief visual displays. Proc. Natl. Acad. Sci. USA 52, 446–454 (1964).

    Article  CAS  Google Scholar 

  4. Shaw, M.L. Identifying attentional and decision-making components in information processing. in Attention and Performance (ed. R.S. Nickerson) 277–296 (Erlbaum, Hillsdale, New Jersey, 1980).

  5. Teichner, W.H. & Krebs, M.J. Visual search for simple targets. Psychol. Bull. 81, 15–28 (1974).

    Article  CAS  Google Scholar 

  6. Palmer, J., Verghese, P. & Pavel, M. The psychophysics of visual search. Vision Res. 40, 1227–1268 (2000).

    Article  CAS  Google Scholar 

  7. Duncan, J. & Humphreys, G.W. Visual search and stimulus similarity. Psychol. Rev. 96, 433–458 (1989).

    Article  CAS  Google Scholar 

  8. Rosenholtz, R. Visual search for orientation among heterogeneous distractors: experimental results and implications for signal detection theory models of search. J. Exp. Psychol. Hum. Percept. Perform. 27, 985–999 (2001).

    Article  CAS  Google Scholar 

  9. Eriksen, C.W. Object location in a complex perceptual field. J. Exp. Psychol. 45, 126–132 (1953).

    Article  CAS  Google Scholar 

  10. Farmer, E.W. & Taylor, R.M. Visual search through color displays: effects of target-background similarity and background uniformity. Percept. Psychophys. 27, 267–272 (1980).

    Article  CAS  Google Scholar 

  11. Knill, D.C. & Richards, W. Perception as Bayesian Inference (Cambridge University Press, New York, 1996).

  12. Knill, D.C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719 (2004).

    Article  CAS  Google Scholar 

  13. Ma, W.J., Beck, J.M., Latham, P.E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).

    Article  CAS  Google Scholar 

  14. Morgan, M.L., DeAngelis, G.C. & Angelaki, D.E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008).

    Article  CAS  Google Scholar 

  15. Verghese, P. Visual search and attention: a signal detection theory approach. Neuron 31, 523–535 (2001).

    Article  CAS  Google Scholar 

  16. Eckstein, M.P., Peterson, M.F., Pham, B.T. & Droll, J.A. Statistical decision theory to relate neurons to behavior in the study of covert visual attention. Vision Res. 49, 1097–1128 (2009).

    Article  Google Scholar 

  17. Najemnik, J. & Geisler, W.S. Optimal eye movement strategies in visual search. Nature 434, 387–391 (2005).

    Article  CAS  Google Scholar 

  18. Eckstein, M.P., Thomas, J.P., Palmer, J. & Shimozaki, S.S. A signal detection model predicts the effects of set size on visual search accuracy for feature, conjunction, triple conjunction, and disjunction displays. Percept. Psychophys. 62, 425–451 (2000).

    Article  CAS  Google Scholar 

  19. Green, D.M. & Swets, J.A. Signal Detection Theory and Psychophysics (John Wiley & Sons, Los Altos, California, 1966).

  20. Peterson, W.W., Birdsall, T.G. & Fox, W.C. The theory of signal detectability. IRE Prof. Group Inf. Theory 4, 171–212 (1954).

    Article  Google Scholar 

  21. Nolte, L.W. & Jaarsma, D. More on the detection of one of M orthogonal signals. J. Acoust. Soc. Am. 41, 497–505 (1967).

    Article  Google Scholar 

  22. Eckstein, M.P. The lower visual search efficiency for conjunctions is due to noise and not serial attentional processing. Psychol. Sci. 9, 111–118 (1998).

    Article  Google Scholar 

  23. Graham, N., Kramer, P. & Yager, D. Signal dection models for multidimensional stimuli: probability distributions and combination rules. J. Math. Psychol. 31, 366–409 (1987).

    Article  Google Scholar 

  24. Quick, R.F. A vector-magnitude model of contrast detection. Kybernetik 16, 65–67 (1974).

    Article  Google Scholar 

  25. Pouget, A., Dayan, P. & Zemel, R.S. Inference and Computation with Population Codes. Annu. Rev. Neurosci. 26, 381–410 (2003).

    Article  CAS  Google Scholar 

  26. Bremmer, F., Ilg, U., Thiele, A., Distler, C. & Hoffman, K. Eye position effects in monkey cortex. I. Visual and pursuit-related activity in extrastriate areas MT and MST. J. Neurophysiol. 77, 944–961 (1997).

    Article  CAS  Google Scholar 

  27. Andersen, R.A., Essick, G. & Siegel, R. Encoding of spatial location by posterior parietal neurons. Science 230, 456–458 (1985).

    Article  CAS  Google Scholar 

  28. Heeger, D.J. Normalization of cell responses in cat striate cortex. Vis. Neurosci. 9, 181–197 (1992).

    Article  CAS  Google Scholar 

  29. Reynolds, J.H. & Heeger, D.J. The normalization model of attention. Neuron 61, 168–185 (2009).

    Article  CAS  Google Scholar 

  30. Beck, J., Latham, P. & Pouget, A. Complex Bayesian inference in neural circuits using divisive normalization. Front. Syst. Neurosci. Conference Abstract: Computational and Systems Neuroscience 2009, doi: 10.3389/conf.neuro.06.2009.03.109 (2 February 2009).

  31. Seung, H.S. & Sompolinsky, H. Simple model for reading neuronal population codes. Proc. Natl. Acad. Sci. USA 90, 10749–10753 (1993).

    Article  CAS  Google Scholar 

  32. Schoups, A., Vogels, R., Qian, N. & Orban, G. Practising orientation identification improves orientation coding in V1 neurons. Nature 412, 549–553 (2001).

    Article  CAS  Google Scholar 

  33. Regan, D. & Beverley, K.I. Spatial–frequency discrimination and detection: comparison of postadaptation thresholds. J. Opt. Soc. Am. 73, 1684–1690 (1983).

    Article  CAS  Google Scholar 

  34. Kiani, R. & Shadlen, M.N. Representation of confidence associated with a decision by neurons in the parietal cortex. Science 324, 759–764 (2009).

    Article  CAS  Google Scholar 

  35. Kim, B. & Basso, M.A. Saccade target selection in the superior colliculus: a signal detection theory approach. J. Neurosci. 28, 2991–3007 (2008).

    Article  CAS  Google Scholar 

  36. Kepecs, A., Uchida, N., Zariwala, H.A. & Mainen, Z.F. Neural correlates, computation and behavioural impact of decision confidence. Nature 455, 227–231 (2008).

    Article  CAS  Google Scholar 

  37. Deneve, S., Latham, P. & Pouget, A. Reading population codes: a neural implementation of ideal observers. Nat. Neurosci. 2, 740–745 (1999).

    Article  CAS  Google Scholar 

  38. Ogawa, T. & Komatsu, H. Target selection in area V4 during a multidimensional visual search task. J. Neurosci. 24, 6371–6382 (2004).

    Article  CAS  Google Scholar 

  39. Bichot, N.P., Rossi, A.F. & Desimone, R. Parallel and serial neural mechanisms for visual search in macaque area V4. Science 308, 529–534 (2005).

    Article  CAS  Google Scholar 

  40. Gottlieb, J.P., Kusunoki, M. & Goldberg, M. The representation of visual salience in monkey parietal cortex. Nature 391, 481–484 (1998).

    Article  CAS  Google Scholar 

  41. Knill, D.C. Mixture models and the probabilistic structure of depth cues. Vision Res. 43, 831–854 (2003).

    Article  Google Scholar 

  42. Kersten, D., Mamassian, P. & Yuille, A. Object perception as Bayesian inference. Annu. Rev. Psychol. 55, 271–304 (2004).

    Article  Google Scholar 

  43. Itti, L. & Koch, C. Computational modeling of visual attention. Nat. Rev. Neurosci. 2, 194–203 (2001).

    Article  CAS  Google Scholar 

  44. Navalpakkam, V. & Itti, L. Search goal tunes visual features optimally. Neuron 53, 605–617 (2007).

    Article  CAS  Google Scholar 

  45. Zhang, L., Tong, M.H., Marks, T.K., Shan, H. & Cottrell, G.W. SUN: a Bayesian framework for saliency using natural statistics. J. Vis. 8, 1–20 (2008).

    CAS  PubMed  Google Scholar 

  46. Vincent, B.T., Baddeley, R.J., Troscianko, T. & Gilchrist, I.D. Optimal feature integration in visual search. J. Vis. 9, 1–11 (2009).

    Article  Google Scholar 

  47. Beck, J.M. et al. Bayesian decision-making with probabilistic population codes. Neuron 60, 1142–1152 (2008).

    Article  CAS  Google Scholar 

  48. Brainard, D.H. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).

    Article  CAS  Google Scholar 

  49. MacKay, D.J. Information Theory, Inference and Learning Algorithms (Cambridge University Press, Cambridge, UK, 2003).

  50. Cover, T.M. & Thomas, J.A. Elements of Information Theory (John Wiley & Sons, New York, 1991).

Download references

Acknowledgements

W.J.M. is supported by award R01EY020958 from the National Eye Institute. V.N. is supported by National Science Foundation grant #0820582. J.M.B. is supported by the Gatsby Charitable Foundation and R.v.d.B. by the Netherlands Organization for Scientific Research (NWO). A.P. is supported by Multidisciplinary University Research Initiative grant N00014-07-1-0937, National Institute on Drug Abuse grant #BCS0346785, a research grant from the James S. McDonnell Foundation and award P30EY001319 from the National Eye Institute.

Author information

Author notes

  1. Vidhya Navalpakkam

    Present address: Present address: Yahoo! Research, Santa Clara, California, USA.,

  2. Wei Ji Ma, Vidhya Navalpakkam, Jeffrey M Beck and Ronald van den Berg: These authors contributed equally to this work.

Authors and Affiliations

  1. Department of Neuroscience, Baylor College of Medicine, Houston, Texas, USA

    Wei Ji Ma & Ronald van den Berg

  2. Department of Biology, California Institute of Technology, Pasadena, California, USA

    Vidhya Navalpakkam

  3. Gatsby Computational Neuroscience Unit, University College London, London, UK

    Jeffrey M Beck

  4. Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York, USA

    Alexandre Pouget

Authors
  1. Wei Ji Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vidhya Navalpakkam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeffrey M Beck
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ronald van den Berg
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alexandre Pouget
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

W.J.M., V.N. and R.v.d.B. designed the experiments. V.N. and R.v.d.B. collected the data. W.J.M., V.N. and R.v.d.B. analyzed the data. W.J.M., J.B. and A.P. developed the theory. J.B. performed the network simulations. W.J.M. and A.P. wrote the manuscript. V.N., J.B. and R.v.d.B. contributed to the writing of the manuscript.

Corresponding author

Correspondence to Wei Ji Ma.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–16, Supplementary Tables 1 and 2, and Supplementary Results (PDF 3590 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ma, W., Navalpakkam, V., Beck, J. et al. Behavior and neural basis of near-optimal visual search. Nat Neurosci 14, 783–790 (2011). https://doi.org/10.1038/nn.2814

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.2814

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing