Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

The neural network RTNet exhibits the signatures of human perceptual decision-making

Abstract

Convolutional neural networks show promise as models of biological vision. However, their decision behaviour, including the facts that they are deterministic and use equal numbers of computations for easy and difficult stimuli, differs markedly from human decision-making, thus limiting their applicability as models of human perceptual behaviour. Here we develop a new neural network, RTNet, that generates stochastic decisions and human-like response time (RT) distributions. We further performed comprehensive tests that showed RTNet reproduces all foundational features of human accuracy, RT and confidence and does so better than all current alternatives. To test RTNet’s ability to predict human behaviour on novel images, we collected accuracy, RT and confidence data from 60 human participants performing a digit discrimination task. We found that the accuracy, RT and confidence produced by RTNet for individual novel images correlated with the same quantities produced by human participants. Critically, human participants who were more similar to the average human performance were also found to be closer to RTNet’s predictions, suggesting that RTNet successfully captured average human behaviour. Overall, RTNet is a promising model of human RTs that exhibits the critical signatures of perceptual decision-making.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Model architectures.
Fig. 2: Experiment task.
Fig. 3: Decision stochasticity in humans and all networks.
Fig. 4: Behavioural effects shown by human participants and the models.
Fig. 5: Image-by-image correlation between human data and each model across all experimental conditions for individual participants.
Fig. 6: Image-by-image correlation between human data and each network in each experimental condition.
Fig. 7: Humans who are more similar to the group average are also more similar to each model.
Fig. 8: Comparison between individual participants and the models in predicting the group data.

Similar content being viewed by others

Data availability

The behavioural data have been made publicly available at https://osf.io/akwty.

Code availability

All code and trained models are publicly available at https://osf.io/akwty.

References

  1. Ratcliff, R. A theory of memory retrieval. Psychol. Rev. 85, 59–108 (1978).

    Article  Google Scholar 

  2. Ratcliff, R. & McKoon, G. The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput. 20, 873–922 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  3. Ashby, F. G. & Townsend, J. T. Varieties of perceptual independence. Psychol. Rev. 93, 154–179 (1986).

    Article  CAS  PubMed  Google Scholar 

  4. Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (John Wiley, 1966).

  5. Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1, 417–446 (2015).

    Article  PubMed  Google Scholar 

  6. Kriegeskorte, N. & Golan, T. Neural network models and deep learning. Curr. Biol. 29, R231–R236 (2019).

    Article  PubMed  Google Scholar 

  7. Kietzmann, T. C., McClure, P. & Kriegeskorte, N. Deep neural networks in computational neuroscience. Oxf. Res. Encycl. Neurosci. https://doi.org/10.1093/ACREFORE/9780190264086.013.46 (2019).

  8. Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).

    Article  CAS  PubMed  Google Scholar 

  9. Iuzzolino, M. L., Mozer, M. C. & Bengio, S. Improving anytime prediction with parallel cascaded networks and a temporal-difference loss. Adv. Neural Inf. Process. Syst. 33, 27631–27644 (2021).

    Google Scholar 

  10. Spoerer, C. J., Kietzmann, T. C., Mehrer, J., Charest, I. & Kriegeskorte, N. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS Comput. Biol. 16, e1008215 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Zhang, L. et al. SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models. in Advances in Neural Information Processing Systems 32 Vol. 32 (eds Wallach, H. et al.) (Curran Associates, 2019).

  12. Subramanian, A., Sizikova, E., Kumbhar, O., Majaj, N. & Pelli, D. G. Benchmarking dynamic neural-network models of the human speed–accuracy trade off. J. Vis. 22, 4359 (2022).

    Article  Google Scholar 

  13. Huang, G. et al. Multi-scale dense networks for resource efficient image classification. In Proc. 6th International Conference on Learning Representations (ICLR, 2018).

  14. Kalanthroff, E., Davelaar, E. J., Henik, A., Goldfarb, L. & Usher, M. Task conflict and proactive control: a computational theory of the Stroop task. Psychol. Rev. 125, 59–82 (2018).

    Article  PubMed  Google Scholar 

  15. Mewhort, D. J. K., Braun, J. G. & Heathcote, A. Response time distributions and the Stroop task: a test of the Cohen, Dunbar, and McClelland (1990) model. J. Exp. Psychol. Hum. Percept. Perform. 18, 872–882 (1992).

    Article  CAS  PubMed  Google Scholar 

  16. Cohen, J. D., Dunbar, K. & McClelland, J. L. On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97, 332–361 (1990).

    Article  CAS  PubMed  Google Scholar 

  17. Koivisto, M., Railo, H., Revonsuo, A., Vanni, S. & Salminen-Vaparanta, N. Recurrent processing in V1/V2 contributes to categorization of natural scenes. J. Neurosci. 31, 2488–2492 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Tang, H. et al. Recurrent computations for visual pattern completion. Proc. Natl Acad. Sci. USA 115, 8835–8840 (2017).

    Article  Google Scholar 

  19. Lamme, V. A. F. & Roelfsema, P. R. The distinct modes of vision offered by feedforward and recurrent processing. Trends Neurosci. 23, 571–579 (2000).

    Article  CAS  PubMed  Google Scholar 

  20. Kar, K. & DiCarlo, J. J. Fast recurrent processing via ventrolateral prefrontal cortex is needed by the primate ventral stream for robust core visual object recognition. Neuron 109, 164–176.e5 (2021).

    Article  CAS  PubMed  Google Scholar 

  21. Kar, K., Kubilius, J., Schmidt, K., Issa, E. B. & DiCarlo, J. J. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nat. Neurosci. 22, 974–983 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Goetschalckx, L. et al. Computing a human-like reaction time metric from stable recurrent vision models. In Advances in Neural Information Processing Systems (eds Oh, A. et al.) 14338–14365 (Curran Associates, 2023).

  23. Bogacz, R., Brown, E., Moehlis, J., Holmes, P. & Cohen, J. D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700–765 (2006).

    Article  PubMed  Google Scholar 

  24. Heathcote, A. & Matzke, D. Winner takes all! What are race models, and why and how should psychologists use them? Curr. Dir. Psychol. Sci. 31, 383–394 (2022).

    Article  Google Scholar 

  25. Vickers, D. Evidence for an accumulator model of psychophysical discrimination. Ergonomics 13, 37–58 (2007).

    Article  Google Scholar 

  26. Forstmann, B. U., Ratcliff, R. & Wagenmakers, E.-J. Sequential sampling models in cognitive neuroscience: advantages, applications, and extensions. Annu. Rev. Psychol. 67, 641–666 (2016).

    Article  CAS  PubMed  Google Scholar 

  27. Rahnev, D. Confidence in the real world. Trends Cogn. Sci. 24, 590–591 (2020).

    Article  PubMed  Google Scholar 

  28. Yeon, J. & Rahnev, D. The suboptimality of perceptual decision making with multiple alternatives. Nat. Commun. 11, 3857 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  29. Drugowitsch, J., Wyart, V., Devauchelle, A. D. & Koechlin, E. Computational precision of mental inference as critical source of human choice suboptimality. Neuron 92, 1398–1411 (2016).

    Article  CAS  PubMed  Google Scholar 

  30. Li, H. H. & Ma, W. J. Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis. Nat. Commun. 11, 2004 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Churchland, A. K., Kiani, R. & Shadlen, M. N. Decision-making with multiple alternatives. Nat. Neurosci. 11, 693–702 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Deng, L. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Process. Mag. 29, 141–142 (2012).

    Article  Google Scholar 

  33. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS 2012) (eds Pereira, F. et al.) (Curran Associates, 2012); https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html

  34. Geirhos, R. et al. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018) (eds Bengio, S. et al.) (Curran Associates, 2018); https://proceedings.neurips.cc/paper/2018/hash/0937fb5864ed06ffb59ae5f9b5ed67a9-Abstract.html

  35. Geirhos, R. et al. Comparing deep neural networks against humans: object recognition when the signal gets weaker. arXiv https://doi.org/10.48550/arxiv.1706.06969 (2017).

  36. Beck, J. M., Ma, W. J., Pitkow, X., Latham, P. E. & Pouget, A. Not noisy, just wrong: the role of suboptimal inference in behavioral variability. Neuron 74, 30–39 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Renart, A. & Machens, C. K. Variability in neural activity and behavior. Curr. Opin. Neurobiol. 25, 211–220 (2014).

    Article  CAS  PubMed  Google Scholar 

  38. Heitz, R. P. The speed–accuracy trade off: history, physiology, methodology, and behavior. Front. Neurosci. 8, 150 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  39. Heitz, R. P. & Schall, J. D. Neural mechanisms of speed–accuracy trade off. Neuron 76, 616–628 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Ratcliff, R. & Rouder, J. N. Modeling response times for two-choice decisions. Psychol. Sci. 9, 347–356 (1998).

    Article  Google Scholar 

  41. Wagenmakers, E.-J. & Brown, S. On the linear relation between the mean and the standard deviation of a response time distribution. Psychol. Rev. 114, 830–841 (2007).

    Article  PubMed  Google Scholar 

  42. Brown, S. & Heathcote, A. The simplest complete model of choice response time: linear ballistic accumulation. Cogn. Psychol. 57, 153–178 (2008).

    Article  PubMed  Google Scholar 

  43. Forstmann, B. U. et al. Striatum and pre-SMA facilitate decision-making under time pressure. Proc. Natl Acad. Sci. USA 105, 17538–17542 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Luce, R. D. Response Times (Oxford Univ. Press, 1986); https://doi.org/10.1093/acprof:oso/9780195070019.001.0001

  45. Ratcliff, R. A diffusion model account of response time and accuracy in a brightness discrimination task: fitting real data and failing to fit fake but plausible data. Psychon. Bull. Rev. 9, 278–291 (2002).

    Article  PubMed  Google Scholar 

  46. Rahnev, D. Visual metacognition: measures, models, and neural correlates. Am. Psychol. 76, 1445–1453 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  47. Wyart, V. & Koechlin, E. Choice variability and suboptimality in uncertain environments. Curr. Opin. Behav. Sci. 11, 109–115 (2016).

    Article  Google Scholar 

  48. Findling, C. & Wyart, V. Computation noise in human learning and decision-making: origin, impact, function. Curr. Opin. Behav. Sci. 38, 124–132 (2021).

    Article  Google Scholar 

  49. Rafiei, F. & Rahnev, D. Qualitative speed–accuracy tradeoff effects that cannot be explained by the diffusion model under the selective influence assumption. Sci. Rep. 11, 45 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. https://doi.org/10.1146/annurev.neuro.29.051605.113038 (2007).

  51. Yeung, N. & Summerfield, C. Metacognition in human decision-making: confidence and error monitoring. Phil. Trans. R. Soc. B 367, 1310–1321 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  52. Heathcote, A. & Love, J. Linear deterministic accumulator models of simple choice. Front. Psychol. 3, 292 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  53. Ratcliff, R. & Smith, P. L. A comparison of sequential sampling models for two-choice reaction time. Psychol. Rev. 111, 333–367 (2004).

    Article  PubMed  PubMed Central  Google Scholar 

  54. Brown, S. & Heathcote, A. A ballistic model of choice response time. Psychol. Rev. 112, 117–128 (2005).

    Article  PubMed  Google Scholar 

  55. Tillman, G., Van Zandt, T. & Logan, G. D. Sequential sampling models without random between-trial variability: the racing diffusion model of speeded decision making. Psychon. Bull. Rev. 27, 911–936 (2020).

    Article  PubMed  Google Scholar 

  56. Mizuseki, K., Sirota, A., Pastalkova, E. & Buzsáki, G. Theta oscillations provide temporal windows for local circuit computation in the entorhinal–hippocampal loop. Neuron 64, 267–280 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  57. Nayebi, A. et al. Task-driven convolutional recurrent models of the visual system. Adv. Neural Inf. Process. Syst. 31, 5290–5301 (2018).

    Google Scholar 

  58. Issa, E. B., Cadieu, C. F. & Dicarlo, J. J. Neural dynamics at successive stages of the ventral visual stream are consistent with hierarchical error signals. eLife 7, e42870 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  59. van Bergen, R. S. & Kriegeskorte, N. Going in circles is the way forward: the role of recurrence in visual inference. Curr. Opin. Neurobiol. 65, 176–193 (2020).

    Article  CAS  PubMed  Google Scholar 

  60. Kaufman, M. T. & Churchland, A. K. Sensory noise drives bad decisions. Nature 496, 172–173 (2013).

    Article  CAS  PubMed  Google Scholar 

  61. Brunton, B. W., Botvinick, M. M. & Brody, C. D. Rats and humans can optimally accumulate evidence for decision-making. Science 340, 95–98 (2013).

    Article  CAS  PubMed  Google Scholar 

  62. Osborne, L. C., Lisberger, S. G. & Bialek, W. A sensory source for motor variation. Nature 437, 412–416 (2005).

    Article  CAS  PubMed  Google Scholar 

  63. Huk, A. C. & Shadlen, M. N. Neural activity in macaque parietal cortex reflects temporal integration of visual motion signals during perceptual decision making. J. Neurosci. 25, 10420–10436 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Huk, A. C., Katz, L. N. & Yates, J. L. The role of the lateral intraparietal area in (the study of) decision making. Annu. Rev. Neurosci. 40, 349–372 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Bahl, A. & Engert, F. Neural circuits for evidence accumulation and decision making in larval zebrafish. Nat. Neurosci. 23, 94–102 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Hanks, T. D., Kiani, R. & Shadlen, M. N. A neural mechanism of speed–accuracy tradeoff in macaque area LIP. eLife 3, e02260 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  67. Spoerer, C. J., McClure, P. & Kriegeskorte, N. Recurrent convolutional neural networks: a better model of biological object recognition. Front. Psychol. 8, 1551 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  68. Kietzmann, T. C. et al. Recurrence is required to capture the representational dynamics of the human visual system. Proc. Natl Acad. Sci. USA 116, 21854–21863 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  69. Schwarzschild, A. et al. Can you learn an algorithm? Generalizing from easy to hard problems with recurrent networks. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021) (eds Ranzato, M. et al.) (Curran Associates, 2021); https://proceedings.neurips.cc/paper/2021/hash/3501672ebc68a5524629080e3ef60aef-Abstract.html

  70. Zhou, D. et al. Least-to-most prompting enables complex reasoning in large language models. arXiv https://doi.org/10.48550/arxiv.2205.10625 (2022).

  71. Saltelli, A. et al. Sensitivity analysis for neural networks: natural computing. Risk Anal. 159, 179–201 (2009).

    Google Scholar 

  72. Ko, J. H., Kim, D., Na, T., Kung, J. & Mukhopadhyay, S. Adaptive weight compression for memory-efficient neural networks. In Proc. 2017 Design, Automation and Test in Europe 199–204 (IEEE, 2017); https://doi.org/10.23919/DATE.2017.7926982

  73. Koutník, J., Gomez, F. & Schmidhuber, J. Evolving neural networks in compressed weight space. In Proc. 12th Annual Genetic and Evolutionary Computation Conference 619–625 (Association for Computing Machinery, 2010); https://doi.org/10.1145/1830483.1830596

  74. Kung, J., Kim, D. & Mukhopadhyay, S. A power-aware digital feedforward neural network platform with backpropagation driven approximate synapses. In Proc. 2015 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED) 85–90 (IEEE, 2015).

  75. Tsvetkov, C., Malhotra, G., Evans, B. D. & Bowers, J. S. The role of capacity constraints in convolutional neural networks for learning random versus natural data. Neural Netw. 161, 515–524 (2023).

    Article  PubMed  Google Scholar 

  76. Malhotra, G., Leslie, D. S., Ludwig, C. J. H. & Bogacz, R. Overcoming indecision by changing the decision boundary. J. Exp. Psychol. Gen. 146, 776–805 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  77. Drugowitsch, J., Moreno-Bote, R. N., Churchland, A. K., Shadlen, M. N. & Pouget, A. The cost of accumulating evidence in perceptual decision making. J. Neurosci. 32, 3612–3628 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  78. Rahnev, D. & Denison, R. N. Suboptimality in perceptual decision making. Behav. Brain Sci. 41, e223 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  79. Evans, N. J., Bennett, A. J. & Brown, S. D. Optimal or not; depends on the task. Psychon. Bull. Rev. 26, 1027–1034 (2019).

    Article  PubMed  Google Scholar 

  80. Brainard, D. H. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).

    Article  CAS  PubMed  Google Scholar 

  81. Chen, Y. C. A tutorial on kernel density estimation and recent advances. Biostat. Epidemiol. https://doi.org/10.1080/24709360.2017.1396742 (2017).

  82. Waskom, M. L. seaborn: statistical data visualization. J. Open Source Softw. 6, 3021 (2021).

    Article  Google Scholar 

  83. Jospin, L. V., Buntine, W., Boussaid, F., Laga, H. & Bennamoun, M. Hands-on Bayesian neural networks—a tutorial for deep learning users. IEEE Comput. Intell. Mag. 17, 29–48 (2020).

    Article  Google Scholar 

  84. Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (eds Wallach, H. et al.) (Curran Associates, 2019); https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html

  85. Bingham, E. et al. Pyro: deep universal probabilistic programming. J. Mach. Learn. Res. 20, 1–6 (2019).

    Google Scholar 

  86. Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. In Proc. 2nd International Conference on Learning Representations (ICLR, 2013). Preprint at https://arxiv.org/abs/1312.6114 (2022).

  87. Kingma, D. P. & Ba, J. L. Adam: a method for stochastic optimization. In Proc. 3rd International Conference on Learning Representations (ICLR, 2014). Preprint at https://arxiv.org/abs/1412.6980 (2017).

  88. Deng, J. et al. ImageNet: a large-scale hierarchical image database. In Proc. 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2010); https://doi.org/10.1109/CVPR.2009.5206848

  89. Kumbhar, O., Sizikova, E., Majaj, N. & Pelli, D. G. Anytime prediction as a model of human reaction time. Preprint at https://arxiv.org/abs/2011.12859 (2020).

Download references

Acknowledgements

This work was supported by the National Institutes of Health (award no. R01MH119189) and the Office of Naval Research (award no. N00014-20-1-2622), both awarded to D.R. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank S. Varma and P. Verhaeghen for helpful suggestions about the analyses, as well as A. Shin and H. S. Pandi for assistance with data collection.

Author information

Authors and Affiliations

Authors

Contributions

F.R. and M.S. performed the research and analysed the data. F.R. collected the data and wrote the first draft of the paper. M.S. and D.R. edited the paper. All authors designed the research.

Corresponding author

Correspondence to Farshad Rafiei.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Sushrut Thorat and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rafiei, F., Shekhar, M. & Rahnev, D. The neural network RTNet exhibits the signatures of human perceptual decision-making. Nat Hum Behav 8, 1752–1770 (2024). https://doi.org/10.1038/s41562-024-01914-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-024-01914-8

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing