Abstract
Convolutional neural networks show promise as models of biological vision. However, their decision behaviour, including the facts that they are deterministic and use equal numbers of computations for easy and difficult stimuli, differs markedly from human decision-making, thus limiting their applicability as models of human perceptual behaviour. Here we develop a new neural network, RTNet, that generates stochastic decisions and human-like response time (RT) distributions. We further performed comprehensive tests that showed RTNet reproduces all foundational features of human accuracy, RT and confidence and does so better than all current alternatives. To test RTNet’s ability to predict human behaviour on novel images, we collected accuracy, RT and confidence data from 60 human participants performing a digit discrimination task. We found that the accuracy, RT and confidence produced by RTNet for individual novel images correlated with the same quantities produced by human participants. Critically, human participants who were more similar to the average human performance were also found to be closer to RTNet’s predictions, suggesting that RTNet successfully captured average human behaviour. Overall, RTNet is a promising model of human RTs that exhibits the critical signatures of perceptual decision-making.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The behavioural data have been made publicly available at https://osf.io/akwty.
Code availability
All code and trained models are publicly available at https://osf.io/akwty.
References
Ratcliff, R. A theory of memory retrieval. Psychol. Rev. 85, 59–108 (1978).
Ratcliff, R. & McKoon, G. The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput. 20, 873–922 (2008).
Ashby, F. G. & Townsend, J. T. Varieties of perceptual independence. Psychol. Rev. 93, 154–179 (1986).
Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (John Wiley, 1966).
Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1, 417–446 (2015).
Kriegeskorte, N. & Golan, T. Neural network models and deep learning. Curr. Biol. 29, R231–R236 (2019).
Kietzmann, T. C., McClure, P. & Kriegeskorte, N. Deep neural networks in computational neuroscience. Oxf. Res. Encycl. Neurosci. https://doi.org/10.1093/ACREFORE/9780190264086.013.46 (2019).
Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).
Iuzzolino, M. L., Mozer, M. C. & Bengio, S. Improving anytime prediction with parallel cascaded networks and a temporal-difference loss. Adv. Neural Inf. Process. Syst. 33, 27631–27644 (2021).
Spoerer, C. J., Kietzmann, T. C., Mehrer, J., Charest, I. & Kriegeskorte, N. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS Comput. Biol. 16, e1008215 (2020).
Zhang, L. et al. SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models. in Advances in Neural Information Processing Systems 32 Vol. 32 (eds Wallach, H. et al.) (Curran Associates, 2019).
Subramanian, A., Sizikova, E., Kumbhar, O., Majaj, N. & Pelli, D. G. Benchmarking dynamic neural-network models of the human speed–accuracy trade off. J. Vis. 22, 4359 (2022).
Huang, G. et al. Multi-scale dense networks for resource efficient image classification. In Proc. 6th International Conference on Learning Representations (ICLR, 2018).
Kalanthroff, E., Davelaar, E. J., Henik, A., Goldfarb, L. & Usher, M. Task conflict and proactive control: a computational theory of the Stroop task. Psychol. Rev. 125, 59–82 (2018).
Mewhort, D. J. K., Braun, J. G. & Heathcote, A. Response time distributions and the Stroop task: a test of the Cohen, Dunbar, and McClelland (1990) model. J. Exp. Psychol. Hum. Percept. Perform. 18, 872–882 (1992).
Cohen, J. D., Dunbar, K. & McClelland, J. L. On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97, 332–361 (1990).
Koivisto, M., Railo, H., Revonsuo, A., Vanni, S. & Salminen-Vaparanta, N. Recurrent processing in V1/V2 contributes to categorization of natural scenes. J. Neurosci. 31, 2488–2492 (2011).
Tang, H. et al. Recurrent computations for visual pattern completion. Proc. Natl Acad. Sci. USA 115, 8835–8840 (2017).
Lamme, V. A. F. & Roelfsema, P. R. The distinct modes of vision offered by feedforward and recurrent processing. Trends Neurosci. 23, 571–579 (2000).
Kar, K. & DiCarlo, J. J. Fast recurrent processing via ventrolateral prefrontal cortex is needed by the primate ventral stream for robust core visual object recognition. Neuron 109, 164–176.e5 (2021).
Kar, K., Kubilius, J., Schmidt, K., Issa, E. B. & DiCarlo, J. J. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nat. Neurosci. 22, 974–983 (2019).
Goetschalckx, L. et al. Computing a human-like reaction time metric from stable recurrent vision models. In Advances in Neural Information Processing Systems (eds Oh, A. et al.) 14338–14365 (Curran Associates, 2023).
Bogacz, R., Brown, E., Moehlis, J., Holmes, P. & Cohen, J. D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700–765 (2006).
Heathcote, A. & Matzke, D. Winner takes all! What are race models, and why and how should psychologists use them? Curr. Dir. Psychol. Sci. 31, 383–394 (2022).
Vickers, D. Evidence for an accumulator model of psychophysical discrimination. Ergonomics 13, 37–58 (2007).
Forstmann, B. U., Ratcliff, R. & Wagenmakers, E.-J. Sequential sampling models in cognitive neuroscience: advantages, applications, and extensions. Annu. Rev. Psychol. 67, 641–666 (2016).
Rahnev, D. Confidence in the real world. Trends Cogn. Sci. 24, 590–591 (2020).
Yeon, J. & Rahnev, D. The suboptimality of perceptual decision making with multiple alternatives. Nat. Commun. 11, 3857 (2020).
Drugowitsch, J., Wyart, V., Devauchelle, A. D. & Koechlin, E. Computational precision of mental inference as critical source of human choice suboptimality. Neuron 92, 1398–1411 (2016).
Li, H. H. & Ma, W. J. Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis. Nat. Commun. 11, 2004 (2020).
Churchland, A. K., Kiani, R. & Shadlen, M. N. Decision-making with multiple alternatives. Nat. Neurosci. 11, 693–702 (2008).
Deng, L. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Process. Mag. 29, 141–142 (2012).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS 2012) (eds Pereira, F. et al.) (Curran Associates, 2012); https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
Geirhos, R. et al. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018) (eds Bengio, S. et al.) (Curran Associates, 2018); https://proceedings.neurips.cc/paper/2018/hash/0937fb5864ed06ffb59ae5f9b5ed67a9-Abstract.html
Geirhos, R. et al. Comparing deep neural networks against humans: object recognition when the signal gets weaker. arXiv https://doi.org/10.48550/arxiv.1706.06969 (2017).
Beck, J. M., Ma, W. J., Pitkow, X., Latham, P. E. & Pouget, A. Not noisy, just wrong: the role of suboptimal inference in behavioral variability. Neuron 74, 30–39 (2012).
Renart, A. & Machens, C. K. Variability in neural activity and behavior. Curr. Opin. Neurobiol. 25, 211–220 (2014).
Heitz, R. P. The speed–accuracy trade off: history, physiology, methodology, and behavior. Front. Neurosci. 8, 150 (2014).
Heitz, R. P. & Schall, J. D. Neural mechanisms of speed–accuracy trade off. Neuron 76, 616–628 (2012).
Ratcliff, R. & Rouder, J. N. Modeling response times for two-choice decisions. Psychol. Sci. 9, 347–356 (1998).
Wagenmakers, E.-J. & Brown, S. On the linear relation between the mean and the standard deviation of a response time distribution. Psychol. Rev. 114, 830–841 (2007).
Brown, S. & Heathcote, A. The simplest complete model of choice response time: linear ballistic accumulation. Cogn. Psychol. 57, 153–178 (2008).
Forstmann, B. U. et al. Striatum and pre-SMA facilitate decision-making under time pressure. Proc. Natl Acad. Sci. USA 105, 17538–17542 (2008).
Luce, R. D. Response Times (Oxford Univ. Press, 1986); https://doi.org/10.1093/acprof:oso/9780195070019.001.0001
Ratcliff, R. A diffusion model account of response time and accuracy in a brightness discrimination task: fitting real data and failing to fit fake but plausible data. Psychon. Bull. Rev. 9, 278–291 (2002).
Rahnev, D. Visual metacognition: measures, models, and neural correlates. Am. Psychol. 76, 1445–1453 (2021).
Wyart, V. & Koechlin, E. Choice variability and suboptimality in uncertain environments. Curr. Opin. Behav. Sci. 11, 109–115 (2016).
Findling, C. & Wyart, V. Computation noise in human learning and decision-making: origin, impact, function. Curr. Opin. Behav. Sci. 38, 124–132 (2021).
Rafiei, F. & Rahnev, D. Qualitative speed–accuracy tradeoff effects that cannot be explained by the diffusion model under the selective influence assumption. Sci. Rep. 11, 45 (2021).
Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. https://doi.org/10.1146/annurev.neuro.29.051605.113038 (2007).
Yeung, N. & Summerfield, C. Metacognition in human decision-making: confidence and error monitoring. Phil. Trans. R. Soc. B 367, 1310–1321 (2012).
Heathcote, A. & Love, J. Linear deterministic accumulator models of simple choice. Front. Psychol. 3, 292 (2012).
Ratcliff, R. & Smith, P. L. A comparison of sequential sampling models for two-choice reaction time. Psychol. Rev. 111, 333–367 (2004).
Brown, S. & Heathcote, A. A ballistic model of choice response time. Psychol. Rev. 112, 117–128 (2005).
Tillman, G., Van Zandt, T. & Logan, G. D. Sequential sampling models without random between-trial variability: the racing diffusion model of speeded decision making. Psychon. Bull. Rev. 27, 911–936 (2020).
Mizuseki, K., Sirota, A., Pastalkova, E. & Buzsáki, G. Theta oscillations provide temporal windows for local circuit computation in the entorhinal–hippocampal loop. Neuron 64, 267–280 (2009).
Nayebi, A. et al. Task-driven convolutional recurrent models of the visual system. Adv. Neural Inf. Process. Syst. 31, 5290–5301 (2018).
Issa, E. B., Cadieu, C. F. & Dicarlo, J. J. Neural dynamics at successive stages of the ventral visual stream are consistent with hierarchical error signals. eLife 7, e42870 (2018).
van Bergen, R. S. & Kriegeskorte, N. Going in circles is the way forward: the role of recurrence in visual inference. Curr. Opin. Neurobiol. 65, 176–193 (2020).
Kaufman, M. T. & Churchland, A. K. Sensory noise drives bad decisions. Nature 496, 172–173 (2013).
Brunton, B. W., Botvinick, M. M. & Brody, C. D. Rats and humans can optimally accumulate evidence for decision-making. Science 340, 95–98 (2013).
Osborne, L. C., Lisberger, S. G. & Bialek, W. A sensory source for motor variation. Nature 437, 412–416 (2005).
Huk, A. C. & Shadlen, M. N. Neural activity in macaque parietal cortex reflects temporal integration of visual motion signals during perceptual decision making. J. Neurosci. 25, 10420–10436 (2005).
Huk, A. C., Katz, L. N. & Yates, J. L. The role of the lateral intraparietal area in (the study of) decision making. Annu. Rev. Neurosci. 40, 349–372 (2017).
Bahl, A. & Engert, F. Neural circuits for evidence accumulation and decision making in larval zebrafish. Nat. Neurosci. 23, 94–102 (2019).
Hanks, T. D., Kiani, R. & Shadlen, M. N. A neural mechanism of speed–accuracy tradeoff in macaque area LIP. eLife 3, e02260 (2014).
Spoerer, C. J., McClure, P. & Kriegeskorte, N. Recurrent convolutional neural networks: a better model of biological object recognition. Front. Psychol. 8, 1551 (2017).
Kietzmann, T. C. et al. Recurrence is required to capture the representational dynamics of the human visual system. Proc. Natl Acad. Sci. USA 116, 21854–21863 (2019).
Schwarzschild, A. et al. Can you learn an algorithm? Generalizing from easy to hard problems with recurrent networks. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021) (eds Ranzato, M. et al.) (Curran Associates, 2021); https://proceedings.neurips.cc/paper/2021/hash/3501672ebc68a5524629080e3ef60aef-Abstract.html
Zhou, D. et al. Least-to-most prompting enables complex reasoning in large language models. arXiv https://doi.org/10.48550/arxiv.2205.10625 (2022).
Saltelli, A. et al. Sensitivity analysis for neural networks: natural computing. Risk Anal. 159, 179–201 (2009).
Ko, J. H., Kim, D., Na, T., Kung, J. & Mukhopadhyay, S. Adaptive weight compression for memory-efficient neural networks. In Proc. 2017 Design, Automation and Test in Europe 199–204 (IEEE, 2017); https://doi.org/10.23919/DATE.2017.7926982
Koutník, J., Gomez, F. & Schmidhuber, J. Evolving neural networks in compressed weight space. In Proc. 12th Annual Genetic and Evolutionary Computation Conference 619–625 (Association for Computing Machinery, 2010); https://doi.org/10.1145/1830483.1830596
Kung, J., Kim, D. & Mukhopadhyay, S. A power-aware digital feedforward neural network platform with backpropagation driven approximate synapses. In Proc. 2015 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED) 85–90 (IEEE, 2015).
Tsvetkov, C., Malhotra, G., Evans, B. D. & Bowers, J. S. The role of capacity constraints in convolutional neural networks for learning random versus natural data. Neural Netw. 161, 515–524 (2023).
Malhotra, G., Leslie, D. S., Ludwig, C. J. H. & Bogacz, R. Overcoming indecision by changing the decision boundary. J. Exp. Psychol. Gen. 146, 776–805 (2017).
Drugowitsch, J., Moreno-Bote, R. N., Churchland, A. K., Shadlen, M. N. & Pouget, A. The cost of accumulating evidence in perceptual decision making. J. Neurosci. 32, 3612–3628 (2012).
Rahnev, D. & Denison, R. N. Suboptimality in perceptual decision making. Behav. Brain Sci. 41, e223 (2018).
Evans, N. J., Bennett, A. J. & Brown, S. D. Optimal or not; depends on the task. Psychon. Bull. Rev. 26, 1027–1034 (2019).
Brainard, D. H. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).
Chen, Y. C. A tutorial on kernel density estimation and recent advances. Biostat. Epidemiol. https://doi.org/10.1080/24709360.2017.1396742 (2017).
Waskom, M. L. seaborn: statistical data visualization. J. Open Source Softw. 6, 3021 (2021).
Jospin, L. V., Buntine, W., Boussaid, F., Laga, H. & Bennamoun, M. Hands-on Bayesian neural networks—a tutorial for deep learning users. IEEE Comput. Intell. Mag. 17, 29–48 (2020).
Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (eds Wallach, H. et al.) (Curran Associates, 2019); https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
Bingham, E. et al. Pyro: deep universal probabilistic programming. J. Mach. Learn. Res. 20, 1–6 (2019).
Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. In Proc. 2nd International Conference on Learning Representations (ICLR, 2013). Preprint at https://arxiv.org/abs/1312.6114 (2022).
Kingma, D. P. & Ba, J. L. Adam: a method for stochastic optimization. In Proc. 3rd International Conference on Learning Representations (ICLR, 2014). Preprint at https://arxiv.org/abs/1412.6980 (2017).
Deng, J. et al. ImageNet: a large-scale hierarchical image database. In Proc. 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2010); https://doi.org/10.1109/CVPR.2009.5206848
Kumbhar, O., Sizikova, E., Majaj, N. & Pelli, D. G. Anytime prediction as a model of human reaction time. Preprint at https://arxiv.org/abs/2011.12859 (2020).
Acknowledgements
This work was supported by the National Institutes of Health (award no. R01MH119189) and the Office of Naval Research (award no. N00014-20-1-2622), both awarded to D.R. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank S. Varma and P. Verhaeghen for helpful suggestions about the analyses, as well as A. Shin and H. S. Pandi for assistance with data collection.
Author information
Authors and Affiliations
Contributions
F.R. and M.S. performed the research and analysed the data. F.R. collected the data and wrote the first draft of the paper. M.S. and D.R. edited the paper. All authors designed the research.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Human Behaviour thanks Sushrut Thorat and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rafiei, F., Shekhar, M. & Rahnev, D. The neural network RTNet exhibits the signatures of human perceptual decision-making. Nat Hum Behav 8, 1752–1770 (2024). https://doi.org/10.1038/s41562-024-01914-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-024-01914-8