Understanding how people rate their confidence is critical for the characterization of a wide range of perceptual, memory, motor and cognitive processes. To enable the continued exploration of these processes, we created a large database of confidence studies spanning a broad set of paradigms, participant populations and fields of study. The data from each study are structured in a common, easy-to-use format that can be easily imported and analysed using multiple software packages. Each dataset is accompanied by an explanation regarding the nature of the collected data. At the time of publication, the Confidence Database (which is available at https://osf.io/s46pr/) contained 145 datasets with data from more than 8,700 participants and almost 4 million trials. The database will remain open for new submissions indefinitely and is expected to continue to grow. Here we show the usefulness of this large collection of datasets in four different analyses that provide precise estimations of several foundational confidence-related effects.
Subscribe to Journal
Get full journal access for 1 year
only $8.67 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The Confidence Database is available at https://osf.io/s46pr/.
Codes reproducing all analyses in this paper are available at https://osf.io/s46pr/.
Mamassian, P. Visual confidence. Annu. Rev. Vis. Sci. 2, 459–481 (2016).
Weidemann, C. T. & Kahana, M. J. Assessing recognition memory using confidence ratings and response times. R. Soc. Open Sci. 3, 150670 (2016).
Peirce, C. S. & Jastrow, J. On small differences in sensation. Mem. Natl Acad. Sci. 3, 75–83 (1884).
Ratcliff, R., Van Zandt, T. & McKoon, G. Process dissociation, single-process theories, and recognition memory. J. Exp. Psychol. Gen. 124, 352–374 (1995).
Azzopardi, P. & Cowey, A. Is blindsight like normal, near-threshold vision? Proc. Natl Acad. Sci. USA 94, 14190–14194 (1997).
Robey, A. M., Dougherty, M. R. & Buttaccio, D. R. Making retrospective confidence judgments improves learners’ ability to decide what not to study. Psychol. Sci. 28, 1683–1693 (2017).
Wixted, J. T. & Wells, G. L. The relationship between eyewitness confidence and identification accuracy: a new synthesis. Psychol. Sci. Publ. Int. 18, 10–65 (2017).
Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics. (John Wiley & Sons Ltd, 1966).
Mueller, S. T. & Weidemann, C. T. Decision noise: an explanation for observed violations of signal detection theory. Psychon. Bull. Rev. 15, 465–494 (2008).
Balakrishnan, J. D. & Ratcliff, R. Testing models of decision making using confidence ratings in classification. J. Exp. Psychol. Hum. Percept. Perform. 22, 615–633 (1996).
Yi, Y. & Merfeld, D. M. A quantitative confidence signal detection model: 1. Fitting psychometric functions. J. Neurophysiol. 115, 1932–1945 (2016).
David, A. S., Bedford, N., Wiffen, B. & Gilleen, J. Failures of metacognition and lack of insight in neuropsychiatric disorders. Proc. R. Soc. B 367, 1379–1390 (2012).
Hardwicke, T. E. & Ioannidis, J. P. A. Populating the data ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS One 13, e0201856 (2018).
Vines, T. H. et al. The Availability of research data declines rapidly with article age. Curr. Biol. 24, 94–97 (2014).
Wicherts, J. M., Borsboom, D., Kats, J. & Molenaar, D. The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726–728 (2006).
Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).
Nelson, L. D., Simmons, J. & Simonsohn, U. Psychology’s renaissance. Annu. Rev. Psychol. 69, 511–534 (2018).
Weston, S. J., Ritchie, S. J., Rohrer, J. M. & Przybylski, A. K. Recommendations for increasing the transparency of analysis of preexisting data sets. Adv. Methods Pract. Psychol. Sci. 2, 214–227 (2019).
Cumming, G. The new statistics: why and how. Psychol. Sci. 25, 7–29 (2014).
Funder, D. C. & Ozer, D. J. Evaluating effect size in psychological research: sense and nonsense. Adv. Methods Pract. Psychol. Sci. 2, 156–168 (2019).
Allen, M., Poggiali, D., Whitaker, K., Marshall, T. R. & Kievit, R. Raincloud plots: a multi-platform tool for robust data visualization. Wellcome Open Res. 4, 63 (2019).
Pleskac, T. J. & Busemeyer, J. R. Two-stage dynamic signal detection: a theory of choice, decision time, and confidence. Psychol. Rev. 117, 864–901 (2010).
Moran, R., Teodorescu, A. R. & Usher, M. Post choice information integration as a causal determinant of confidence: novel data and a computational account. Cogn. Psychol. 78, 99–147 (2015).
Nikolov, S., Rahnev, D. & Lau, H. Probabilistic model of onset detection explains paradoxes in human time perception. Front. Psychol. 1, 37 (2010).
Urai, A. E., Braun, A. & Donner, T. H. Pupil-linked arousal is driven by decision uncertainty and alters serial choice bias. Nat. Commun. 8, 14637 (2017).
Rahnev, D., Koizumi, A., McCurdy, L. Y., D’Esposito, M. & Lau, H. Confidence leak in perceptual decision making. Psychol. Sci. 26, 1664–1680 (2015).
Laming, D. Autocorrelation of choice-reaction times. Acta Psychol. 43, 381–412 (1979).
Fischer, J. & Whitney, D. Serial dependence in visual perception. Nat. Neurosci. 17, 738–743 (2014).
Manassi, M., Liberman, A., Kosovicheva, A., Zhang, K. & Whitney, D. Serial dependence in position occurs at the time of perception. Psychon. Bull. Rev. 25, 2245–2253 (2018).
Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest package: tests in linear mixed effects models. J. Stat. Softw. 82, 1–26 (2017).
Metcalfe, J. & Shimamura, A. P. Metacognition: Knowing About Knowing. (MIT Press, 1994).
Fleming, S. M. & Lau, H. How to measure metacognition. Front. Hum. Neurosci. 8, 443 (2014).
Rosenthal, C. R. R., Andrews, S. K. K., Antoniades, C. A. A., Kennard, C. & Soto, D. Learning and recognition of a non-conscious sequence of events in human primary visual cortex. Curr. Biol. 26, 834–841 (2016).
Scott, R. B., Dienes, Z., Barrett, A. B., Bor, D. & Seth, A. K. Blind insight: metacognitive discrimination despite chance task performance. Psychol. Sci. 25, 2199–2208 (2014).
Faivre, N., Filevich, E., Solovey, G., Kühn, S. & Blanke, O. Behavioral, modeling, and electrophysiological evidence for supramodality in human metacognition. J. Neurosci. 38, 263–277 (2018).
Morales, J., Lau, H. & Fleming, S. M. Domain-general and domain-specific patterns of activity supporting metacognition in human prefrontal cortex. J. Neurosci. 38, 3534–3546 (2018).
Houtkoop, B. L. et al. Data sharing in psychology: a survey on barriers and preconditions. Adv. Methods Pract. Psychol. Sci. 1, 70–85 (2018).
King, G. An introduction to the dataverse network as an infrastructure for data sharing. Sociol. Methods Res. 36, 173–199 (2007).
Alter, G. & Gonzalez, R. Responsible practices for data sharing. Am. Psychol. 73, 146–156 (2018).
Martone, M. E., Garcia-Castro, A. & VandenBos, G. R. Data sharing in psychology. Am. Psychol. 73, 111–125 (2018).
Mello, M. M. et al. Preparing for responsible sharing of clinical trial data. N. Engl. J. Med. 369, 1651–1658 (2013).
Tenopir, C. et al. Data sharing by scientists: practices and perceptions. PLoS One 6, e21101 (2011).
Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K. & McGillivray, B. The citation advantage of linking publications to research data. Preprint at arXiv https://arxiv.org/abs/1907.02565 (2019).
Milham, M. P. et al. Assessment of the impact of shared brain imaging data on the scientific literature. Nat. Commun. 9, 2818 (2018).
The organization of the Confidence Database was supported by the National Institute of Mental Health under award number R56MH119189 to D.R. The funder had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
The authors declare no competing interests.
Peer review information Primary Handling Editor: Marike Schiffer.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Rahnev, D., Desender, K., Lee, A.L.F. et al. The Confidence Database. Nat Hum Behav (2020). https://doi.org/10.1038/s41562-019-0813-1