Abstract
Understanding how people rate their confidence is critical for the characterization of a wide range of perceptual, memory, motor and cognitive processes. To enable the continued exploration of these processes, we created a large database of confidence studies spanning a broad set of paradigms, participant populations and fields of study. The data from each study are structured in a common, easy-to-use format that can be easily imported and analysed using multiple software packages. Each dataset is accompanied by an explanation regarding the nature of the collected data. At the time of publication, the Confidence Database (which is available at https://osf.io/s46pr/) contained 145 datasets with data from more than 8,700 participants and almost 4 million trials. The database will remain open for new submissions indefinitely and is expected to continue to grow. Here we show the usefulness of this large collection of datasets in four different analyses that provide precise estimations of several foundational confidence-related effects.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The Confidence Database is available at https://osf.io/s46pr/.
Code availability
Codes reproducing all analyses in this paper are available at https://osf.io/s46pr/.
References
Mamassian, P. Visual confidence. Annu. Rev. Vis. Sci. 2, 459–481 (2016).
Weidemann, C. T. & Kahana, M. J. Assessing recognition memory using confidence ratings and response times. R. Soc. Open Sci. 3, 150670 (2016).
Peirce, C. S. & Jastrow, J. On small differences in sensation. Mem. Natl Acad. Sci. 3, 75–83 (1884).
Ratcliff, R., Van Zandt, T. & McKoon, G. Process dissociation, single-process theories, and recognition memory. J. Exp. Psychol. Gen. 124, 352–374 (1995).
Azzopardi, P. & Cowey, A. Is blindsight like normal, near-threshold vision? Proc. Natl Acad. Sci. USA 94, 14190–14194 (1997).
Robey, A. M., Dougherty, M. R. & Buttaccio, D. R. Making retrospective confidence judgments improves learners’ ability to decide what not to study. Psychol. Sci. 28, 1683–1693 (2017).
Wixted, J. T. & Wells, G. L. The relationship between eyewitness confidence and identification accuracy: a new synthesis. Psychol. Sci. Publ. Int. 18, 10–65 (2017).
Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics. (John Wiley & Sons Ltd, 1966).
Mueller, S. T. & Weidemann, C. T. Decision noise: an explanation for observed violations of signal detection theory. Psychon. Bull. Rev. 15, 465–494 (2008).
Balakrishnan, J. D. & Ratcliff, R. Testing models of decision making using confidence ratings in classification. J. Exp. Psychol. Hum. Percept. Perform. 22, 615–633 (1996).
Yi, Y. & Merfeld, D. M. A quantitative confidence signal detection model: 1. Fitting psychometric functions. J. Neurophysiol. 115, 1932–1945 (2016).
David, A. S., Bedford, N., Wiffen, B. & Gilleen, J. Failures of metacognition and lack of insight in neuropsychiatric disorders. Proc. R. Soc. B 367, 1379–1390 (2012).
Hardwicke, T. E. & Ioannidis, J. P. A. Populating the data ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS One 13, e0201856 (2018).
Vines, T. H. et al. The Availability of research data declines rapidly with article age. Curr. Biol. 24, 94–97 (2014).
Wicherts, J. M., Borsboom, D., Kats, J. & Molenaar, D. The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726–728 (2006).
Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).
Nelson, L. D., Simmons, J. & Simonsohn, U. Psychology’s renaissance. Annu. Rev. Psychol. 69, 511–534 (2018).
Weston, S. J., Ritchie, S. J., Rohrer, J. M. & Przybylski, A. K. Recommendations for increasing the transparency of analysis of preexisting data sets. Adv. Methods Pract. Psychol. Sci. 2, 214–227 (2019).
Cumming, G. The new statistics: why and how. Psychol. Sci. 25, 7–29 (2014).
Funder, D. C. & Ozer, D. J. Evaluating effect size in psychological research: sense and nonsense. Adv. Methods Pract. Psychol. Sci. 2, 156–168 (2019).
Allen, M., Poggiali, D., Whitaker, K., Marshall, T. R. & Kievit, R. Raincloud plots: a multi-platform tool for robust data visualization. Wellcome Open Res. 4, 63 (2019).
Pleskac, T. J. & Busemeyer, J. R. Two-stage dynamic signal detection: a theory of choice, decision time, and confidence. Psychol. Rev. 117, 864–901 (2010).
Moran, R., Teodorescu, A. R. & Usher, M. Post choice information integration as a causal determinant of confidence: novel data and a computational account. Cogn. Psychol. 78, 99–147 (2015).
Nikolov, S., Rahnev, D. & Lau, H. Probabilistic model of onset detection explains paradoxes in human time perception. Front. Psychol. 1, 37 (2010).
Urai, A. E., Braun, A. & Donner, T. H. Pupil-linked arousal is driven by decision uncertainty and alters serial choice bias. Nat. Commun. 8, 14637 (2017).
Rahnev, D., Koizumi, A., McCurdy, L. Y., D’Esposito, M. & Lau, H. Confidence leak in perceptual decision making. Psychol. Sci. 26, 1664–1680 (2015).
Laming, D. Autocorrelation of choice-reaction times. Acta Psychol. 43, 381–412 (1979).
Fischer, J. & Whitney, D. Serial dependence in visual perception. Nat. Neurosci. 17, 738–743 (2014).
Manassi, M., Liberman, A., Kosovicheva, A., Zhang, K. & Whitney, D. Serial dependence in position occurs at the time of perception. Psychon. Bull. Rev. 25, 2245–2253 (2018).
Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest package: tests in linear mixed effects models. J. Stat. Softw. 82, 1–26 (2017).
Metcalfe, J. & Shimamura, A. P. Metacognition: Knowing About Knowing. (MIT Press, 1994).
Fleming, S. M. & Lau, H. How to measure metacognition. Front. Hum. Neurosci. 8, 443 (2014).
Rosenthal, C. R. R., Andrews, S. K. K., Antoniades, C. A. A., Kennard, C. & Soto, D. Learning and recognition of a non-conscious sequence of events in human primary visual cortex. Curr. Biol. 26, 834–841 (2016).
Scott, R. B., Dienes, Z., Barrett, A. B., Bor, D. & Seth, A. K. Blind insight: metacognitive discrimination despite chance task performance. Psychol. Sci. 25, 2199–2208 (2014).
Faivre, N., Filevich, E., Solovey, G., Kühn, S. & Blanke, O. Behavioral, modeling, and electrophysiological evidence for supramodality in human metacognition. J. Neurosci. 38, 263–277 (2018).
Morales, J., Lau, H. & Fleming, S. M. Domain-general and domain-specific patterns of activity supporting metacognition in human prefrontal cortex. J. Neurosci. 38, 3534–3546 (2018).
Houtkoop, B. L. et al. Data sharing in psychology: a survey on barriers and preconditions. Adv. Methods Pract. Psychol. Sci. 1, 70–85 (2018).
King, G. An introduction to the dataverse network as an infrastructure for data sharing. Sociol. Methods Res. 36, 173–199 (2007).
Alter, G. & Gonzalez, R. Responsible practices for data sharing. Am. Psychol. 73, 146–156 (2018).
Martone, M. E., Garcia-Castro, A. & VandenBos, G. R. Data sharing in psychology. Am. Psychol. 73, 111–125 (2018).
Mello, M. M. et al. Preparing for responsible sharing of clinical trial data. N. Engl. J. Med. 369, 1651–1658 (2013).
Tenopir, C. et al. Data sharing by scientists: practices and perceptions. PLoS One 6, e21101 (2011).
Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K. & McGillivray, B. The citation advantage of linking publications to research data. Preprint at arXiv https://arxiv.org/abs/1907.02565 (2019).
Milham, M. P. et al. Assessment of the impact of shared brain imaging data on the scientific literature. Nat. Commun. 9, 2818 (2018).
Acknowledgements
The organization of the Confidence Database was supported by the National Institute of Mental Health under award number R56MH119189 to D.R. The funder had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
The Confidence Database was conceived and organized by D.R., who also drafted the paper. K. Desender, A.L.F.L. and D.R. performed the analyses. D.R., K. Davranche, A.L.F.L., W.T.A., D.A.-L., B.A., P.A., L.Y.A., F.B., J.W.B., I.B., D.P.B., T.F.B., J.C.-T., A.C., T.K.C., K.S.D., R.N.D., T.C.D., K.S.D., Y.A.D., N.F., K.F., E.F., T.G., R.M.G., V.d.G., S.G., N.H., M.H., T.-Y.H., X.H., I.I., M.J., J.K., M. Koculak, M. Konishi, C.K., P.D.K., S.C.K., M.L., K.M.L., C.M.L., L.L., B.M., A. Martin, S.M., J.M., A. Mazancieux, D.M.M., D.O., E.R.P., B.P., M.P., C.P., M.G.P., G.P., F.P., M. Rausch, S.R., G.R., M. Rouault, J. Sackur, S. Sadeghi, J. Samaha, T.X.F.S., M. Shekhar, M.T.S., M. Siedlecka, Z.S., C.S., D.S., S. Sun, J.J.A.v.B., S.W., C.T.W., G.W., M.W., X.X., Q.Y., J.Y., F.Z. and A.Z. contributed to the database. All of the contributors at the time of publication are listed as authors in alphabetical order except for the first three authors. All of the authors edited and approved the final version of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information Primary Handling Editor: Marike Schiffer.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Rahnev, D., Desender, K., Lee, A.L.F. et al. The Confidence Database. Nat Hum Behav 4, 317–325 (2020). https://doi.org/10.1038/s41562-019-0813-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-019-0813-1
