Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The Confidence Database

Abstract

Understanding how people rate their confidence is critical for the characterization of a wide range of perceptual, memory, motor and cognitive processes. To enable the continued exploration of these processes, we created a large database of confidence studies spanning a broad set of paradigms, participant populations and fields of study. The data from each study are structured in a common, easy-to-use format that can be easily imported and analysed using multiple software packages. Each dataset is accompanied by an explanation regarding the nature of the collected data. At the time of publication, the Confidence Database (which is available at https://osf.io/s46pr/) contained 145 datasets with data from more than 8,700 participants and almost 4 million trials. The database will remain open for new submissions indefinitely and is expected to continue to grow. Here we show the usefulness of this large collection of datasets in four different analyses that provide precise estimations of several foundational confidence-related effects.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Datasets in the Confidence Database at the time of publication.
Fig. 2: Correlating confidence with choice and confidence RT.
Fig. 3: Serial dependence in confidence RT.
Fig. 4: The prevalence of estimates of negative metacognitive sensitivity.
Fig. 5: The use of confidence scales in perception and memory studies.

Data availability

The Confidence Database is available at https://osf.io/s46pr/.

Code availability

Codes reproducing all analyses in this paper are available at https://osf.io/s46pr/.

References

  1. 1.

    Mamassian, P. Visual confidence. Annu. Rev. Vis. Sci. 2, 459–481 (2016).

  2. 2.

    Weidemann, C. T. & Kahana, M. J. Assessing recognition memory using confidence ratings and response times. R. Soc. Open Sci. 3, 150670 (2016).

    Article  Google Scholar 

  3. 3.

    Peirce, C. S. & Jastrow, J. On small differences in sensation. Mem. Natl Acad. Sci. 3, 75–83 (1884).

    Google Scholar 

  4. 4.

    Ratcliff, R., Van Zandt, T. & McKoon, G. Process dissociation, single-process theories, and recognition memory. J. Exp. Psychol. Gen. 124, 352–374 (1995).

    CAS  Article  Google Scholar 

  5. 5.

    Azzopardi, P. & Cowey, A. Is blindsight like normal, near-threshold vision? Proc. Natl Acad. Sci. USA 94, 14190–14194 (1997).

    CAS  Article  Google Scholar 

  6. 6.

    Robey, A. M., Dougherty, M. R. & Buttaccio, D. R. Making retrospective confidence judgments improves learners’ ability to decide what not to study. Psychol. Sci. 28, 1683–1693 (2017).

    Article  Google Scholar 

  7. 7.

    Wixted, J. T. & Wells, G. L. The relationship between eyewitness confidence and identification accuracy: a new synthesis. Psychol. Sci. Publ. Int. 18, 10–65 (2017).

    Article  Google Scholar 

  8. 8.

    Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics. (John Wiley & Sons Ltd, 1966).

  9. 9.

    Mueller, S. T. & Weidemann, C. T. Decision noise: an explanation for observed violations of signal detection theory. Psychon. Bull. Rev. 15, 465–494 (2008).

    Article  Google Scholar 

  10. 10.

    Balakrishnan, J. D. & Ratcliff, R. Testing models of decision making using confidence ratings in classification. J. Exp. Psychol. Hum. Percept. Perform. 22, 615–633 (1996).

    CAS  Article  Google Scholar 

  11. 11.

    Yi, Y. & Merfeld, D. M. A quantitative confidence signal detection model: 1. Fitting psychometric functions. J. Neurophysiol. 115, 1932–1945 (2016).

  12. 12.

    David, A. S., Bedford, N., Wiffen, B. & Gilleen, J. Failures of metacognition and lack of insight in neuropsychiatric disorders. Proc. R. Soc. B 367, 1379–1390 (2012).

    Google Scholar 

  13. 13.

    Hardwicke, T. E. & Ioannidis, J. P. A. Populating the data ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS One 13, e0201856 (2018).

    Article  Google Scholar 

  14. 14.

    Vines, T. H. et al. The Availability of research data declines rapidly with article age. Curr. Biol. 24, 94–97 (2014).

    CAS  Article  Google Scholar 

  15. 15.

    Wicherts, J. M., Borsboom, D., Kats, J. & Molenaar, D. The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726–728 (2006).

    Article  Google Scholar 

  16. 16.

    Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).

    Article  Google Scholar 

  17. 17.

    Nelson, L. D., Simmons, J. & Simonsohn, U. Psychology’s renaissance. Annu. Rev. Psychol. 69, 511–534 (2018).

    Article  Google Scholar 

  18. 18.

    Weston, S. J., Ritchie, S. J., Rohrer, J. M. & Przybylski, A. K. Recommendations for increasing the transparency of analysis of preexisting data sets. Adv. Methods Pract. Psychol. Sci. 2, 214–227 (2019).

  19. 19.

    Cumming, G. The new statistics: why and how. Psychol. Sci. 25, 7–29 (2014).

    Article  Google Scholar 

  20. 20.

    Funder, D. C. & Ozer, D. J. Evaluating effect size in psychological research: sense and nonsense. Adv. Methods Pract. Psychol. Sci. 2, 156–168 (2019).

    Article  Google Scholar 

  21. 21.

    Allen, M., Poggiali, D., Whitaker, K., Marshall, T. R. & Kievit, R. Raincloud plots: a multi-platform tool for robust data visualization. Wellcome Open Res. 4, 63 (2019).

    Article  Google Scholar 

  22. 22.

    Pleskac, T. J. & Busemeyer, J. R. Two-stage dynamic signal detection: a theory of choice, decision time, and confidence. Psychol. Rev. 117, 864–901 (2010).

    Article  Google Scholar 

  23. 23.

    Moran, R., Teodorescu, A. R. & Usher, M. Post choice information integration as a causal determinant of confidence: novel data and a computational account. Cogn. Psychol. 78, 99–147 (2015).

    Article  Google Scholar 

  24. 24.

    Nikolov, S., Rahnev, D. & Lau, H. Probabilistic model of onset detection explains paradoxes in human time perception. Front. Psychol. 1, 37 (2010).

    PubMed  PubMed Central  Google Scholar 

  25. 25.

    Urai, A. E., Braun, A. & Donner, T. H. Pupil-linked arousal is driven by decision uncertainty and alters serial choice bias. Nat. Commun. 8, 14637 (2017).

    Article  Google Scholar 

  26. 26.

    Rahnev, D., Koizumi, A., McCurdy, L. Y., D’Esposito, M. & Lau, H. Confidence leak in perceptual decision making. Psychol. Sci. 26, 1664–1680 (2015).

    Article  Google Scholar 

  27. 27.

    Laming, D. Autocorrelation of choice-reaction times. Acta Psychol. 43, 381–412 (1979).

    CAS  Article  Google Scholar 

  28. 28.

    Fischer, J. & Whitney, D. Serial dependence in visual perception. Nat. Neurosci. 17, 738–743 (2014).

    CAS  Article  Google Scholar 

  29. 29.

    Manassi, M., Liberman, A., Kosovicheva, A., Zhang, K. & Whitney, D. Serial dependence in position occurs at the time of perception. Psychon. Bull. Rev. 25, 2245–2253 (2018).

    Article  Google Scholar 

  30. 30.

    Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest package: tests in linear mixed effects models. J. Stat. Softw. 82, 1–26 (2017).

    Article  Google Scholar 

  31. 31.

    Metcalfe, J. & Shimamura, A. P. Metacognition: Knowing About Knowing. (MIT Press, 1994).

  32. 32.

    Fleming, S. M. & Lau, H. How to measure metacognition. Front. Hum. Neurosci. 8, 443 (2014).

    Article  Google Scholar 

  33. 33.

    Rosenthal, C. R. R., Andrews, S. K. K., Antoniades, C. A. A., Kennard, C. & Soto, D. Learning and recognition of a non-conscious sequence of events in human primary visual cortex. Curr. Biol. 26, 834–841 (2016).

    CAS  Article  Google Scholar 

  34. 34.

    Scott, R. B., Dienes, Z., Barrett, A. B., Bor, D. & Seth, A. K. Blind insight: metacognitive discrimination despite chance task performance. Psychol. Sci. 25, 2199–2208 (2014).

    Article  Google Scholar 

  35. 35.

    Faivre, N., Filevich, E., Solovey, G., Kühn, S. & Blanke, O. Behavioral, modeling, and electrophysiological evidence for supramodality in human metacognition. J. Neurosci. 38, 263–277 (2018).

    CAS  Article  Google Scholar 

  36. 36.

    Morales, J., Lau, H. & Fleming, S. M. Domain-general and domain-specific patterns of activity supporting metacognition in human prefrontal cortex. J. Neurosci. 38, 3534–3546 (2018).

    CAS  Article  Google Scholar 

  37. 37.

    Houtkoop, B. L. et al. Data sharing in psychology: a survey on barriers and preconditions. Adv. Methods Pract. Psychol. Sci. 1, 70–85 (2018).

    Article  Google Scholar 

  38. 38.

    King, G. An introduction to the dataverse network as an infrastructure for data sharing. Sociol. Methods Res. 36, 173–199 (2007).

    Article  Google Scholar 

  39. 39.

    Alter, G. & Gonzalez, R. Responsible practices for data sharing. Am. Psychol. 73, 146–156 (2018).

    Article  Google Scholar 

  40. 40.

    Martone, M. E., Garcia-Castro, A. & VandenBos, G. R. Data sharing in psychology. Am. Psychol. 73, 111–125 (2018).

    Article  Google Scholar 

  41. 41.

    Mello, M. M. et al. Preparing for responsible sharing of clinical trial data. N. Engl. J. Med. 369, 1651–1658 (2013).

    CAS  Article  Google Scholar 

  42. 42.

    Tenopir, C. et al. Data sharing by scientists: practices and perceptions. PLoS One 6, e21101 (2011).

    CAS  Article  Google Scholar 

  43. 43.

    Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K. & McGillivray, B. The citation advantage of linking publications to research data. Preprint at arXiv https://arxiv.org/abs/1907.02565 (2019).

  44. 44.

    Milham, M. P. et al. Assessment of the impact of shared brain imaging data on the scientific literature. Nat. Commun. 9, 2818 (2018).

    Article  Google Scholar 

Download references

Acknowledgements

The organization of the Confidence Database was supported by the National Institute of Mental Health under award number R56MH119189 to D.R. The funder had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

Authors

Contributions

The Confidence Database was conceived and organized by D.R., who also drafted the paper. K. Desender, A.L.F.L. and D.R. performed the analyses. D.R., K. Davranche, A.L.F.L., W.T.A., D.A.-L., B.A., P.A., L.Y.A., F.B., J.W.B., I.B., D.P.B., T.F.B., J.C.-T., A.C., T.K.C., K.S.D., R.N.D., T.C.D., K.S.D., Y.A.D., N.F., K.F., E.F., T.G., R.M.G., V.d.G., S.G., N.H., M.H., T.-Y.H., X.H., I.I., M.J., J.K., M. Koculak, M. Konishi, C.K., P.D.K., S.C.K., M.L., K.M.L., C.M.L., L.L., B.M., A. Martin, S.M., J.M., A. Mazancieux, D.M.M., D.O., E.R.P., B.P., M.P., C.P., M.G.P., G.P., F.P., M. Rausch, S.R., G.R., M. Rouault, J. Sackur, S. Sadeghi, J. Samaha, T.X.F.S., M. Shekhar, M.T.S., M. Siedlecka, Z.S., C.S., D.S., S. Sun, J.J.A.v.B., S.W., C.T.W., G.W., M.W., X.X., Q.Y., J.Y., F.Z. and A.Z. contributed to the database. All of the contributors at the time of publication are listed as authors in alphabetical order except for the first three authors. All of the authors edited and approved the final version of the manuscript.

Corresponding author

Correspondence to Dobromir Rahnev.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rahnev, D., Desender, K., Lee, A.L.F. et al. The Confidence Database. Nat Hum Behav 4, 317–325 (2020). https://doi.org/10.1038/s41562-019-0813-1

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing