The Confidence Database

Abstract

Understanding how people rate their confidence is critical for the characterization of a wide range of perceptual, memory, motor and cognitive processes. To enable the continued exploration of these processes, we created a large database of confidence studies spanning a broad set of paradigms, participant populations and fields of study. The data from each study are structured in a common, easy-to-use format that can be easily imported and analysed using multiple software packages. Each dataset is accompanied by an explanation regarding the nature of the collected data. At the time of publication, the Confidence Database (which is available at https://osf.io/s46pr/) contained 145 datasets with data from more than 8,700 participants and almost 4 million trials. The database will remain open for new submissions indefinitely and is expected to continue to grow. Here we show the usefulness of this large collection of datasets in four different analyses that provide precise estimations of several foundational confidence-related effects.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Datasets in the Confidence Database at the time of publication.
Fig. 2: Correlating confidence with choice and confidence RT.
Fig. 3: Serial dependence in confidence RT.
Fig. 4: The prevalence of estimates of negative metacognitive sensitivity.
Fig. 5: The use of confidence scales in perception and memory studies.

Data availability

The Confidence Database is available at https://osf.io/s46pr/.

Code availability

Codes reproducing all analyses in this paper are available at https://osf.io/s46pr/.

References

  1. 1.

    Mamassian, P. Visual confidence. Annu. Rev. Vis. Sci. 2, 459–481 (2016).

  2. 2.

    Weidemann, C. T. & Kahana, M. J. Assessing recognition memory using confidence ratings and response times. R. Soc. Open Sci. 3, 150670 (2016).

  3. 3.

    Peirce, C. S. & Jastrow, J. On small differences in sensation. Mem. Natl Acad. Sci. 3, 75–83 (1884).

  4. 4.

    Ratcliff, R., Van Zandt, T. & McKoon, G. Process dissociation, single-process theories, and recognition memory. J. Exp. Psychol. Gen. 124, 352–374 (1995).

  5. 5.

    Azzopardi, P. & Cowey, A. Is blindsight like normal, near-threshold vision? Proc. Natl Acad. Sci. USA 94, 14190–14194 (1997).

  6. 6.

    Robey, A. M., Dougherty, M. R. & Buttaccio, D. R. Making retrospective confidence judgments improves learners’ ability to decide what not to study. Psychol. Sci. 28, 1683–1693 (2017).

  7. 7.

    Wixted, J. T. & Wells, G. L. The relationship between eyewitness confidence and identification accuracy: a new synthesis. Psychol. Sci. Publ. Int. 18, 10–65 (2017).

  8. 8.

    Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics. (John Wiley & Sons Ltd, 1966).

  9. 9.

    Mueller, S. T. & Weidemann, C. T. Decision noise: an explanation for observed violations of signal detection theory. Psychon. Bull. Rev. 15, 465–494 (2008).

  10. 10.

    Balakrishnan, J. D. & Ratcliff, R. Testing models of decision making using confidence ratings in classification. J. Exp. Psychol. Hum. Percept. Perform. 22, 615–633 (1996).

  11. 11.

    Yi, Y. & Merfeld, D. M. A quantitative confidence signal detection model: 1. Fitting psychometric functions. J. Neurophysiol. 115, 1932–1945 (2016).

  12. 12.

    David, A. S., Bedford, N., Wiffen, B. & Gilleen, J. Failures of metacognition and lack of insight in neuropsychiatric disorders. Proc. R. Soc. B 367, 1379–1390 (2012).

  13. 13.

    Hardwicke, T. E. & Ioannidis, J. P. A. Populating the data ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS One 13, e0201856 (2018).

  14. 14.

    Vines, T. H. et al. The Availability of research data declines rapidly with article age. Curr. Biol. 24, 94–97 (2014).

  15. 15.

    Wicherts, J. M., Borsboom, D., Kats, J. & Molenaar, D. The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726–728 (2006).

  16. 16.

    Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).

  17. 17.

    Nelson, L. D., Simmons, J. & Simonsohn, U. Psychology’s renaissance. Annu. Rev. Psychol. 69, 511–534 (2018).

  18. 18.

    Weston, S. J., Ritchie, S. J., Rohrer, J. M. & Przybylski, A. K. Recommendations for increasing the transparency of analysis of preexisting data sets. Adv. Methods Pract. Psychol. Sci. 2, 214–227 (2019).

  19. 19.

    Cumming, G. The new statistics: why and how. Psychol. Sci. 25, 7–29 (2014).

  20. 20.

    Funder, D. C. & Ozer, D. J. Evaluating effect size in psychological research: sense and nonsense. Adv. Methods Pract. Psychol. Sci. 2, 156–168 (2019).

  21. 21.

    Allen, M., Poggiali, D., Whitaker, K., Marshall, T. R. & Kievit, R. Raincloud plots: a multi-platform tool for robust data visualization. Wellcome Open Res. 4, 63 (2019).

  22. 22.

    Pleskac, T. J. & Busemeyer, J. R. Two-stage dynamic signal detection: a theory of choice, decision time, and confidence. Psychol. Rev. 117, 864–901 (2010).

  23. 23.

    Moran, R., Teodorescu, A. R. & Usher, M. Post choice information integration as a causal determinant of confidence: novel data and a computational account. Cogn. Psychol. 78, 99–147 (2015).

  24. 24.

    Nikolov, S., Rahnev, D. & Lau, H. Probabilistic model of onset detection explains paradoxes in human time perception. Front. Psychol. 1, 37 (2010).

  25. 25.

    Urai, A. E., Braun, A. & Donner, T. H. Pupil-linked arousal is driven by decision uncertainty and alters serial choice bias. Nat. Commun. 8, 14637 (2017).

  26. 26.

    Rahnev, D., Koizumi, A., McCurdy, L. Y., D’Esposito, M. & Lau, H. Confidence leak in perceptual decision making. Psychol. Sci. 26, 1664–1680 (2015).

  27. 27.

    Laming, D. Autocorrelation of choice-reaction times. Acta Psychol. 43, 381–412 (1979).

  28. 28.

    Fischer, J. & Whitney, D. Serial dependence in visual perception. Nat. Neurosci. 17, 738–743 (2014).

  29. 29.

    Manassi, M., Liberman, A., Kosovicheva, A., Zhang, K. & Whitney, D. Serial dependence in position occurs at the time of perception. Psychon. Bull. Rev. 25, 2245–2253 (2018).

  30. 30.

    Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest package: tests in linear mixed effects models. J. Stat. Softw. 82, 1–26 (2017).

  31. 31.

    Metcalfe, J. & Shimamura, A. P. Metacognition: Knowing About Knowing. (MIT Press, 1994).

  32. 32.

    Fleming, S. M. & Lau, H. How to measure metacognition. Front. Hum. Neurosci. 8, 443 (2014).

  33. 33.

    Rosenthal, C. R. R., Andrews, S. K. K., Antoniades, C. A. A., Kennard, C. & Soto, D. Learning and recognition of a non-conscious sequence of events in human primary visual cortex. Curr. Biol. 26, 834–841 (2016).

  34. 34.

    Scott, R. B., Dienes, Z., Barrett, A. B., Bor, D. & Seth, A. K. Blind insight: metacognitive discrimination despite chance task performance. Psychol. Sci. 25, 2199–2208 (2014).

  35. 35.

    Faivre, N., Filevich, E., Solovey, G., Kühn, S. & Blanke, O. Behavioral, modeling, and electrophysiological evidence for supramodality in human metacognition. J. Neurosci. 38, 263–277 (2018).

  36. 36.

    Morales, J., Lau, H. & Fleming, S. M. Domain-general and domain-specific patterns of activity supporting metacognition in human prefrontal cortex. J. Neurosci. 38, 3534–3546 (2018).

  37. 37.

    Houtkoop, B. L. et al. Data sharing in psychology: a survey on barriers and preconditions. Adv. Methods Pract. Psychol. Sci. 1, 70–85 (2018).

  38. 38.

    King, G. An introduction to the dataverse network as an infrastructure for data sharing. Sociol. Methods Res. 36, 173–199 (2007).

  39. 39.

    Alter, G. & Gonzalez, R. Responsible practices for data sharing. Am. Psychol. 73, 146–156 (2018).

  40. 40.

    Martone, M. E., Garcia-Castro, A. & VandenBos, G. R. Data sharing in psychology. Am. Psychol. 73, 111–125 (2018).

  41. 41.

    Mello, M. M. et al. Preparing for responsible sharing of clinical trial data. N. Engl. J. Med. 369, 1651–1658 (2013).

  42. 42.

    Tenopir, C. et al. Data sharing by scientists: practices and perceptions. PLoS One 6, e21101 (2011).

  43. 43.

    Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K. & McGillivray, B. The citation advantage of linking publications to research data. Preprint at arXiv https://arxiv.org/abs/1907.02565 (2019).

  44. 44.

    Milham, M. P. et al. Assessment of the impact of shared brain imaging data on the scientific literature. Nat. Commun. 9, 2818 (2018).

Download references

Acknowledgements

The organization of the Confidence Database was supported by the National Institute of Mental Health under award number R56MH119189 to D.R. The funder had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

The Confidence Database was conceived and organized by D.R., who also drafted the paper. K. Desender, A.L.F.L. and D.R. performed the analyses. D.R., K. Davranche, A.L.F.L., W.T.A., D.A.-L., B.A., P.A., L.Y.A., F.B., J.W.B., I.B., D.P.B., T.F.B., J.C.-T., A.C., T.K.C., K.S.D., R.N.D., T.C.D., K.S.D., Y.A.D., N.F., K.F., E.F., T.G., R.M.G., V.d.G., S.G., N.H., M.H., T.-Y.H., X.H., I.I., M.J., J.K., M. Koculak, M. Konishi, C.K., P.D.K., S.C.K., M.L., K.M.L., C.M.L., L.L., B.M., A. Martin, S.M., J.M., A. Mazancieux, D.M.M., D.O., E.R.P., B.P., M.P., C.P., M.G.P., G.P., F.P., M. Rausch, S.R., G.R., M. Rouault, J. Sackur, S. Sadeghi, J. Samaha, T.X.F.S., M. Shekhar, M.T.S., M. Siedlecka, Z.S., C.S., D.S., S. Sun, J.J.A.v.B., S.W., C.T.W., G.W., M.W., X.X., Q.Y., J.Y., F.Z. and A.Z. contributed to the database. All of the contributors at the time of publication are listed as authors in alphabetical order except for the first three authors. All of the authors edited and approved the final version of the manuscript.

Correspondence to Dobromir Rahnev.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rahnev, D., Desender, K., Lee, A.L.F. et al. The Confidence Database. Nat Hum Behav (2020). https://doi.org/10.1038/s41562-019-0813-1

Download citation