Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds

Nature Human Behaviour volume 2, pages 126–132 (2018)Cite this article

Subjects

Abstract

The aggregation of many independent estimates can outperform the most accurate individual judgement1,2,3. This centenarian finding1,2, popularly known as the 'wisdom of crowds'3, has been applied to problems ranging from the diagnosis of cancer4 to financial forecasting5. It is widely believed that social influence undermines collective wisdom by reducing the diversity of opinions within the crowd. Here, we show that if a large crowd is structured in small independent groups, deliberation and social influence within groups improve the crowd’s collective accuracy. We asked a live crowd (N = 5,180) to respond to general-knowledge questions (for example, "What is the height of the Eiffel Tower?"). Participants first answered individually, then deliberated and made consensus decisions in groups of five, and finally provided revised individual estimates. We found that averaging consensus decisions was substantially more accurate than aggregating the initial independent opinions. Remarkably, combining as few as four consensus choices outperformed the wisdom of thousands of individuals.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Aggregating debates and the wisdom of crowds.
Fig. 2: The effect of deliberation on bias and variance.
Fig. 3: The superior wisdom of deliberative crowds.

Similar content being viewed by others

References

  1. Condorcet, M. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix (L’impremerie royale, Paris, 1785).

    Google Scholar 

  2. Galton, F. Vox populi. Nature 7, 450–451 (1907).

    Article  Google Scholar 

  3. Surowiecki, J. The Wisdom of Crowds (Little, Brown, London, 2004).

    Google Scholar 

  4. Kurvers, R. H. et al. Boosting medical diagnostics by pooling independent judgments. Proc. Natl Acad. Sci. USA 113, 8777–8782 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Ray, R. Prediction markets and the financial "wisdom of crowds”. J. Behav. Financ. 7, 2–4 (2006).

    Article  Google Scholar 

  6. Jowett, B. The Republic of Plato (Clarendon Press, Oxford, 1888).

    Google Scholar 

  7. Forsythe, R., Nelson, F., Neumann, G. R. & Wright, J. Anatomy of an experimental political stock market. Am. Econ. Rev. 82, 1142–1161 (1992).

    Google Scholar 

  8. Keller, A. et al. Predicting human olfactory perception from chemical features of odor molecules. Science 355, 820–826 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. MacKay, C. Extraordinary Popular Delusions the Madness of Crowds (Wordsworth Editions Limited, Ware, 1841).

    Google Scholar 

  10. Tversky, A. & Kahneman, D. Judgment under uncertainty: heuristics and Biases. Science 185, 1124–1131 (1974).

    Article  CAS  PubMed  Google Scholar 

  11. Raafat, R. M., Chater, N. & Frith, C. Herding in humans. Trends Cogn. Sci. 13, 420–428 (2009).

    Article  PubMed  Google Scholar 

  12. Chari, V. V. & Kehoe, P. J. Financial crises as herds: overturning the critiques. J. Econ. Theory 119, 128–150 (2004).

    Article  Google Scholar 

  13. Salganik, M. J., Dodds, P. S. & Watts, D. J. Experimental study of inequality and unpredictability in an artificial cultural market. Science 311, 854–856 (2006).

    Article  CAS  PubMed  Google Scholar 

  14. Muchnik, L., Aral, S. & Taylor, S. J. Social influence bias: a randomized experiment. Science 341, 647–651 (2013).

    Article  CAS  PubMed  Google Scholar 

  15. Festinger, L., Riecken, H. W. & Schachter, S. When Prophecy Fails: A Social and Psychological Study of a Modern Group that Predicted the End of the World (Harper-Torchbooks, New York, NY, 1956).

    Book  Google Scholar 

  16. Lorenz, J., Rauhut, H., Schweitzer, F. & Helbing, D. How social influence can undermine the wisdom of crowd effect. Proc. Natl Acad. Sci. USA 108, 9020–9025 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Madirolas, G. & de Polavieja, G. G. Improving collective estimations using resistance to social influence. PLoS Comput. Biol. 11, e1004594 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  18. Mellers, B. et al. Psychological strategies for winning a geopolitical forecasting tournament. Psychol. Sci. 25, 1106–1115 (2014).

    Article  PubMed  Google Scholar 

  19. Gürçay, B., Mellers, B. A. & Baron, J. The power of social influence on estimation accuracy. J. Behav. Decis. Mak. 28, 250–261 (2015).

    Article  Google Scholar 

  20. Bahrami, B. et al. Optimally interacting minds. Science 329, 1081–1085 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Juni, M. Z. & Eckstein, M. P. Flexible human collective wisdom. J. Exp. Psychol. Hum. Percept. Peform. 41, 1588–1611 (2015).

    Article  Google Scholar 

  22. Mercier, H. & Sperber, D. Why do humans reason? Arguments for an argumentative theory. Behav. Brain. Sci. 34, 57–74 (2011).

    Article  PubMed  Google Scholar 

  23. Mercier, H. & Sperber, D. “Two heads are better” stands to reason. Science 336, 979 (2012).

    Article  CAS  PubMed  Google Scholar 

  24. Smith, M. K. et al. Why peer discussion improves student performance on in-class concept questions. Science 323, 122–124 (2009).

    Article  CAS  PubMed  Google Scholar 

  25. Laughlin, P. R., Bonner, B. L. & Miner, A. G. Groups perform better than the best individuals on letters-to-numbers problems. Organ. Behav. Hum. Decis. Process. 88, 605–620 (2002).

    Article  Google Scholar 

  26. Geil, D. M. M. Collaborative reasoning: evidence for collective rationality. Think. Reason. 4, 231–248 (1998).

    Article  Google Scholar 

  27. Leys, C., Ley, C., Klein, O., Bernard, P. & Licata, L. Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median. J. Exp. Soc. Psychol. 49, 764–766 (2013).

    Article  Google Scholar 

  28. Myers, D. G. & Lamm, H. The group polarization phenomenon. Psychol. Bull. 83, 602–627 (1976).

    Article  Google Scholar 

  29. Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc. Natl Acad. Sci. USA 101, 16385–16389 (2004).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Goldstein, D. G., McAfee, R. P. & Suri, S. The wisdom of smaller, smarter crowds. In Proc. Fifteenth ACM Conference on Economics and Computation Ser. 471–488 (ACM, Palo Alto, CA, 2014).

    Book  Google Scholar 

  31. Mannes, A. E., Soll, J. B. & Larrick, R. P. The wisdom of select crowds. J. Pers. Soc. Psychol. 107, 276–299 (2014).

    Article  PubMed  Google Scholar 

  32. Vul, E. & Pashler, H. Measuring the crowd within: probabilistic representations within individuals. Psychol. Sci. 19, 645–647 (2008).

    Article  PubMed  Google Scholar 

  33. Herzog, S. M. & Hertwig, R. The wisdom of many in one mind: improving individual judgments with dialectical bootstrapping. Psychol. Sci. 20, 231–237 (2009).

    Article  PubMed  Google Scholar 

  34. Ariely, D. et al. The effects of averaging subjective probability estimates between and within judges. J. Exp. Psychol. Appl. 6, 130–146 (2000).

    Article  CAS  PubMed  Google Scholar 

  35. Prelec, D., Seung, H. S. & McCoy, J. A solution to the single-question crowd wisdom problem. Nature 541, 532–535 (2017).

    Article  CAS  PubMed  Google Scholar 

  36. Lorenz, J., Rauhut, H. & Kittel, B. Majoritarian democracy undermines truth-finding in deliberative committees. Res. Polit. 2, 1–10 (2015).

    Google Scholar 

  37. Landemore, H. & Page, S. E. Deliberation and disagreement: problem solving, prediction, and positive dissensus. J. Pol. Philos. Econ. 14, 229–254 (2015).

    Article  Google Scholar 

  38. Li, V., Herce Castañón, S., Solomon, J. A., Vandormael, H. & Summerfield, C. Robust averaging protects decisions from noise in neural computations. PLoS Comput. Biol. 13, e1005723 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  39. Asch, S. E. Opinions and social pressure. Sci. Am. 193, 31–35 (1955).

    Article  Google Scholar 

  40. Lyman, F. T. in The Responsive Classroom Discussion: The Inclusion of All Students (ed. Anderson, A. S.) 113 (Univ. Maryland Press, Potomac, MD, 1981).

  41. Dalkey, N. & Helmer, O. An experimental application of the Delphi method to the use of experts. Manag. Sci. 9, 458–467 (1963).

    Article  Google Scholar 

  42. Tetlock, P. Expert Political Judgment: How Good Is It? How Can We Know? (Princeton Univ. Press, Princeton, NJ, 2005).

  43. Sunstein, C. R. Infotopia: How Many Minds Produce Knowledge (Oxford Univ. Press, Oxford, 2006).

  44. Harvey, N. & Fischer, I. Taking advice: accepting help, improving judgment, and sharing responsibility. Organ. Behav. Hum. Decis. Process. 70, 117–133 (1997).

    Article  Google Scholar 

  45. Eisenberger, N. I., Lieberman, M. D. & Williams, K. D. Does rejection hurt? An FMRI study of social exclusion. Science 302, 290–292 (2003).

    Article  CAS  PubMed  Google Scholar 

  46. Mahmoodi, A. et al. Equality bias impairs collective decision-making across cultures. Proc. Natl Acad. Sci. USA 112, 3835–3840 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  47. Galton, F. One vote, one value. Nature 75, 414 (1907).

    Article  Google Scholar 

  48. Mill, J. S. On Liberty (John W. Parker and Son, London, 1859).

    Google Scholar 

  49. Fishkin, J. S. & Luskin, R. C. Experimenting with a democratic ideal: deliberative polling and public opinion. Acta Polit. 40, 284–298 (2005).

    Article  Google Scholar 

  50. Austen-Smith, D. & Banks, J. S. Information aggregation, rationality, and the Condorcet jury theorem. Am. Political Sci. Rev. 90, 34–45 (1996).

    Article  Google Scholar 

  51. Cohen, J. in Deliberative Democracy: Essays on Reason and Politics (eds Bohman, J. & Rehg, W.) Ch. 3 (MIT Press, Boston, MA, 1997).

  52. Lopez-Rosenfeld, M. et al. Neglect in human communication: quantifying the cost of cell-phone interruptions in face to face dialogs. PLoS ONE 10, e0125772 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  53. Niella, T., Stier-Moses, N. & Sigman, M. Nudging cooperation in a crowd experiment. PLoS ONE 11, e0147125 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

J.N. and B.B. were supported by the European Research Council StG (NEUROCODEC, #309865); M.S. was supported by the James McDonnell Foundation 21st Century Science Initiative in Understanding Human Cognition—Scholar Award (Grant #220020334), and by Agencia Nacional de Promoción Científica y Tecnológica (Argentina)—Préstamo BID PICT (Grant #2013-1653). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank M. Sartorio for assistance in data collection.

Author information

Authors and Affiliations

  1. Universidad Torcuato Di Tella, Buenos Aires, Argentina

    Joaquin Navajas, Tamara Niella & Mariano Sigman

  2. Institute of Cognitive Neuroscience, University College London, London, UK

    Joaquin Navajas & Bahador Bahrami

  3. Psychology Department, University of Oregon, Eugene, OR, USA

    Tamara Niella

  4. TED, Buenos Aires, Argentina

    Gerry Garbulsky

  5. Faculty of Psychology and Educational Sciences, Ludwig Maximilian University, Munich, Germany

    Bahador Bahrami

Authors
  1. Joaquin Navajas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tamara Niella
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerry Garbulsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bahador Bahrami
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mariano Sigman
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.N., T.N., G.G. and M.S. designed and conducted the experiments. J.N., B.B. and M.S. developed the analysis approach. J.N. analysed the data. J.N., B.B. and M.S. wrote the paper.

Corresponding author

Correspondence to Joaquin Navajas.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figures 1–4, Supplementary Table 1

Life Sciences Reporting Summary

Supplementary Video

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Navajas, J., Niella, T., Garbulsky, G. et al. Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nat Hum Behav 2, 126–132 (2018). https://doi.org/10.1038/s41562-017-0273-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-017-0273-4

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing