Letter | Published:

Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds

Abstract

The aggregation of many independent estimates can outperform the most accurate individual judgement1,2,3. This centenarian finding1,2, popularly known as the 'wisdom of crowds'3, has been applied to problems ranging from the diagnosis of cancer4 to financial forecasting5. It is widely believed that social influence undermines collective wisdom by reducing the diversity of opinions within the crowd. Here, we show that if a large crowd is structured in small independent groups, deliberation and social influence within groups improve the crowd’s collective accuracy. We asked a live crowd (N = 5,180) to respond to general-knowledge questions (for example, "What is the height of the Eiffel Tower?"). Participants first answered individually, then deliberated and made consensus decisions in groups of five, and finally provided revised individual estimates. We found that averaging consensus decisions was substantially more accurate than aggregating the initial independent opinions. Remarkably, combining as few as four consensus choices outperformed the wisdom of thousands of individuals.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Condorcet, M. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix (L’impremerie royale, Paris, 1785).

  2. 2.

    Galton, F. Vox populi. Nature 7, 450–451 (1907).

  3. 3.

    Surowiecki, J. The Wisdom of Crowds (Little, Brown, London, 2004).

  4. 4.

    Kurvers, R. H. et al. Boosting medical diagnostics by pooling independent judgments. Proc. Natl Acad. Sci. USA 113, 8777–8782 (2016).

  5. 5.

    Ray, R. Prediction markets and the financial "wisdom of crowds”. J. Behav. Financ. 7, 2–4 (2006).

  6. 6.

    Jowett, B. The Republic of Plato (Clarendon Press, Oxford, 1888).

  7. 7.

    Forsythe, R., Nelson, F., Neumann, G. R. & Wright, J. Anatomy of an experimental political stock market. Am. Econ. Rev. 82, 1142–1161 (1992).

  8. 8.

    Keller, A. et al. Predicting human olfactory perception from chemical features of odor molecules. Science 355, 820–826 (2017).

  9. 9.

    MacKay, C. Extraordinary Popular Delusions the Madness of Crowds (Wordsworth Editions Limited, Ware, 1841).

  10. 10.

    Tversky, A. & Kahneman, D. Judgment under uncertainty: heuristics and Biases. Science 185, 1124–1131 (1974).

  11. 11.

    Raafat, R. M., Chater, N. & Frith, C. Herding in humans. Trends Cogn. Sci. 13, 420–428 (2009).

  12. 12.

    Chari, V. V. & Kehoe, P. J. Financial crises as herds: overturning the critiques. J. Econ. Theory 119, 128–150 (2004).

  13. 13.

    Salganik, M. J., Dodds, P. S. & Watts, D. J. Experimental study of inequality and unpredictability in an artificial cultural market. Science 311, 854–856 (2006).

  14. 14.

    Muchnik, L., Aral, S. & Taylor, S. J. Social influence bias: a randomized experiment. Science 341, 647–651 (2013).

  15. 15.

    Festinger, L., Riecken, H. W. & Schachter, S. When Prophecy Fails: A Social and Psychological Study of a Modern Group that Predicted the End of the World (Harper-Torchbooks, New York, NY, 1956).

  16. 16.

    Lorenz, J., Rauhut, H., Schweitzer, F. & Helbing, D. How social influence can undermine the wisdom of crowd effect. Proc. Natl Acad. Sci. USA 108, 9020–9025 (2011).

  17. 17.

    Madirolas, G. & de Polavieja, G. G. Improving collective estimations using resistance to social influence. PLoS Comput. Biol. 11, e1004594 (2015).

  18. 18.

    Mellers, B. et al. Psychological strategies for winning a geopolitical forecasting tournament. Psychol. Sci. 25, 1106–1115 (2014).

  19. 19.

    Gürçay, B., Mellers, B. A. & Baron, J. The power of social influence on estimation accuracy. J. Behav. Decis. Mak. 28, 250–261 (2015).

  20. 20.

    Bahrami, B. et al. Optimally interacting minds. Science 329, 1081–1085 (2010).

  21. 21.

    Juni, M. Z. & Eckstein, M. P. Flexible human collective wisdom. J. Exp. Psychol. Hum. Percept. Peform. 41, 1588–1611 (2015).

  22. 22.

    Mercier, H. & Sperber, D. Why do humans reason? Arguments for an argumentative theory. Behav. Brain. Sci. 34, 57–74 (2011).

  23. 23.

    Mercier, H. & Sperber, D. “Two heads are better” stands to reason. Science 336, 979 (2012).

  24. 24.

    Smith, M. K. et al. Why peer discussion improves student performance on in-class concept questions. Science 323, 122–124 (2009).

  25. 25.

    Laughlin, P. R., Bonner, B. L. & Miner, A. G. Groups perform better than the best individuals on letters-to-numbers problems. Organ. Behav. Hum. Decis. Process. 88, 605–620 (2002).

  26. 26.

    Geil, D. M. M. Collaborative reasoning: evidence for collective rationality. Think. Reason. 4, 231–248 (1998).

  27. 27.

    Leys, C., Ley, C., Klein, O., Bernard, P. & Licata, L. Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median. J. Exp. Soc. Psychol. 49, 764–766 (2013).

  28. 28.

    Myers, D. G. & Lamm, H. The group polarization phenomenon. Psychol. Bull. 83, 602–627 (1976).

  29. 29.

    Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc. Natl Acad. Sci. USA 101, 16385–16389 (2004).

  30. 30.

    Goldstein, D. G., McAfee, R. P. & Suri, S. The wisdom of smaller, smarter crowds. In Proc. Fifteenth ACM Conference on Economics and Computation Ser. 471–488 (ACM, Palo Alto, CA, 2014).

  31. 31.

    Mannes, A. E., Soll, J. B. & Larrick, R. P. The wisdom of select crowds. J. Pers. Soc. Psychol. 107, 276–299 (2014).

  32. 32.

    Vul, E. & Pashler, H. Measuring the crowd within: probabilistic representations within individuals. Psychol. Sci. 19, 645–647 (2008).

  33. 33.

    Herzog, S. M. & Hertwig, R. The wisdom of many in one mind: improving individual judgments with dialectical bootstrapping. Psychol. Sci. 20, 231–237 (2009).

  34. 34.

    Ariely, D. et al. The effects of averaging subjective probability estimates between and within judges. J. Exp. Psychol. Appl. 6, 130–146 (2000).

  35. 35.

    Prelec, D., Seung, H. S. & McCoy, J. A solution to the single-question crowd wisdom problem. Nature 541, 532–535 (2017).

  36. 36.

    Lorenz, J., Rauhut, H. & Kittel, B. Majoritarian democracy undermines truth-finding in deliberative committees. Res. Polit. 2, 1–10 (2015).

  37. 37.

    Landemore, H. & Page, S. E. Deliberation and disagreement: problem solving, prediction, and positive dissensus. J. Pol. Philos. Econ. 14, 229–254 (2015).

  38. 38.

    Li, V., Herce Castañón, S., Solomon, J. A., Vandormael, H. & Summerfield, C. Robust averaging protects decisions from noise in neural computations. PLoS Comput. Biol. 13, e1005723 (2017).

  39. 39.

    Asch, S. E. Opinions and social pressure. Sci. Am. 193, 31–35 (1955).

  40. 40.

    Lyman, F. T. in The Responsive Classroom Discussion: The Inclusion of All Students (ed. Anderson, A. S.) 113 (Univ. Maryland Press, Potomac, MD, 1981).

  41. 41.

    Dalkey, N. & Helmer, O. An experimental application of the Delphi method to the use of experts. Manag. Sci. 9, 458–467 (1963).

  42. 42.

    Tetlock, P. Expert Political Judgment: How Good Is It? How Can We Know? (Princeton Univ. Press, Princeton, NJ, 2005).

  43. 43.

    Sunstein, C. R. Infotopia: How Many Minds Produce Knowledge (Oxford Univ. Press, Oxford, 2006).

  44. 44.

    Harvey, N. & Fischer, I. Taking advice: accepting help, improving judgment, and sharing responsibility. Organ. Behav. Hum. Decis. Process. 70, 117–133 (1997).

  45. 45.

    Eisenberger, N. I., Lieberman, M. D. & Williams, K. D. Does rejection hurt? An FMRI study of social exclusion. Science 302, 290–292 (2003).

  46. 46.

    Mahmoodi, A. et al. Equality bias impairs collective decision-making across cultures. Proc. Natl Acad. Sci. USA 112, 3835–3840 (2015).

  47. 47.

    Galton, F. One vote, one value. Nature 75, 414 (1907).

  48. 48.

    Mill, J. S. On Liberty (John W. Parker and Son, London, 1859).

  49. 49.

    Fishkin, J. S. & Luskin, R. C. Experimenting with a democratic ideal: deliberative polling and public opinion. Acta Polit. 40, 284–298 (2005).

  50. 50.

    Austen-Smith, D. & Banks, J. S. Information aggregation, rationality, and the Condorcet jury theorem. Am. Political Sci. Rev. 90, 34–45 (1996).

  51. 51.

    Cohen, J. in Deliberative Democracy: Essays on Reason and Politics (eds Bohman, J. & Rehg, W.) Ch. 3 (MIT Press, Boston, MA, 1997).

  52. 52.

    Lopez-Rosenfeld, M. et al. Neglect in human communication: quantifying the cost of cell-phone interruptions in face to face dialogs. PLoS ONE 10, e0125772 (2015).

  53. 53.

    Niella, T., Stier-Moses, N. & Sigman, M. Nudging cooperation in a crowd experiment. PLoS ONE 11, e0147125 (2016).

Download references

Acknowledgements

J.N. and B.B. were supported by the European Research Council StG (NEUROCODEC, #309865); M.S. was supported by the James McDonnell Foundation 21st Century Science Initiative in Understanding Human Cognition—Scholar Award (Grant #220020334), and by Agencia Nacional de Promoción Científica y Tecnológica (Argentina)—Préstamo BID PICT (Grant #2013-1653). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank M. Sartorio for assistance in data collection.

Author information

J.N., T.N., G.G. and M.S. designed and conducted the experiments. J.N., B.B. and M.S. developed the analysis approach. J.N. analysed the data. J.N., B.B. and M.S. wrote the paper.

Competing interests

The authors declare no competing interests.

Correspondence to Joaquin Navajas.

Supplementary information

Supplementary Information

Supplementary Figures 1–4, Supplementary Table 1

Life Sciences Reporting Summary

Supplementary Video

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Further reading

Fig. 1: Aggregating debates and the wisdom of crowds.
Fig. 2: The effect of deliberation on bias and variance.
Fig. 3: The superior wisdom of deliberative crowds.