Letter | Published:

Election polling errors across time and space

Nature Human Behaviourvolume 2pages276283 (2018) | Download Citation

Subjects

Abstract

Are election polling misses becoming more prevalent? Are they more likely in some contexts than others? Here we undertake an over-time and cross-national assessment of prediction errors in pre-election polls. Our analysis draws on more than 30,000 national polls from 351 general elections in 45 countries between 1942 and 2017. We proceed in the following way. First, building on previous studies, we show how errors in national polls evolve in a structured way over the election timeline. Second, we examine errors in polls in the final week of the election campaign to assess performance across election years. Third, we undertake a pooled analysis of polling errors—controlling for a number of institutional and party features—that enables us to test whether poll errors have increased or decreased over time. We find that, contrary to conventional wisdom, the recent performance of polls has not been outside the ordinary. However, the performance of polls does vary across political contexts and in understandable ways.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Sturgis, P. et al. Report of the Inquiry into the 2015 British General Election Opinion Polls (Market Research Society and British Polling Council, 2016).

  2. 2.

    An Evaluation of 2016 Election Polls in the US (AAPOR, 2017); http://www.aapor.org/Education-Resources/Reports/An-Evaluation-of-2016-Election-Polls-in-the-U-S.aspx

  3. 3.

    Silver, N. The world may have a polling problem. FiveThirtyEight http://fivethirtyeight.com/live-blog/uk-general-election-2015/ (2015).

  4. 4.

    Williams, J. P. The problem with polls. U.S. News and World Report (28 September 2015); https://www.usnews.com/news/the-report/articles/2015/09/28/why-public-opinion-polls-are-increasingly-inaccurate

  5. 5.

    Zukin, C. What’s the matter With polling? The New York Times (21 June 2015); https://www.nytimes.com/2015/06/21/opinion/sunday/whats-the-matter-with-polling.html?_r=0

  6. 6.

    Cassino, D. Why pollsters were completely and utterly wrong. Harvard Business Review (2016); https://hbr.org/2016/11/why-pollsters-were-completely-and-utterly-wrong

  7. 7.

    Santos, R. Why the polls get it wrong. The Los Angeles Times (27 March 2016); http://www.latimes.com/opinion/op-ed/la-oe-0327-santos-polling-problems-20160327-story.html

  8. 8.

    Barfar, A. & Padmanabhan, B. Election watch. Significance 13, 30–33 (2016).

  9. 9.

    Performance of the Polls in the EU Referendum (British Polling Council, 2016); http://www.britishpollingcouncil.org/performance-of-the-polls-in-the-eu-referendum/

  10. 10.

    Jennings, W. & Wlezien, C. The timeline of elections: a comparative perspective. Am. J. Political Sci. 60, 219–233 (2016).

  11. 11.

    Erikson, R. S. & Wlezien, C. The Timeline of Presidential Elections: How Campaigns Do (and Do Not) Matter. (Univ. Chicago Press: Chicago, IL, 2012).

  12. 12.

    Wlezien, C., Jennings, W. & Erikson, R. S. The ‘timeline’ method of studying electoral dynamics. Elect. Stud. 48, 45–56 (2017).

  13. 13.

    Keeter, S., Hatley, N., Kennedy, C. & Lau, A. What Low Response Rates Mean for Telephone Surveys (Pew Research Center, 2017); http://www.pewresearch.org/2017/05/15/what-low-response-rates-mean-for-telephone-surveys/

  14. 14.

    Mellon, J. & Prosser, C. Missing nonvoters and misweighted samples: explaining the 2015 great British polling miss. Public Opin. Q. 81, 661–687 (2017).

  15. 15.

    Erikson, R. S. & Wlezien, C. Presidential polls as a time series: the case of 1996. Public Opin. Q. 63, 163–177 (1999).

  16. 16.

    An Evaluation of the Methodology of the 2008 Pre-Election Primary Polls (AAPOR, 2009); https://www.aapor.org/AAPOR_Main/media/MainSiteFiles/AAPOR_Rept_FINAL-Rev-4-13-09.pdf

  17. 17.

    Pickup, M. & Johnston, R. Campaign trial heats as election forecasts: measurement error and bias in 2004 presidential campaign polls. Int. J. Forecast. 24, 272–284 (2008).

  18. 18.

    Ford, R., Jennings, W., Pickup, M. & Wlezien, C. From polls to votes to seats: forecasting the 2015 British general election. Elect. Stud. 41, 244–249 (2016).

  19. 19.

    Kriesi, H. The transformation of cleavage politics: the 1997 Stein Rokkan lecture. Eur. J. Political Res. 33, 165–185 (1998).

  20. 20.

    Mair, P. Ruling The Void: The Hollowing of Western Democracy (Verso, London, 2013).

  21. 21.

    Franklin, M. The Decline of Class Voting in Britain: Changes in the Basis of Electoral Choice, 1964–1983 (Clarendon Press, Oxford, 1985).

  22. 22.

    Knutsen, O. Class Voting in Western Europe: A Comparative Longitudinal Study (Lexington Books, Lanham, MD, 2006).

  23. 23.

    Evans, G. and Tilley, J. The New Politics of Class: The Political Exclusion of the British Working Class (Oxford Univ. Press, Oxford, 2017).

  24. 24.

    Kayser, M. & Wlezien, C. Performance pressure: patterns of partisanship and the economic vote. Eur. J. Polit. Res. 50, 365–394 (2011).

  25. 25.

    Groves, R. M. Survey Errors and Survey Costs (John Wiley & Sons, Hoboken, NJ, 2004).

  26. 26.

    Martin, E. A., Michael, W. T. & Kennedy, C. A review and proposal for a new measure of poll accuracy. Public Opin. Q. 69, 342–369 (2005).

  27. 27.

    Arzheimer, K. & Evans, J. A new multinomial accuracy measure for polling bias. Polit. Anal. 22, 31–44 (2014).

  28. 28.

    Laakso, M. & Taagepera, R. The “effective” number of parties: a measure with application to West Europe. Comp. Polit. Stud. 12, 3–27 (1979).

  29. 29.

    Kennedy, C. et al. An Evaluation of 2016 Election Polls in the United States: AAPOR Task Force Report (AAPOR, 2017); http://www.aapor.org/getattachment/Education-Resources/Reports/AAPOR-2016-Election-Polling-Report.pdf.aspx

  30. 30.

    Our Survey Methodology in Detail (Pew Research Center, 2018); http://www.pewresearch.org/methodology/u-s-survey-research/our-survey-methodology-in-detail/

Download references

Acknowledgements

An earlier version of this article was presented at the Advertising Research Foundation’s ForecastxScience meeting, Google, Sunnyvale, California, 14–15 November 2017. We thank C. Kennedy, D. Rothschild, P. Sturgis and G. Terhanian for comments on earlier versions of this paper, and to Lord Lipsey and other members of the House of Lords Select Committee on Political Polling and Digital Media for their comments and questions. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

  1. Department of Politics and International Relations, University of Southampton, Southampton, UK

    • Will Jennings
  2. Department of Government, University of Texas at Austin, Austin, Texas, USA

    • Christopher Wlezien

Authors

  1. Search for Will Jennings in:

  2. Search for Christopher Wlezien in:

Contributions

C.W. and W.J. developed the original study concept. W.J. and C.W. gathered and analysed the data, and drafted and revised the manuscript. W.J. wrote the computer code and generated the figures and tables in Stata.

Competing interests

The authors declare no competing interests.

Corresponding authors

Correspondence to Will Jennings or Christopher Wlezien.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41562-018-0315-6

Further reading