Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Megastudies improve the impact of applied behavioural science

Nature volume 600, pages 478–483 (2021)Cite this article

Subjects

Abstract

Policy-makers are increasingly turning to behavioural science for insights about how to improve citizens’ decisions and outcomes1. Typically, different scientists test different intervention ideas in different samples using different outcomes over different time intervals2. The lack of comparability of such individual investigations limits their potential to inform policy. Here, to address this limitation and accelerate the pace of discovery, we introduce the megastudy—a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. In a megastudy targeting physical exercise among 61,293 members of an American fitness chain, 30 scientists from 15 different US universities worked in small independent teams to design a total of 54 different four-week digital programmes (or interventions) encouraging exercise. We show that 45% of these interventions significantly increased weekly gym visits by 9% to 27%; the top-performing intervention offered microrewards for returning to the gym after a missed workout. Only 8% of interventions induced behaviour change that was significant and measurable after the four-week intervention. Conditioning on the 45% of interventions that increased exercise during the intervention, we detected carry-over effects that were proportionally similar to those measured in previous research3,4,5,6. Forecasts by impartial judges failed to predict which interventions would be most effective, underscoring the value of testing many ideas at once and, therefore, the potential for megastudies to improve the evidentiary value of behavioural science.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Measured versus predicted changes in weekly gym visits induced by interventions.

Data availability

The data analysed in this paper were provided by 24 Hour Fitness and we have their legal permission to share the deidentified data. We have therefore made deidentified data available at https://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3. Furthermore, tables of all of the preregistration links for each of the substudies with the interventions and the prediction studies are available in Supplementary Tables 2 and 30.

Code availability

The code to replicate the analyses and figures in the paper and Supplementary Information is available online (https://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3).

References

  1. Behavioural Insights and Public Policy: Lessons from Around the World (OECD Publishing, 2017).

  2. Benartzi, S. et al. Should governments invest more in nudging? Psychol. Sci. 28, 1041–1055 (2017).

    Article  Google Scholar 

  3. Charness, G. & Gneezy, U. Incentives to EXercise. Econometrica 77, 909–931 (2009).

    Article  Google Scholar 

  4. Acland, D. & Levy, M. R. Naiveté, projection bias, and habit formation in gym attendance. Manage. Sci. 61, 146–160 (2015).

    Article  Google Scholar 

  5. Royer, H., Stehr, M. & Sydnor, J. Incentives, commitments, and habit formation in exercise: evidence from a field experiment with workers at a Fortune-500 company. Am. Econ. J. Appl. Econ. 7, 51–84 (2015).

    Article  Google Scholar 

  6. Beshears, J., Lee, H. N., Milkman, K. L., Mislavsky, R. & Wisdom, J. Creating exercise habits using incentives: the tradeoff between flexibility and routinization. Manage. Sci. 67, 4139-4171 (2020).

    Article  Google Scholar 

  7. DellaVigna, S. & Linos, E. RCTs to Scale: Comprehensive Evidence from Two Nudge Units 65 (National Bureau of Economic Research, 2020).

  8. DellaVigna, S. & Pope, D. What motivates effort? Evidence and expert forecasts. Rev. Econ. Stud. 85, 1029–1069 (2018).

    Article  Google Scholar 

  9. DellaVigna, S. & Pope, D. Predicting experimental results: who knows what? J. Polit. Econ. 126, 2410–2456 (2018).

    Article  Google Scholar 

  10. DellaVigna, S., Pope, D. & Vivalt, E. Predict science to improve science. Science 366, 428–429 (2019).

    Article  CAS  ADS  Google Scholar 

  11. Kristal, A. S. & Whillans, A. V. What we can learn from five naturalistic field experiments that failed to shift commuter behaviour. Nat. Hum. Behav. 4, 169–176 (2020).

    Article  Google Scholar 

  12. Donoho, D. 50 years of data science. J. Comput. Graph. Stat. 26, 745–766 (2017).

    Article  MathSciNet  Google Scholar 

  13. Liberman, M. Fred Jelinek. Comput. Linguist. 36, 595–599 (2010).

    Article  Google Scholar 

  14. Lai, C. K. et al. Reducing implicit racial preferences: I. A comparative investigation of 17 interventions. J. Exp. Psychol. Gen. 143, 1765–1785 (2014).

    Article  Google Scholar 

  15. Lai, C. K. et al. Reducing implicit racial preferences: II. Intervention effectiveness across time. J. Exp. Psychol. Gen. 145, 1001–1016 (2016).

    Article  Google Scholar 

  16. Mellers, B. et al. Psychological strategies for winning a geopolitical forecasting tournament. Psychol. Sci. 25, 1106–1115 (2014).

    Article  Google Scholar 

  17. Open Science Collaboration Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).

    Article  Google Scholar 

  18. Milkman, K. L., Minson, J. A. & Volpp, K. G. M. Holding the hunger games hostage at the gym: an evaluation of temptation bundling. Manage. Sci. 60, 283–299 (2014).

    Article  Google Scholar 

  19. Ward, B. W., Clarke, T. C., Nugent, C. N. & Schiller, J. S. Early Release of Selected Estimates Based on Data From the 2015 National Health Interview Survey 120 (National Center for Health Statistics, 2015).

  20. Lee, I.-M. et al. Effect of physical inactivity on major non-communicable diseases worldwide: an analysis of burden of disease and life expectancy. Lancet 380, 219–229 (2012).

    Article  Google Scholar 

  21. Gollwitzer, P. M. Implementation intentions: strong effects of simple plans. Am. Psychol. 54, 493–503 (1999).

    Article  Google Scholar 

  22. Milkman, K. L., Beshears, J., Choi, J. J., Laibson, D. & Madrian, B. C. Using implementation intentions prompts to enhance influenza vaccination rates. Proc. Natl Acad. Sci. USA 108, 10415–10420 (2011).

    Article  CAS  ADS  Google Scholar 

  23. Rogers, T., Milkman, K. L., John, L. K. & Norton, M. I. Beyond good intentions: prompting people to make plans improves follow-through on important tasks. Behav. Sci. Pol. 1, 33–41 (2015).

    Article  Google Scholar 

  24. Karlan, D., McConnell, M., Mullainathan, S. & Zinman, J. Getting to the top of mind: how reminders increase saving. Manage. Sci. 62, 3393–3411 (2016).

    Article  Google Scholar 

  25. Homonoff, T. A. Can small incentives have large effects? The impact of taxes versus bonuses on disposable bag use. Am. Econ. J. Econ. Pol. 10, 177–210 (2018).

    Article  Google Scholar 

  26. Storey, J. D. & Tibshirani, R. Statistical significance for genomewide studies. Proc. Natl Acad. Sci. USA 100, 9440–9445 (2003).

    Article  MathSciNet  CAS  ADS  Google Scholar 

  27. Allcott, H. Social norms and energy conservation. J. Publ. Econ. 95, 1082–1095 (2011).

    Article  Google Scholar 

  28. Chapman, G. B., Li, M., Colby, H. & Yoon, H. Opting in vs opting out of influenza vaccination. JAMA 304, 43–44 (2010).

    Article  CAS  Google Scholar 

  29. Milkman, K. L., et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment. Proc. Natl Acad. Sci. USA 118, e2101165118 (2021).

    Article  CAS  Google Scholar 

  30. Lee, M. R. & Shen, M. Winner’s curse: bias estimation for total effects of features in online controlled experiments. In Proc. 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 491–499 (ACM, 2018).

  31. White, H. Asymptotic Theory for Econometricians (Elsevier, 1984).

  32. Dubner, S. J. How goes the behavior-change revolution? (Ep. 382). Freakonomics https://freakonomics.com/podcast/live-philadelphia/ (2019).

Download references

Acknowledgements

Support for this research was provided in part by the Robert Wood Johnson Foundation, the AKO Foundation, J. Alexander, M. J. Leder, W. G. Lichtenstein, the Pershing Square Fund for Research on the Foundations of Human Behavior from Harvard University and by Roybal Center grants (P30AG034546 and 5P30AG034532) from the National Institute on Aging. The views expressed here do not necessarily reflect the views of any of these individuals or entities. We thank 24 Hour Fitness for partnering with the Behavior Change for Good Initiative at the University of Pennsylvania to make this research possible.

Author information

Author notes

  1. Hung Ho

    Present address: Department of Marketing, Booth School of Business, University of Chicago, Chicago, IL, USA

  2. Timothy W. Lee

    Present address: McCormick School of Engineering, Northwestern University, Evanston, IL, USA

Authors and Affiliations

  1. Department of Operations, Information and Decisions, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA

    Katherine L. Milkman, Hung Ho, Aneesh Rai, Erika Kirgios & Angela L. Duckworth

  2. Behavior Change for Good Initiative, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA

    Dena Gromet, Joseph S. Kay & Timothy W. Lee

  3. Harris School of Public Policy, University of Chicago, Chicago, IL, USA

    Pepi Pandiloski & Jens Ludwig

  4. Department of Psychology, Princeton University, Princeton, NJ, USA

    Yeji Park

  5. Department of Negotiation, Organizations & Markets, Harvard Business School, Harvard University, Boston, MA, USA

    Max Bazerman, John Beshears, Edward Chang & Ashley Whillans

  6. Pritzker School of Law, Northwestern University, Chicago, IL, USA

    Lauri Bonacorsi

  7. Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA

    Colin Camerer

  8. Department of Social and Decision Sciences, Carnegie Mellon University, Pittsburgh, PA, USA

    Gretchen Chapman, Samantha Horn, George Loewenstein, Silvia Saccardo & Joachim H. Talloen

  9. Department of Psychology, Arizona State University, Tempe, AZ, USA

    Robert Cialdini

  10. Department of Management and Organizations, Anderson School of Management, University of California Los Angeles, Los Angeles, CA, USA

    Hengchen Dai

  11. Department of Behavioral Science, Booth School of Business, University of Chicago, Chicago, IL, USA

    Lauren Eskreis-Winkler, Ayelet Fishbach & Sendhil Mullainathan

  12. Department of Psychology, Stanford University, Stanford, CA, USA

    James J. Gross & Jamie Taxer

  13. Department of Psychology, New York University, New York, NY, USA

    Alexa Hubbard & Yaacov Trope

  14. Department of Psychology, Rutgers University, New Brunswick, NJ, USA

    Steven J. Jones

  15. Department of Finance, Kellogg School of Management, Northwestern University, Evanston, IL, USA

    Dean Karlan

  16. Mathematica, Princeton, NJ, USA

    Tim Kautz

  17. Department of Marketing, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA

    Joowon Klusowski & Barbara Mellers

  18. Department of Organizational Behavior, Harvard Business School, Harvard University, Boston, MA, USA

    Ariella Kristal

  19. Department of Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor, MI, USA

    Rahul Ladhania

  20. Department of Operations, Information & Technology, Stanford Graduate School of Business, Stanford, CA, USA

    Jann Spiess

  21. Department of Psychology, San Francisco State University, San Francisco, CA, USA

    Gaurav Suri

  22. Department of Computer and Information Sciences, University of Pennsylvania, Philadelphia, PA, USA

    Lyle Ungar

  23. Department of Medical Ethics and Health Policy, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

    Kevin G. Volpp

  24. Department of Economics, Dartmouth College, Hanover, NH, USA

    Jonathan Zinman

  25. Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA

    Angela L. Duckworth

Authors
  1. Katherine L. Milkman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dena Gromet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hung Ho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joseph S. Kay
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Timothy W. Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Pepi Pandiloski
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yeji Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Aneesh Rai
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Max Bazerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. John Beshears
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Lauri Bonacorsi
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Colin Camerer
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Edward Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Gretchen Chapman
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Robert Cialdini
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Hengchen Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Lauren Eskreis-Winkler
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Ayelet Fishbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. James J. Gross
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Samantha Horn
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Alexa Hubbard
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Steven J. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Dean Karlan
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Tim Kautz
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Erika Kirgios
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Joowon Klusowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Ariella Kristal
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Rahul Ladhania
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. George Loewenstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  30. Jens Ludwig
    View author publications

    You can also search for this author in PubMed Google Scholar

  31. Barbara Mellers
    View author publications

    You can also search for this author in PubMed Google Scholar

  32. Sendhil Mullainathan
    View author publications

    You can also search for this author in PubMed Google Scholar

  33. Silvia Saccardo
    View author publications

    You can also search for this author in PubMed Google Scholar

  34. Jann Spiess
    View author publications

    You can also search for this author in PubMed Google Scholar

  35. Gaurav Suri
    View author publications

    You can also search for this author in PubMed Google Scholar

  36. Joachim H. Talloen
    View author publications

    You can also search for this author in PubMed Google Scholar

  37. Jamie Taxer
    View author publications

    You can also search for this author in PubMed Google Scholar

  38. Yaacov Trope
    View author publications

    You can also search for this author in PubMed Google Scholar

  39. Lyle Ungar
    View author publications

    You can also search for this author in PubMed Google Scholar

  40. Kevin G. Volpp
    View author publications

    You can also search for this author in PubMed Google Scholar

  41. Ashley Whillans
    View author publications

    You can also search for this author in PubMed Google Scholar

  42. Jonathan Zinman
    View author publications

    You can also search for this author in PubMed Google Scholar

  43. Angela L. Duckworth
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K.L.M., D.G., A.R., M.B., J.B., L.B., E.C., G.C., R.C., H.D., L.E.-W., A.F., J.J.G., S.H., A.H., S.J.J., D.K., E.K., J.K., A.K., G.L., B.M., S.M., S.S., G.S., J.H.T., J.T., Y.T., L.U., K.G.V., A.W., J.Z. and A.L.D. designed the research. K.L.M., D.G., J.S.K., P.P., Y.P., A.L.D. and A.R. performed the research. H.H., T.W.L., P.P. and Y.P. analysed the data. K.L.M. and A.L.D wrote the paper. D.G., H.H., J.S.K., T.W.L., P.P., Y.P., A.R., M.B., J.B., C.C., G.C., H.D., A.F., J.J.G., D.K., T.K., E.K., J.K., R.L., J.L., B.M., S.M., S.S., J.S., A.W. and J.Z. provided feedback on the paper. K.L.M., D.G., J.S.K., T.K., R.L. and S.M. supervised data analysis. K.L.M., D.G., H.H., J.S.K. and T.W.L. prepared the Supplementary Information.

Corresponding authors

Correspondence to Katherine L. Milkman or Angela L. Duckworth.

Ethics declarations

Competing interests

The authors declare no competing interests. The authors did not receive commercial benefits from the fitness chain or speaking/consulting fees related to any of the interventions presented here.

Additional information

Peer review information Nature thanks Charles Shearer and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Measured vs. predicted change in likelihood of gym visit in a given week.

The measured change (blue) vs. change predicted by third-party observers (gold) in whether participants visited the gym that was induced by each of our megastudy’s 53 experimental conditions compared to a Placebo Control condition during a four-week intervention period is depicted here. Error bars represent 95% confidence intervals. See Extended Data Table 7 for complete OLS regression results graphed here in blue, Supplementary Information 11 for more details about the prediction data graphed here in gold, and Supplementary Table 1 for full descriptions of each treatment condition in our megastudy. Sample weights were included in the pooled third-party prediction data to ensure equal weighting of each of our three participant samples (professors, practitioners and prolific respondents). The superscripts a–e denote the different incentive amounts offered in different versions of the bonus for returning after missed workouts, higher incentives and rigidity rewarded conditions, which are described in Supplementary Table 1. In conditions with the same name, superscripts that come earlier in the alphabet indicate larger incentives.

Extended Data Table 1 Regression-estimated effects of each experimental condition on whether participants visited the gym in a given week during the four-week intervention period relative to the Planning, Reminders and Micro-Incentives to Exercise condition
Full size table
Extended Data Table 2 Regression-estimated effects of each experimental condition on whether participants visited the gym in a given week during the four-week post-intervention period relative to the Placebo Control condition
Full size table
Extended Data Table 3 The percentage of other conditions that each experimental condition outperformed for our dependent variable measuring whether participants visited the gym in a given week at p < .05 during the four-week intervention period
Full size table
Extended Data Table 4 Participants’ mean age (in years), gender, length of gym membership (in weeks), and mean weekly gym visits in the four-week pre-intervention period across the 54 study conditions
Full size table
Extended Data Table 5 Percentage of significant p-values and absolute difference in coefficients from pairwise comparisons of the 54 study conditions in our megastudy on each variable listed (alpha = .05)
Full size table
Extended Data Table 6 Regression-estimated effects of each experimental condition on total weekly gym visits during the four-week intervention period relative to the Placebo Control condition
Full size table
Extended Data Table 7 Regression-estimated effects of each experimental condition on whether participants visited the gym in a given week during the four-week intervention period relative to the Placebo Control condition
Full size table
Extended Data Table 8 Regression-estimated effects of each experimental condition on total weekly gym visits during the four-week intervention period relative to the Planning, Reminders, and Micro-Incentives to Exercise condition
Full size table
Extended Data Table 9 Regression-estimated effects of each experimental condition on total weekly gym visits during the four-week post-intervention period relative to the Placebo Control condition
Full size table

Supplementary information

Supplementary Information

Supplementary Figures, Tables and Notes.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Milkman, K.L., Gromet, D., Ho, H. et al. Megastudies improve the impact of applied behavioural science. Nature 600, 478–483 (2021). https://doi.org/10.1038/s41586-021-04128-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-021-04128-4

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing