There is widespread public and academic interest in understanding the uses and effects of digital media. Scholars primarily use self-report measures of the quantity or duration of media use as proxies for more objective measures, but the validity of these self-reports remains unclear. Advancements in data collection techniques have produced a collection of studies indexing both self-reported and log-based measures. To assess the alignment between these measures, we conducted a pre-registered meta-analysis of this research. Based on 106 effect sizes, we found that self-reported media use correlates only moderately with logged measurements, that self-reports were rarely an accurate reflection of logged media use and that measures of problematic media use show an even weaker association with usage logs. These findings raise concerns about the validity of findings relying solely on self-reported measures of media use.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The raw and processed data are available on the Open Science Framework website (https://osf.io/dhx48/). These data include all extracted effect sizes, study descriptives and descriptive statistics. In cases where raw data were provided by study authors, as with all included studies, we only provide the necessary descriptive statistics and effective sizes used to compute the summary statistics in the meta-analyses, but do not share these original authors’ data. The data have been assigned a unique identifier: https://doi.org/10.17605/OSF.IO/JS6YE.
The code (written in the R statistical language) used to analyse the relevant data is provided on the Open Science Framework website (https://osf.io/dhx48/). All materials needed to reproduce the analyses are available at this link.
Dickson, K. et al. Screen-Based Activities and Children and Young People’s Mental Health and Psychosocial Wellbeing: A Systematic Map of Reviews (EPPI-Centre, Univ. College London, 2019).
Twenge, J. M. Have smartphones destroyed a generation? The Atlantic. https://www.theatlantic.com/magazine/archive/2017/09/has-the-smartphone-destroyed-a-generation/534198/ (September 2017).
Orben, A. The Sisyphean cycle of technology panics. Perspect Psychol. Sci. 15, 1143–1157 (2020).
Orben, A. & Przybylski, A. K. The association between adolescent well-being and digital technology use. Nat. Hum. Behav. 3, 173 (2019).
Scharkow, M. The accuracy of self-reported internet use—a validation study using client log data. Commun. Methods Measures 10, 13–27 (2016).
House of Commons Science and Technology Committee. 14th Report - Impact of Social Media and Screen-Use on Young People’s Health https://tinyurl.com/y5omat3s (2019).
Allen, R. L. The reliability and stability of television exposure. Commun. Res. 8, 233–256 (1981).
Bartels, L. M. Messages received: the political impact of media exposure. Am. Political Sci. Rev. 87, 267–285 (1993).
Price, V. The impact of varying reference periods in survey questions about media use. Journal. Q. 70, 615–627 (1993).
Collopy, F. Biases in retrospective self-reports of time use: an empirical study of computer users. Manag. Sci. 42, 758–767 (1996).
Greenberg, B. et al. Comparing survey and diary measures of internet and traditional media use. Commun. Rep. 18, 1–8 (2005).
Coromina, L. & Saris, W. E. Quality of media use measurement. Int. J. Public Opin. Res. 21, 424–450 (2009).
Gerpott, T. J. & Thomas, S. Empirical research on mobile internet usage: a meta-analysis of the literature. Telecommun. Policy 38, 291–310 (2014).
Howard, M. C. & Jayne, B. S. An analysis of more than 1,400 articles, 900 scales, and 17 years of research: the state of scales in cyberpsychology, behavior, and social networking. Cyberpsychol. Behav. Soc. Netw. 18, 181–187 (2015).
Griffioen, N., van Rooij, M., Lichtwarck-Aschoff, A. & Granic, I. Toward improved methods in social media research. Technol. Mind Behav. https://doi.org/10.1037/tmb0000005 (2020).
Jungselius, B., & Weilenmann, A. Conceptualizing ‘use’ in social media studies. In Proc. 9th Int. Conf. Social Media Soc. (pp. 325–329) (2018).
Humphreys, L., Karnowski, V. & von Pape, T. Smartphones as metamedia: a framework for identifying the niches structuring smartphone use. Int. J. Commun. 12, 2793–2809 (2018).
Abendroth, A., Parry, D. A., le Roux, D. B., & Gundlach, J. An analysis of problematic media use and technology use addiction scales–what are they actually assessing? in Conference on e-Business, e-Services and e-Society 211–222 (Springer, 2020).
Davidson, B. I., Shaw, H., & Ellis, D. A. Fuzzy constructs in assessment: the overlap between mental health and technology ‘use’. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/6durk (2020).
Flake, J. K. & Fried, E. I. Measurement schmeasurement: questionable measurement practices and how to avoid them. Adv. Methods. Pract. Psychol. 3, 456–465 (2020).
Ellis, D. A., Davidson, B. I., Shaw, H. & Geyer, K. Do smartphone usage scales predict behavior? Int. J. Hum. Comput. Stud. 130, 86–92 (2019).
Ellis, D. A. Are smartphones really that bad? improving the psychological measurement of technology-related behaviors. Comput. Hum. Behav. 97, 60–66 (2019).
Lemmens, J. S., Valkenburg, P. M. & Peter, J. Development and validation of a game addiction scale for adolescents. Media Psychol. 12, 77–95 (2009).
Jobe, J. B. Cognitive psychology and self-reports: models and methods. Qual. Life Res. 12, 219–227 (2003).
Sudman, S., Bradburn, N. M., & Schwarz, N. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology (Jossey-Bass, 1996).
Schwarz, N. & Oyserman, D. Asking questions about behavior: cognition, communication, and questionnaire construction. Am. J. Eval. 22, 127–160 (2001).
Tourangeau, R., et al. Cognitive Aspects of Survey Methodology: Building a Bridge between Disciplines: Report of the Advanced Research Seminar on Cognitive Aspects of Survey Methodology (National Academic, 1984).
Belson, W. A. The Design and Understanding of Survey Questions (Gower, 1981).
Mieczkowski, H., Lee, A. Y. & Hancock, J. T. Priming effects of social media use scales on well-being outcomes: the influence of intensity and addiction scales on self-reported depression. Soc. Media+ Soc. 6, 2056305120961784 (2020).
Neisser, U. Nested structure in autobiographical memory. in Autobiographical Memory (ed. Rubin D. C.) 71–88 (Cambridge Univ. Press, 1986).
Sewall, C. J., Bear, T. M., Merranko, J. & Rosen, D. How psychosocial well-being and usage amount predict inaccuracies in retrospective estimates of digital technology use. Mob. Media Commun. 8, 379–399 (2020).
Andrews, S., Ellis, D. A., Shaw, H. & Piwek, L. Beyond self-report: tools to compare estimated and real-world smartphone use. PLoS ONE 10, e0139004 (2015).
Vorderer, P., Krӧmer, N. & Schneider, F. M. Permanently online–permanently connected: explorations into university students’ use of social media and mobile smart devices. Comput. Hum. Behav. 63, 694–703 (2016).
Piewk, L., Ellis, D. A. & Andrews, S. Can programming frameworks bring smartphones into the mainstream of psychological science? Front. Psychol. 7, 1252 (2016).
Ryding, F. C. & Kuss, D. J. Passive objective measures in the assessment of problematic smartphone use: a systematic review. Addict. Behav. Rep. 11, 100257 (2020).
Boase, J. & Ling, R. Measuring mobile phone use: self-report versus log data. J. Comput.-Mediat. Commun. 18, 508–519 (2013).
Vanden Abeele, M., Beullens, K. & Roe, K. Measuring mobile phone use: gender, age and real usage level in relation to the accuracy and validity of self-reported mobile phone use. Mob. Media Commun. 1, 213–236 (2013).
Kobayashi, T. & Boase, J. No such effect? The implications of measurement error in self-report measures of mobile communication use. Commun. Methods Measures 6, 126–143 (2012).
Deng, T. et al. Measuring smartphone usage and task switching with log tracking and self-reports. Mob. Media Commun. 7, 3–23 (2019).
Jürgens, P., Stark, B. & Magin, M. Two half-truths make a whole? On bias in self-reports and tracking data. Soc. Sci. Comput. Rev. 38, 600–615 (2019).
Ohme, J., Araujo, T., de Vreese, C. H. & Piotrowski, J. T. Mobile data donations: assessing self-report accuracy and sample biases with the iOS Screen Time function. Mob. Media Commun. 9, 293–313 (2021).
Parry, D., et al. Associations between log-based and self-reported digital media use: a systematic review and meta-analysis protocol. Preprint at OSF Preprints https://osf.io/xrg5c (2020).
Araujo, T., Wonneberger, A., Neijens, P. & de Vreese, C. How much time do you spend online? Understanding and improving the accuracy of self-reported measures of Internet use. Commun. Methods Measures 11, 173–190 (2017).
Burke, M., Marlow, C., & Lento, T. Social network activity and social well-being. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1909–1912 (2010).
Elhai, J. et al. Depression and emotion regulation predict objective smartphone use measured over one week. Personal. Individ. Differences 133, 21–28 (2018).
Ernala, S. K., Burke, M., Leavitt, A., & Ellison, N. B. How well do people report time spent on Facebook? An evaluation of established survey questions with recommendations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems 1–14 (2020).
Felisoni, D. D. & Godoi, A. S. Cell phone usage and academic performance: an experiment. Comput. Educ. 117, 175–187 (2018).
Inyang, I., Benke, G., Morrissey, J., McKenzie, R. & Abramson, M. How well do adolescents recall use of mobile telephones? Results of a validation study. BMC Med. Res. Method. 9, 36 (2009).
Jones-Jang, S. M. et al. Good news! communication findings may be underestimated: comparing effect sizes with self-reported and logged smartphone use data. J. Comput.-Mediat. Commun. 25, 346–363 (2020).
Junco, R. Comparing actual and self-reported measures of Facebook use. Comput. Hum. Behav. 29, 626–631 (2013).
Kahn, A. S., Ratan, R. & Williams, D. Why we distort in self-report: predictors of self-report errors in video game play. J. Comput.-Mediat. Commun. 19, 1010–1023 (2014).
Katapally, T. R. & Chu, L. M. Methodology to derive objective screen-state from smartphones: A SMART platform study. Int. J. Environ. Res. Public Health 16, 2275 (2019).
Lee, H., Ahn, H., Choi, S. & Choi, W. The SAMS: smartphone addiction management system and verification. J. Med. Syst. 38 (2014).
Lee, H., Ahn, H., Nguyen, T. G., Choi, S. W. & Kim, D. J. Comparing the self-report and measured smartphone usage of college students: a pilot study. Pyschiatry Invest. 14, 198–204 (2017).
Lin, Y. et al. Time distortion associated with smartphone addiction: Identifying smartphone addiction via a mobile application (app). J. Psychiatr. Res. 65, 139–145 (2015).
Loid, K., Täht, K. & Rozgonjuk, D. Do pop-up notifications regarding smartphone use decrease screen time, phone checking behavior, and self-reported problematic smartphone use? Evidence from a two-month experimental study. Comput. Hum. Behav. 102, 22–30 (2020).
Mikkelsen, S. et al. Validity of questionnaire self-reports on computer, mouse and keyboard usage during a four-week period. Occup. Environ. Med. 64, 541–547 (2007).
Montag, C. et al. Recorded behavior as a valuable resource for diagnostics in mobile phone addiction: evidence from psychoinformatics. Behav. Sci. 5, 434–442 (2015).
Newell, C. A. Masters dissertation Smartphone and Social Media Use and Its Health Associations (Univ. Texas at Arlington, 2018).
Noë, B. et al. Identifying indicators of smartphone addiction through user–app interaction. Comput. Hum. Behav. 99, 56–65 (2019).
Oeldorf-Hirsch, A., & Chen, Y. Who Cares about Screen Time? Predicting the use of Mobile Phone Tracking Features. Presented at the 70th Annual Conference of the International Communication Association (ICA), Gold Coast, Australia (virtual conference) (2020)
Pan, Y. C., Lin, H. H., Chiu, Y. C., Lin, S. H. & Lin, Y. H. Temporal stability of smartphone use data: determining fundamental time unit and independent cycle. JMIR mHealth uHealth 7, e13421 (2019).
Prasad, S. et al. A study of magnitude and psychological correlates of smartphone use in medical students: a pilot study with a novel telemetric approach. Indian J. Psychol. Med. 40, 468–475 (2018).
Rosen, L. D. et al. The role of executive functioning and technological anxiety (FOMO) in college course performance as mediated by technology usage and multitasking habits. Educ. Psychol. 24, 14–25 (2017).
Rozgonjuk, D., Levine, J. C., Hall, B. J. & Elhai, J. D. The association between problematic smartphone use, depression and anxiety symptom severity, and objectively measured smartphone use over one week. Comput. Hum. Behav. 87, 10–17 (2018).
Rozgonjuk, D., Pruunsild, P., Jürimäe, K., Schwarz, R. J. & Aru, J. Instagram use frequency is associated with problematic smartphone use, but not with depression and anxiety symptom severity. Mob. Media Commun. 8, 400–418 (2020).
Sela, Y., Zach, M., Amichay-Hamburger, Y., Mishali, M. & Omer, H. Family environment and problematic internet use among adolescents: the mediating roles of depression and fear of missing out. Comput. Hum. Behav. 106, 106226 (2020).
Shaw, H. et al. Quantifying smartphone ‘use’: choice of measurement impacts relationships between ‘usage’ and health. Technol. Mind Behav. https://doi.org/10.1037/tmb0000022 (2020).
Shin, C., & Dey, A. K. Automatically detecting problematic use of smartphones. Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing 335–344 (2013).
Shin, M. & Lee, K. Measuring smartphone usage time is not sufficient to predict smartphone addiction. J. Theor. Appl. Inform. Technol. 95, 5296–5303 (2017).
Shum, M., Kelsh, M. A., Sheppard, A. R. & Zhao, K. An evaluation of self‐reported mobile phone use compared to billing records among a group of engineers and scientists. Bioelectromagnetics 32, 37–48 (2011).
Singh, V. K., & Jain, A. Toward harmonizing self-reported and logged social data for understanding human behavior. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems 2233–2238 (2017).
Timotijevic, L., Barnett, J., Shepherd, R. & Senior, V. Factors influencing self‐report of mobile phone use: the role of response prompt, time reference and mobile phone use in recall. Appl. Cogn. Psychol. 23, 664–683 (2009).
Tokola, K., Kurttio, P., Salminen, T. & Auvinen, A. Reducing overestimation in reported mobile phone use associated with epidemiological studies. Bioelectromagnetics 29, 559–563 (2008).
van Berkel, N. et al. Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports. Int. J. Hum. Comput. Stud. 125, 118–128 (2019).
Wilmer, H. H., Hampton, W. H., Olino, T. M., Olson, I. R. & Chein, J. M. Wired to be connected? Links between mobile technology engagement, intertemporal preference and frontostriatal white matter connectivity. Soc. Cogn. Affect. Neurosci. 14, 367–379 (2019).
Yuan, N. et al. How much do parents actually use their smartphones? Pilot study comparing self-report to passive sensing. Pediatr. Res. 86, 416–418 (2019).
Moyer, A. & Finney, J. W. Rating methodological quality: toward improved assessment and investigation. Account. Res. 12, 299–313 (2005).
Crowe, M. & Sheppard, L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J. Clin. Epidemiol. 64, 79–89 (2011).
Protogerou, C. & Hagger, M. S. A case for a study quality appraisal in survey studies in psychology. Front. Psychol. 9, 2788 (2019).
Protogerou, C. & Hagger, M. S. A checklist to assess the quality of survey studies in psychology. Methods Psychol. 3, 272–299 (2020).
Rodgers, M. A., & Pustejovsky, J. E. Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychol. Methods https://doi.org/10.1037/met0000300 (2020).
Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36, 1–48 (2010).
Geyer, K., Ellis, D. A., Shaw, H., & Davidson, B. I. Open source smartphone apps and tools for measuring, quantifying, and visualizing screen time. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/eqhfa (2020).
Nunnally, J. Psychometric Theory 3rd edn (McGraw-Hill Education, 1994).
Carlson, K. D. & Herdman, A. O. Understanding the impact of convergent validity on research results. Organ. Res. Methods 15, 17–32 (2012).
Social Media Addiction Reduction Technology Act, S. 2314, 116th Congress. https://www.congress.gov/bill/116th-congress/senate-bill/2314/all-info (2019).
Salganik, M. J. Bit by Bit: Social Research in the Digital Age (Princeton Univ. Press, 2019).
Sen, I., Flöck, F., Weller, K., Weiß, B., & Wagner, C. From the total survey error framework to an error framework for digital traces of humans: translation tutorial. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency https://doi.org/10.1145/3351095.3375669 (2020).
Dienlin, T. et al. An agenda for open science in communication. J. Commun. 71, 1–26 (2020).
Bishara, A. J. & Hittner, J. B. Reducing bias and error in the correlation coefficient due to nonnormality. Educ. Psycholog. Meas. 75, 785–804 (2015).
Lazer, D. M. et al. Computational social science: obstacles and opportunities. Science 369, 1060–1062 (2020).
Sewall, C. J., & Parry, D. A., What is the role of depression in the discrepancy between estimated and actual iPhone use? An exploratory analysis using cubic response surface analysis. Preprint at OSF Preprints https://doi.org/10.31219/osf.io/e9pfa (2021).
Van Smeden, M., Lash, T. L. & Groenwold, R. H. Reflection on modern methods: five myths about measurement error in epidemiological research. Int. J. Epidemiol. 49, 3444–347 (2020).
Kaye, L. K., Orben, A., Ellis, D. A., Hunger, S. C. & Houghton, S. The conceptual and methodological mayhem of ‘screen time’. Int. J. Environ. Res. Public Health 17, 3661 (2020).
Meier, A. & Reinecke, L. Computer-mediated communication, social media, and mental health: a conceptual and empirical meta-review. Commun. Res. https://doi.org/10.1177/0093650220958224 (2020).
Büchi, M. A proto-theory of digital well-being. Preprint at OSF Preprints https://doi.org/10.31219/osf.io/k3e2j (2020)
Harris, B., Regan, T., Schueler, J. & Fields, S. A. Problematic mobile phone and smartphone use scales: a systematic review. Front. Psychol. 11, 672 (2020).
Laconi, S., Rodgers, R. F. & Chabrol, H. The measurement of internet addiction: a critical review of existing scales and their psychometric properties. Comput. Hum. Behav. 41, 190–202 (2014).
Webster, J. & Watson, R. Analyzing the past to prepare for the future: writing a literature review. Manag. Inform. Syst. Q. 26, xiii–xxiii (2002).
Hedges, L. V., Tipton, E. & Johnson, M. C. Robust variance estimation in meta-regression with dependent effect size estimates. Res. Synth. Methods 1, 39–65 (2010).
Fisher, Z., & Tipton, E. Robumeta: an R-package for robust variance estimation in meta-analysis. Preprint at arXiv https://arxiv.org/abs/1503.02220 (2015).
Fu, R. et al. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the effective health care program. J. Clin. Epidemiol. 64, 1187–1197 (2011).
Gilpin, A. R. Table for conversion of Kendall’s tau to Spearman’s rho within the context of measures of magnitude of effect for meta-analysis. Educ. Psycholog. Meas. 53, 87–92 (1993).
Rupinski, M. T. & Dunlap, W. P. Approximating Pearson product–moment correlations from Kendall’s tau and Spearman’s rho. Educ. Psychol. Meas. 56, 419–429 (1996).
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. Introduction to Meta-Analysis (Wiley, 2011).
Cohen, J. Statistical Power Analysis for the Behavioral Sciences (Academic, 1988).
Friedrich, J. O., Adhikari, N. K. & Beyene, J. The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: a simulation study. BMC Med. Res. Method. 8, 32 (2008).
Lajeunesse, M. J. On the meta-analysis of response ratios for studies with correlated and multi-group designs. Ecology 92, 2049–2055 (2011).
Egger, M., Smith, G. D., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test. Br. Med. J. 315, 629–634 (1997).
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R. & Rushton, L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J. Clin. Epidemiol. 61, 991–996 (2008).
Tanner-Smith, E. E., Tipton, E. & Polanin, J. R. Handling complex meta-analytic data structures using robust variance estimates: a tutorial in R. J. Dev. Life-Course Criminol. 2, 85–112 (2016).
Pustejovsky, J. E. clubSandwich: cluster-robust (sandwich) variance estimators with small-sample corrections (R package version 0.4.2) (2017).
The authors thank the researchers who made their data available for inclusion in this review and especially those who completed extra analyses to provide relevant data and statistics. Additionally, the authors thank those who shared our public calls for relevant data and the valuable input provided by the reviewers of this paper. The authors received no specific funding for this work.
The author(s) declare that there are no conflicts of interest with respect to the authorship or the publication of this article.
Peer review information Nature Human Behaviour thanks Anna Chaimani and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Note: k: number of separate effect sizes included for the moderator level; r = Pearson correlation coefficient; F values correspond to the Approximate Hotelling-Zhang with small sample correction omnibus tests for moderators with more than two levels; 95% CI corresponds to the r values for individual moderator levels; p corresponds to the F value for moderators or the subgroup analysis for individual moderator levels. *This analysis did not include the adolescent population group as only two effect sizes were available. †This analysis did not include the other category as only a single effect size was available.
Note: k: number of separate effect sizes included for the moderator level; R = response ratio; Exp(𝛽) = exponential transformation of metaregression coefficient from a model in which a categorical moderator with two levels was entered as a predictor. F values correspond to the Approximate Hotelling-Zhang with small sample correction omnibus tests for moderators with more than two levels; 95% CI corresponds to the r values for individual moderator levels; p corresponds to the F value for moderators or the subgroup analysis for individual moderator levels. *This analysis did not include the adolescent population category, the general population category and the unknown population category as only two, one, and three effect sizes were available, respectively.
Note: k: number of included effect sizes. *: One study used both a built-in tool and a third-party tool.
About this article
Cite this article
Parry, D.A., Davidson, B.I., Sewall, C.J.R. et al. A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use. Nat Hum Behav (2021). https://doi.org/10.1038/s41562-021-01117-5