Remote health assessments that gather real-world data (RWD) outside clinic settings require a clear understanding of appropriate methods for data collection, quality assessment, analysis and interpretation. Here we examine the performance and limitations of smartphones in collecting RWD in the remote mPower observational study of Parkinson’s disease (PD). Within the first 6 months of study commencement, 960 participants had enrolled and performed at least five self-administered active PD symptom assessments (speeded tapping, gait/balance, phonation or memory). Task performance, especially speeded tapping, was predictive of self-reported PD status (area under the receiver operating characteristic curve (AUC) = 0.8) and correlated with in-clinic evaluation of disease severity (r = 0.71; P < 1.8 × 10−6) when compared with motor Movement Disorder Society-Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). Although remote assessment requires careful consideration for accurate interpretation of RWD, our results support the use of smartphones and wearables in objective and personalized disease assessments.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Real-world behavioral dataset from two fully remote smartphone-based randomized clinical trials for depression
Scientific Data Open Access 27 August 2022
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
The code, a docker container with all installed packages and a snakemake script that reproduces all of the figure and analysis, is available in GitHub: https://github.com/Sage-Bionetworks/mPowerRerun.
Sherman, R. E. et al. Real-world evidence—what is it and what can it tell us? N. Engl. J. Med. 375, 2293–2297 (2016).
Steinhubl, S. R. et al. Effect of a home-based wearable continuous ECG monitoring patch on detection of undiagnosed atrial fibrillation: the mSToPS randomized clinical trial. JAMA 320, 146–155 (2018).
Juvenile Diabetes Research Foundation Continuous Glucose Monitoring Study Group. Effectiveness of continuous glucose monitoring in a clinical care environment: evidence from the Juvenile Diabetes Research Foundation continuous glucose monitoring (JDRF-CGM) trial. Diabetes Care 33, 17–22 (2010).
Anguera, J. A., Jordan, J. T., Castaneda, D., Gazzaley, A. & Areán, P. A. Conducting a fully mobile and randomised clinical trial for depression: access, engagement and expense. BMJ Innov. 2, 14–21 (2016).
Quer, G. et al. Home monitoring of blood pressure: short-term changes during serial measurements for 56398 subjects. IEEE J. Biomed. Health Inform. 22, 1691–1698 (2018).
Chaibub Neto, E. et al. Learning Disease vs Participant Signatures: a permutation test approach to detect identity confounding in machine learning diagnostic applications. Preprint at https://arxiv.org/abs/1712.03120 (2017).
Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C. & Kording, K. P. The need to approximate the use-case in clinical machine learning. Gigascience 6, 1–9 (2017).
Dorsey, E. R. et al. The use of smartphones for health research. Acad. Med. 92, 157–160 (2017).
Pratap, A. et al. Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants. NPJ Digit. Med. 3, 21 (2020).
Arora, S. et al. Detecting and monitoring the symptoms of Parkinson’s disease using smartphones: a pilot study. Parkinsonism Relat. Disord. 21, 650–653 (2015).
Espay, A. J. et al. Technology in Parkinson’s disease: challenges and opportunities. Mov. Disord. 31, 1272–1282 (2016).
Ellis, R. J. et al. A validated smartphone-based assessment of gait and gait variability in Parkinson’s disease. PLoS ONE 10, e0141694 (2015).
Kostikis, N., Hristu-Varsakelis, D., Arnaoutoglou, M. & Kotsavasiloglou, C. A smartphone-based tool for assessing Parkinsonian hand tremor. IEEE J. Biomed. Health Inform. 19, 1835–1842 (2015).
Goetz, C. G. et al. Movement Disorder Society–sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS): scale presentation and clinimetric testing results. Mov. Disord. 23, 2129–2170 (2008).
Hauser, R. A. et al. A home diary to assess functional status in patients with Parkinson’s disease with motor fluctuations and dyskinesia. Clin. Neuropharmacol. 23, 75–81 (2000).
Heldman, D. A. et al. The modified bradykinesia rating scale for Parkinson’s disease: reliability and comparison with kinematic measures. Mov. Disord. 26, 1859–1863 (2011).
Jeon, H. et al. Automatic classification of tremor severity in Parkinson’s disease using a wearable device. Sensors 17, 2067 (2017).
Stamate, C. et al. Deep learning Parkinson’s from smartphone data. in Proc. 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom) 31–40 (IEEE, 2017).
Liddle, J. et al. Measuring the lifespace of people with Parkinson’s disease using smartphones: proof of principle. JMIR Mhealth Uhealth 2, e13 (2014).
Ginis, P. et al. Feasibility and effects of home-based smartphone-delivered automated feedback training for gait in people with Parkinson’s disease: a pilot randomized controlled trial. Parkinsonism Relat. Disord. 22, 28–34 (2016).
Stamate, C. et al. The cloudUPDRS app: a medical device for the clinical assessment of Parkinson’s Disease. Pervasive Mob. Comput. 43, 146–166 (2018).
Lipsmeier, F. et al. Evaluation of smartphone-based testing to generate exploratory outcome measures in a phase 1 Parkinson’s disease clinical trial. Mov. Disord. 33, 1287–1297 (2018).
Zhan, A. et al. Using smartphones and machine learning to quantify Parkinson disease severity: The mobile Parkinson disease score. JAMA Neurol. 75, 876–880 (2018).
Bot, B. M. et al. The mPower study, Parkinson disease mobile data collected using ResearchKit. Sci. Data 3, 160011 (2016).
Chaibub Neto, E. et al. On the analysis of personalized medication response and classification of case vs control patients in mobile health studies: the mPower case study. Preprint at https://arxiv.org/abs/1706.09574 (2017).
Chaibub Neto, E., Tummalacherla, M., Mangravite, L. & Omberg, L. Causality-based tests to detect the influence of confounders on mobile health diagnostic applications: a comparison with restricted permutations. in Machine Learning for Health (ML4H) Workshop at NeurIPS 2019. https://arxiv.org/abs/1911.05139 (2019).
Little, M. A. et al. Using and understanding cross-validation strategies. Perspectives on Saeb et al. Gigascience 6, 1–6 (2017).
Neto, E. C. et al. Detecting the impact of subject characteristics on machine learning-based diagnostic applications. NPJ Digit. Med. 2, 1–6 (2019).
Sakar, B. E. et al. Collection and analysis of a Parkinson speech dataset with multiple types of sound recordings. IEEE J. Biomed. Health Inform. 17, 828–834 (2013).
Mishra, S. R. et al. Supporting coping with Parkinson’s disease through self-tracking. in Proc. 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19) (ACM) 1–16 (2019).
Newey, W. K. & West, K. D. A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55, 703 (1987).
Newey, W. K. & West, K. D. Automatic lag selection in covariance matrix estimation. Rev. Econ. Stud. 61, 631–653 (1994).
Grömping, U. Estimators of relative importance in linear regression based on variance decomposition. Am. Stat. 61, 139–147 (2007).
Thenganatt, M. A. & Jankovic, J. Parkinson disease subtypes. JAMA Neurol. 71, 499 (2014).
Khazaal, Y. et al. Does self-selection affect samples’ representativeness in online surveys? An investigation in online video game research. J. Med. Internet Res. 16, e164 (2014).
Bethlehem, J. Selection bias in web surveys. Int. Stat. Rev. 78, 161–188 (2010).
Badawy, R. et al. Automated quality control for sensor based symptom measurement performed outside the lab. Sensors 18, 1215 (2018).
Chaibub Neto, E. et al. Towards personalized causal inference of medication response in mobile health: an instrumental variable approach for randomized trials with imperfect compliance. Preprint at https://arxiv.org/abs/1604.01055 (2016).
Barnett, I., Torous, J., Staples, P., Keshavan, M. & Onnela, J.-P. Beyond smartphones and sensors: choosing appropriate statistical methods for the analysis of longitudinal data. J. Am. Med. Inform. Assoc. 25, 1669–1674 (2018).
Marbach, D. et al. Wisdom of crowds for robust gene network inference. Nat. Methods 9, 796–804 (2012).
Tavares, A. L. T. et al. Quantitative measurements of alternating finger tapping in Parkinson’s disease correlate with UPDRS motor disability and reveal the improvement in fine motor control from medication and deep brain stimulation. Mov. Disord. 20, 1286–1298 (2005).
Tsanas, A., Little, M. A., McSharry, P. E. & Ramig, L. O. Nonlinear speech analysis algorithms mapped to a standard metric achieve clinically useful quantification of average Parkinson’s disease symptom severity. J. R. Soc. Interface 8, 842–855 (2011).
Sejdic, E., Lowry, K. A., Bellanca, J., Redfern, M. S. & Brach, J. S. A comprehensive assessment of gait accelerometry signals in time, frequency and time-frequency domains. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 603–612 (2014).
Sieberts, S. K. et al. Crowdsourcing digital health measures to predict Parkinson’s disease severity: the Parkinson’s Disease Digital Biomarker DREAM Challenge. NPJ Digit. Med. https://doi.org/10.1038/s41746-021-00414-7 (2021).
Doerr, M. et al. Formative evaluation of participant experience with mobile econsent in the app-mediated Parkinson mPower study: a mixed methods study. JMIR Mhealth Uhealth 5, e14 (2017).
Gill, D. J., Freshman, A., Blender, J. A. & Ravina, B. The Montreal cognitive assessment as a screening tool for cognitive impairment in Parkinson’s disease. Mov. Disord. 23, 1043–1046 (2008).
Weintraub, D., Oehlberg, K. A., Katz, I. R. & Stern, M. B. Test characteristics of the 15-item geriatric depression scale and Hamilton depression rating scale in Parkinson disease. Am. J. Geriatr. Psychiatry 14, 169–175 (2006).
Schwab, R.S. & England, A.C. Projection technique for evaluating surgery in Parkinson’s disease. in Third Symposium on Parkinson’s Disease (eds Gillingham, F. J. & Danoldson, I. M. L.) 152–157 (E & S. Livingston, 1969).
McRae, C., Diem, G., Vo, A., O’Brien, C. & Seeberger, L. Schwab & England: standardization of administration. Mov. Disord. 15, 335–336 (2000).
R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/ (2014).
Taylor, A. L. T. et al. Quantitative measurements of alternating finger tapping in Parkinsons disease correlate with UPDRS motor disability and reveal the improvement in fine motor control from medication and deep brain stimulation. Mov. Disord. 20, 1286–1298 (2005).
Ho, D., Imai, K., King, G. & Stuart, E. A. MatchIt: non-parametric preprocessing for parametric causal inference. J. Stat. Softw. 42, 1–28 (2011).
Rousseeuw, P. J. & Croux, C. Alternatives to the median absolute deviation. J. Am. Stat. Assoc. 88, 1273–1283 (1993).
Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).
Liaw, A. & Wiener, M. Classification and regression by randomForest. R News 2, 18–22 (2002).
Hastie, T, Tibshirani, R & Friedman, J. The Elements of Statistical Learning (Springer, 2001).
Friedman, J., Hastie, T. & Tibshirani, R. Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33, 1–22 (2010).
Pearl, J. Causality: Models, Reasoning, and Inference (Cambridge University Press, 2009).
Pearl, J. & Mackenzie, D. The Book of Why: the New Science of Cause and Effect (Basic Books, 2018). .
Szekely, G., Rizzo, M. L. & Bakirov, N. K. Measuring and testing dependence by correlation of distances. Ann. Stat. 35, 2769–2794 (2007).
Szekely, G. & Rizzo, M. L. Partial distance correlation with methods for dissimilarities. Ann. Stat. 42, 2382–2412 (2014).
Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C. & Kording, K. P. The need to approximate the use-case in clinical machine learning. GigaScience 6, 1–9 (2017).
Chaibub Neto, E. et al. Detecting the impact of subject characteristics on machine learning-based diagnostic applications. NPJ Digit. Med. 2, 99 (2019).
Sarkar, B. E. et al. Collection and analysis of a Parkinson speech dataset with multiple types of sound recordings. IEEE J. Biomed. Health Inform. 17, 828–834 (2013).
Bamber, D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J. Math. Psychol. 12, 387415 (1975).
Mason, S. L. & Graham, N. E. Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: statistical significance and interpretation. Q. J. R. Meteorolog. Soc. 128, 2145–2166 (2002).
Newey, W. K. & West, K. D. A simple, positive-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55, 703708 (1987).
Zeileis, A. Econometric computing with HC and HAC covariance matrix estimation. J. Stat. Softw. 10, 1–17 (2004).
Box, G., Jenkins, G. M. & Reinsel, G. C. Time Series Analysis: Forecasting and Control 3rd edn (Prentice-Hall, 1994).
Hyndman, R. J. & Khandakar, Y. Automatic time series forecasting: the forecast package for R. J. Stat. Softw. 26, 1–22 (2008).
Datta, D. D. & Du W. Nonparametric HAC estimation for time series data with missing observations. International Finance Discussion Papers. The Federal Reserve Board (2012).
Rho, S. H. & Vogelsang, T. J. Heteroskedasticity autocorrelation robust inference in time series regressions with missing data. Econometric Theory 35, 601–629 (2019).
McGregor, J. R. & Babb, J. C. Serially correlated differences in the paired comparison of time series. Biometrika 76, 735–739 (1989).
Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).
Lindeman, R. H., Merenda, P. F., and Gold, R. Z. Introduction to Bivariate and Multivariate Analysis (Scott, Foresman, 1980).
Kruskal, W. Relative importance by averaging over orderings. Am. Stat. 41, 6–10 (1987).
Gromping, U. Relative importance for linear regression in R: the package relaimpo. J. Stat. Softw. 17, 1–27 (2007).
This work was funded through a grant from the Robert Wood Johnson Foundation. These data were contributed by users of the Parkinson mPower mobile application as part of the mPower study developed by Sage Bionetworks and described in Synapse (https://doi.org/10.7303/syn4993293).
B.R.B. currently serves as Editor-in-Chief for the Journal of Parkinson’s Disease, serves on the editorial boards of Practical Neurology and Digital Biomarkers, has received honoraria from serving on the scientific advisory board for Zambon, Biogen, UCB and Walk with Path, has received fees for speaking at conferences from AbbVie, Zambon, Roche, GE Healthcare and Bial, and has received research support from the Netherlands Organization for Scientific Research, the Michael J. Fox Foundation, UCB, Abbvie, Zambon, the Stichting Parkinson Fonds, the Hersenstichting Nederland, the Parkinson’s Foundation, Verily Life Sciences, Horizon 2020, the Topsector Life Sciences and Health and the Parkinson Vereniging. E.R.D. has received honoraria for speaking at American Academy of Neurology courses, American Neurological Association and University of Michigan; received compensation for consulting services from 23andMe, Abbott, Abbvie, American Well, Biogen, BrainNeuroBio, Clintrex, Curasen Therapeutics, DeciBio, Denali Therapeutics, GlaxoSmithKline, Grand Rounds, Karger, Lundbeck, MC10, MedAvante, Medical-legal services, Mednick Associates, National Institute of Neurological Disorders and Stroke, Olson Research Group, Optio, Origent Data Sciences, Inc., Otsuka, Prilenia, Putnam Associates, Roche, Sanofi, Shire, Spark, Sunovion Pharma, Teva, Theravance, UCB and Voyager Therapeutics; research support from Abbvie, Acadia Pharmaceuticals, AMC Health, Biosensics, Burroughs Wellcome Fund, Davis Phinney Foundation, Duke University, Food and Drug Administration, GlaxoSmithKline, Greater Rochester Health Foundation, Huntington Study Group, Michael J. Fox Foundation, the NIH/NINDS, NSF, Nuredis Pharmaceuticals, Patient-Centered Outcomes Research Institute, Pfizer, Prana Biotechnology, Raptor Pharmaceuticals, Roche, Safra Foundation, Teva Pharmaceuticals and University of California Irvine; editorial services for Karger Publications; and ownership interests with Grand Rounds (second opinion service). C.M.T. reports grants from Sage Biometrics, during the conducting of the study; grants from Parkinson Foundation, grants from Gateway LLC, grants from Roche/Genentech, grants from Parkinson Study Group, personal fees from Biogen Idec, personal fees from Acorda, personal fees from Adamas Therapeutics, personal fees from Amneal, personal fees from CNS Ratings, personal fees from Grey Matter LLC, grants from Michael J. Fox Foundation, grants from NIH/NIA, grants from NIH/NINDS, grants from VA Merit, grants from Department of Defense, personal fees from Northwestern University, personal fees from Partners, Harvard University, nonfinancial support from Medtronic, Inc., nonfinancial support from Acadia, nonfinancial support from Boston Scientific, nonfinancial support from Neurocrine, nonfinancial support from Acadia, grants from Biogen Idec research support, nonfinancial support from Biogen Idec, personal fees from Guidemark Health, personal fees from Acadia, personal fees from Neurocrine, personal fees from Lundbeck, personal fees from Cadent, nonfinancial support from Neurocrine and grants from Roche Genentech, outside the submitted work.
Peer review information Nature Biotechnology thanks the anonymous reviewers for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Omberg, L., Chaibub Neto, E., Perumal, T.M. et al. Remote smartphone monitoring of Parkinson’s disease and individual response to therapy. Nat Biotechnol 40, 480–487 (2022). https://doi.org/10.1038/s41587-021-00974-9
This article is cited by
Real-world behavioral dataset from two fully remote smartphone-based randomized clinical trials for depression
Scientific Data (2022)
Nature Reviews Neurology (2021)