Crowdsourced analysis of clinical trial data to predict amyotrophic lateral sclerosis progression

  • Nature Biotechnology volume 33, pages 5157 (2015)
  • doi:10.1038/nbt.3051
  • Download Citation


Amyotrophic lateral sclerosis (ALS) is a fatal neurodegenerative disease with substantial heterogeneity in its clinical presentation. This makes diagnosis and effective treatment difficult, so better tools for estimating disease progression are needed. Here, we report results from the DREAM-Phil Bowen ALS Prediction Prize4Life challenge. In this crowdsourcing competition, competitors developed algorithms for the prediction of disease progression of 1,822 ALS patients from standardized, anonymized phase 2/3 clinical trials. The two best algorithms outperformed a method designed by the challenge organizers as well as predictions by ALS clinicians. We estimate that using both winning algorithms in future trial designs could reduce the required number of patients by at least 20%. The DREAM-Phil Bowen ALS Prediction Prize4Life challenge also identified several potential nonstandard predictors of disease progression including uric acid, creatinine and surprisingly, blood pressure, shedding light on ALS pathobiology. This analysis reveals the potential of a crowdsourcing competition that uses clinical trial data for accelerating ALS research and development.

  • Subscribe to Nature Biotechnology for full access:



Additional access options:

Already a subscriber?  Log in  now or  Register  for online access.


  1. 1.

    et al. The incidence of amyotrophic lateral sclerosis in Friuli Venezia Giulia, Italy, from 2002 to 2009: a retrospective population-based study. Neuroepidemiology 41, 54–61 (2013).

  2. 2.

    et al. Amyotrophic lateral sclerosis in an urban setting: a population based study of inner city London. J. Neurol. 253, 1642–1643 (2006).

  3. 3.

    et al. Amyotrophic lateral sclerosis. Lancet 377, 942–955 (2011).

  4. 4.

    , , & Riluzole for amyotrophic lateral sclerosis (ALS)/motor neuron disease (MND). Cochrane Database Syst. Rev. CD001447 (2007).

  5. 5.

    & Performance of the amyotrophic lateral sclerosis functional rating scale (ALSFRS) in multicenter clinical trials. J. Neurol. Sci. 152, S1–S9 (1997).

  6. 6.

    et al. The ALSFRS-R: a revised ALS functional rating scale that incorporates assessments of respiratory function. J. Neurol. Sci. 169, 13–21 (1999).

  7. 7.

    et al. Alzheimer's disease: diagnostics, prognostics and the road to prevention. EPMA J. 1, 293–303 (2010).

  8. 8.

    et al. A predictive model for progression of chronic kidney disease to kidney failure. J. Am. Med. Assoc. 305, 1553–1559 (2011).

  9. 9.

    et al. Development of a multiple sclerosis functional composite as a clinical trial outcome measure. Brain 122, 871–882 (1999).

  10. 10.

    et al. The coalition against major diseases: developing tools for an integrated drug development process for Alzheimer's and Parkinson's diseases. Clin. Pharmacol. Ther. 86, 365–367 (2009).

  11. 11.

    , , & Revisiting the multiple sclerosis functional composite: proceedings from the national multiple sclerosis society (NMSS) task force on clinical disability measures. Mult. Scler. 18, 1074–1080 (2012).

  12. 12.

    et al. Combining patient-level and summary-level data for Alzheimer's disease modeling and simulation: a beta regression meta-analysis. J. Pharmacokinet. Pharmacodyn. 39, 479–498 (2012).

  13. 13.

    et al. Wisdom of crowds for robust gene network inference. Nat. Methods 9, 796–804 (2012).

  14. 14.

    R Core Team. R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, Austria, 2013).

  15. 15.

    & RandomForest4Life: a random forest for predicting ALS disease progression. Amyotroph. Lateral Scler. Frontotemporal Degener. 15, 444–452 (2014).

  16. 16.

    & Seeking the wisdom of crowds through challenge-based competitions in biomedical research. Clin. Pharmacol. Ther. 93, 396–398 (2013).

  17. 17.

    et al. Disease progression in amyotrophic lateral sclerosis: predictors of survival. Muscle Nerve 25, 709–714 (2002).

  18. 18.

    , , , & Prognosis in amyotrophic lateral sclerosis: a population-based study. Neurology 60, 813–819 (2003).

  19. 19.

    , , , & Body mass index, not dyslipidemia, is an independent predictor of survival in amyotrophic lateral sclerosis. Muscle Nerve 44, 20–24 (2011).

  20. 20.

    et al. Uric acid levels predict survival in men with amyotrophic lateral sclerosis. J. Neurol. 259, 1923–1928 (2012).

  21. 21.

    & The pooled resource open-access, ALSCTC amyotrophic lateral sclerosis disease progression model. Amyotroph. Lateral Scler. Frontotemporal Degener. 15, 119–129 (2014).

  22. 22.

    et al. Prize-based contests can provide solutions to computational biology problems. Nat. Biotechnol. 31, 108–111 (2013).

  23. 23.

    et al. ALS clinical trials: do enrolled patients accurately represent the ALS population? Neurology 77, 1432–1437 (2011).

  24. 24.

    , , & Predictability of disease progression in amyotrophic lateral sclerosis. Muscle Nerve 34, 702–708 (2006).

  25. 25.

    et al. Factors associated with survival in the national registry of veterans with ALS. Amyotroph. Lateral Scler. 10, 332–338 (2009).

  26. 26.

    et al. Analysis of factors that modify susceptibility and rate of progression in amyotrophic lateral sclerosis (ALS). Amyotroph. Lateral Scler. 7, 173–182 (2006).

  27. 27.

    et al. Onset and spreading patterns of lower motor neuron involvements predict survival in sporadic amyotrophic lateral sclerosis. J. Neurol. Neurosurg. Psychiatry 82, 1244–1249 (2011).

  28. 28.

    et al. Medications and laboratory parameters as prognostic factors in amyotrophic lateral sclerosis. Amyotroph. Lateral Scler. 9, 369–374 (2008).

  29. 29.

    et al. Analysis of survival and prognostic factors in amyotrophic lateral sclerosis: a population based study. J. Neurol. Neurosurg. Psychiatry 79, 33–37 (2008).

  30. 30.

    et al. Pattern of spread and prognosis in lower limb-onset ALS. Amyotroph. Lateral Scler. 11, 369–373 (2010).

  31. 31.

    et al. ALSFRS-R score and its ratio: a useful predictor for ALS-progression. J. Neurol. Sci. 275, 69–73 (2008).

  32. 32.

    , , , & Functional outcome measures as clinical trial endpoints in ALS. Neurology 63, 1933–1935 (2004).

  33. 33.

    , , , & Respiratory systems abnormalities and clinical milestones for patients with amyotrophic lateral sclerosis with emphasis upon survival. Amyotroph. Lateral Scler. 8, 36–41 (2007).

  34. 34.

    , , , & What does body mass index measure in amyotrophic lateral sclerosis and why should we care? Muscle Nerve 45, 612 (2012).

  35. 35.

    et al. Body mass index (BMI) as predictor of ALSFRS-R score decline in ALS patients. Amyotroph. Lateral Scler. Frontotemporal Degener. 14, 212–216 (2013).

  36. 36.

    et al. Urate as a predictor of the rate of clinical decline in Parkinson disease. Arch. Neurol. 66, 1460–1468 (2009).

  37. 37.

    , & The relationship between uric acid levels and Huntington's disease progression. Mov. Disord. 25, 224–228 (2010).

  38. 38.

    , , & Serum uric acid levels and the risk of Parkinson disease. Ann. Neurol. 58, 797–800 (2005).

  39. 39.

    , , & Serum uric acid and cognitive function and dementia. Brain 132, 377–382 (2009).

  40. 40.

    , , , & Relationships between disease progression and serum levels of lipid, urate, creatinine and ferritin in Japanese patients with amyotrophic lateral sclerosis: a cross-sectional study. Intern. Med. 51, 1501–1508 (2012).

  41. 41.

    et al. An exploratory study of serum creatinine levels in patients with amyotrophic lateral sclerosis. Neurol. Sci. 35, 1591–1597 (2014).

  42. 42.

    & Serum creatine kinase levels in spinobulbar muscular atrophy and amyotrophic lateral sclerosis. Muscle Nerve 40, 126–129 (2009).

  43. 43.

    , , , & Elevated creatine kinase suggests better prognosis in patients with amyotrophic lateral sclerosis. J. Neurol. Neurosurg. Psychiatry 84, e2 (2013).

  44. 44.

    , , , & An abnormal relationship between blood pressure and pulse rate in amyotrophic lateral sclerosis. Acta Neurol. Scand. 96, 118–122 (1997).

  45. 45.

    , & Autonomic impairment in amyotrophic lateral sclerosis. Curr. Opin. Neurol. 18, 487–493 (2005).

  46. 46.

    , , & Blood pressure measurements in a transgenic SOD1–G93A mouse model of amyotrophic lateral sclerosis. Amyotroph. Lateral Scler. 13, 509–513 (2012).

  47. 47.

    et al. Impairment of cardiac autonomic control in patients with amyotrophic lateral sclerosis. Amyotroph. Lateral Scler. 11, 272–276 (2010).

  48. 48.

    et al. Phase II screening trial of lithium carbonate in amyotrophic lateral sclerosis: examining a more efficient trial design. Neurology 77, 973–979 (2011).

  49. 49.

    et al. The PRO-ACT Database: Design, initial analyses, and predictive features. Neurology 10.1212/WNL.0000000000000951 (8 October 2014).

Download references


We are grateful to the following people for their important assistance with this manuscript: the pharmaceutical companies which provided data to the PRO-ACT data set that enabled this entire endeavor, the Bowen family and D. Lautman for their generous support for this project, R. Betensky from the Harvard School of Public Health for her statistical advice and contributions to the earliest stage of developing the challenge, L. Reinhold from InnoCentive for her support and management of the challenge, our challenge sponsors, Nature, The Economist and Popular Science, the clinicians who participated in the clinicians' assessment and follow-on discussions about the results, L.A. White and D. Kerr from Biogen Idec for their assistance with estimating the financial impact of the algorithms and, of course, the solvers who participated in the challenge and the patients who inspired this effort.

Author information

Author notes

    • Robert Küffner
    •  & Neta Zach

    These authors contributed equally to this work.


  1. Institute of Bioinformatics and Systems Biology, German Research Center for Environmental Health, Munich, Germany.

    • Robert Küffner
  2. Department of Informatics, Ludwig-Maximilians-University, Munich, Germany.

    • Robert Küffner
    •  & Johann Hawe
  3. Prize4Life, Tel Aviv, Israel and Cambridge, Massachusetts, USA.

    • Neta Zach
    •  & Melanie L Leitner
  4. IBM T.J. Watson Research Center, Yorktown Heights, New York, USA.

    • Raquel Norel
    •  & Gustavo Stolovitzky
  5. MGH Biostatistics Center, Massachusetts General Hospital, Boston, Massachusetts, USA.

    • David Schoenfeld
  6. Harvard Medical School, Charlestown, Massachusetts, USA.

    • David Schoenfeld
  7. Sentrana Inc., Washington, DC, USA.

    • Liuxia Wang
    •  & Guang Li
  8. Latham&Watkins LLP, Silicon Valley, California, USA.

    • Lilly Fang
  9. Department of Statistics, Stanford University, Stanford, California, USA.

    • Lester Mackey
  10. Department of Neuroscience, Beaumont Hospital and Trinity College Dublin, Dublin, Ireland.

    • Orla Hardiman
  11. Neurological Clinical Research Institute, Massachusetts General Hospital, Charlestown, Massachusetts, USA.

    • Merit Cudkowicz
    •  & Alexander Sherman
  12. Luxembourg Centre for Systems Biomedicine, University of Luxembourg, Esch/Alzette, Luxembourg.

    • Gokhan Ertaylan
  13. Max Planck Institute for Intelligent Systems, Tübingen, Germany.

    • Moritz Grosse-Wentrup
    • , Timm Meyer
    •  & Bernhard Schölkopf
  14. Institute of Social- and Preventive Medicine, University of Zürich, Zürich, Switzerland.

    • Torsten Hothorn
  15. Orca XL Problem Solvers, Amsterdam, the Netherlands.

    • Jules van Ligtenberg
    •  & Rubio Vaughan
  16. Max Planck Institute for Biological Cybernetics and Bernstein Center for Computational Neuroscience, Tübingen, Germany.

    • Jakob H Macke
  17. Berkeley School of Public Health, University of California, Berkeley, California, USA.

    • Linh Tran
  18. ALS Innovation Hub, Biogen Idec, Cambridge, Massachusetts, USA.

    • Melanie L Leitner


  1. Search for Robert Küffner in:

  2. Search for Neta Zach in:

  3. Search for Raquel Norel in:

  4. Search for Johann Hawe in:

  5. Search for David Schoenfeld in:

  6. Search for Liuxia Wang in:

  7. Search for Guang Li in:

  8. Search for Lilly Fang in:

  9. Search for Lester Mackey in:

  10. Search for Orla Hardiman in:

  11. Search for Merit Cudkowicz in:

  12. Search for Alexander Sherman in:

  13. Search for Gokhan Ertaylan in:

  14. Search for Moritz Grosse-Wentrup in:

  15. Search for Torsten Hothorn in:

  16. Search for Jules van Ligtenberg in:

  17. Search for Jakob H Macke in:

  18. Search for Timm Meyer in:

  19. Search for Bernhard Schölkopf in:

  20. Search for Linh Tran in:

  21. Search for Rubio Vaughan in:

  22. Search for Gustavo Stolovitzky in:

  23. Search for Melanie L Leitner in:


R.K., N.Z., R.N., J.H., D.S., O.H., M.C., G.S. and M.L.L. designed the challenge. R.K. and J.H. prepared the data and baseline algorithm, A.S. helped with data preparation. N.Z. managed the challenge. L.W., G.L., L.F., L.M., G.E., M.G.-W., T.H., J.v.L., J.H.M., T.M., B.S., L.T. and R.V. submitted algorithms. D.S., L.W., G.L., L.F. and L.M. contributed further analysis on challenge performance. R.K. and N.Z. analyzed the results and wrote the paper.

Competing interests

R.N. and G.S. are employees of IBM; L.F. is an employee of Latham&Watkins; R.V. is an employee of Orca XL Problem Solvers; M.L.L. is an employee of Biogen Idec; L.W. and G.L. are employees of Sentrana.

Corresponding authors

Correspondence to Robert Küffner or Neta Zach.

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–13, Supplementary Tables 1–10, Supplementary Notes 1–3, Supplementary Results 1–5 and Supplementary Data 1

Zip files

  1. 1.

    Supplementary Software

    Algorithms participating

  2. 2.

    Supplementary Predictions

    Predictions made by algorithms participating