The early prediction of deterioration could have an important role in supporting healthcare professionals, as an estimated 11% of deaths in hospital follow a failure to promptly recognize and treat deteriorating patients1. To achieve this goal requires predictions of patient risk that are continuously updated and accurate, and delivered at an individual level with sufficient context and enough time to act. Here we develop a deep learning approach for the continuous risk prediction of future deterioration in patients, building on recent work that models adverse events from electronic health records2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17 and using acute kidney injury—a common and potentially life-threatening condition18—as an exemplar. Our model was developed on a large, longitudinal dataset of electronic health records that cover diverse clinical environments, comprising 703,782 adult patients across 172 inpatient and 1,062 outpatient sites. Our model predicts 55.8% of all inpatient episodes of acute kidney injury, and 90.2% of all acute kidney injuries that required subsequent administration of dialysis, with a lead time of up to 48 h and a ratio of 2 false alerts for every true alert. In addition to predicting future acute kidney injury, our model provides confidence assessments and a list of the clinical features that are most salient to each prediction, alongside predicted future trajectories for clinically relevant blood tests9. Although the recognition and prompt treatment of acute kidney injury is known to be challenging, our approach may offer opportunities for identifying patients at risk within a time window that enables early treatment.
Access optionsAccess options
Subscribe to Journal
Get full journal access for 1 year
only $3.90 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The clinical data used for the training, validation and test sets were collected at the US Department of Veterans Affairs and transferred to a secure data centre with strict access controls in de-identified format. Data were used with both local and national permissions. It is not publicly available and restrictions apply to its use. The de-identified dataset (or a test subset) may be available from the US Department of Veterans Affairs, subject to local and national ethical approvals.
We make use of several open-source libraries to conduct our experiments: the machine learning framework TensorFlow (https://github.com/tensorflow/tensorflow) along with the TensorFlow library Sonnet (https://github.com/deepmind/sonnet), which provides implementations of individual model components58. Our experimental framework makes use of proprietary libraries and we are unable to publicly release this code. We detail the experiments and implementation details in the Methods and Supplementary Information to allow for independent replication.
Thomson, R., Luettel, D., Healey, F. & Scobie, S. Safer Care for the Acutely Ill Patient: Learning from Serious Incidents (National Patient Safety Agency, 2007).
Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWscore) for septic shock. Sci. Transl. Med. 7, 299ra122 (2015).
Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. npj Digit. Med. 1, 18 (2018).
Koyner, J. L., Adhikari, R., Edelson, D. P. & Churpek, M. M. Development of a multicenter ward-based AKI prediction model. Clin. J. Am. Soc. Nephrol. 11, 1935–1943 (2016).
Cheng, P., Waitman, L. R., Hu, Y. & Liu, M. Predicting inpatient acute kidney injury over different time horizons: how early and accurate? In AMIA Annual Symposium Proceedings 565 (American Medical Informatics Association, 2017).
Koyner, J. L., Carey, K. A., Edelson, D. P. & Churpek, M. M. The development of a machine learning inpatient acute kidney injury prediction model. Crit. Care Med. 46, 1070–1077 (2018).
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C. & Faisal, A. A. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat. Med. 24, 1716–1720 (2018).
Avati, A. et al. Improving palliative care with deep learning. In 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 311–316 (2017).
Lim, B. & van der Schaar, M. Disease-Atlas: navigating disease trajectories with deep learning. Proc. Mach. Learn. Res. 85, 137–160 (2018).
Futoma, J., Hariharan, S. & Heller, K. A. Learning to detect sepsis with a multitask Gaussian process RNN classifier. In Proc. International Conference on Machine Learning (eds Precup, D. & Teh, Y. W.) 1174–1182 (2017).
Miotto, R., Li, L., Kidd, B. A. & Dudley, J. T. Deep Patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci. Rep. 6, 26094 (2016).
Lipton, Z. C., Kale, D. C., Elkan, C. & Wetzel, R. Learning to diagnose with LSTM recurrent neural networks. Preprint at https://arxiv.org/abs/1511.03677 (2016).
Cheng, Y. P. Z. J. H. & Wang, F. Risk prediction with electronic health records: a deep learning approach. In Proc. SIAM International Conference on Data Mining (eds Venkatasubramanian, S. C. & Meria, W.) 432–440 (2016).
Soleimani, H., Subbaswamy, A. & Saria, S. Treatment-response models for counterfactual reasoning with continuous-time, continuous-valued interventions. In Proc. 33rd Conference on Uncertainty in Artificial Intelligence (AUAI Press Corvallis, 2017).
Alaa, A. M., Yoon, J., Hu, S. & van der Schaar, M. Personalized risk scoring for critical care prognosis using mixtures of Gaussian process experts. IEEE Trans. Biomed. Eng. 65, 207–218 (2018).
Perotte, A., Ranganath, R., Hirsch, J. S., Blei, D. & Elhadad, N. Risk prediction for chronic kidney disease progression using heterogeneous electronic health record data and time series analysis. J. Am. Med. Inform. Assoc. 22, 872–880 (2015).
Bihorac, A. et al. MySurgeryRisk: development and validation of a machine-learning risk algorithm for major complications and death after surgery. Ann. Surg. 269, 652–662 (2019).
Khwaja, A. KDIGO clinical practice guidelines for acute kidney injury. Nephron Clin. Pract. 120, c179–c184 (2012).
Stenhouse, C., Coates, S., Tivey, M., Allsop, P. & Parker, T. Prospective evaluation of a modified early warning score to aid earlier detection of patients developing critical illness on a general surgical ward. Br. J. Anaesth. 84, 663P (2000).
Alge, J. L. & Arthur, J. M. Biomarkers of AKI: a review of mechanistic relevance and potential therapeutic implications. Clin. J. Am. Soc. Nephrol. 10, 147–155 (2015).
Wang, H. E., Muntner, P., Chertow, G. M. & Warnock, D. G. Acute kidney injury and mortality in hospitalized patients. Am. J. Nephrol. 35, 349–355 (2012).
MacLeod, A. NCEPOD report on acute kidney injury—must do better. Lancet 374, 1405–1406 (2009).
Lachance, P. et al. Association between e-alert implementation for detection of acute kidney injury and outcomes: a systematic review. Nephrol. Dial. Transplant. 32, 265–272 (2017).
Johnson, A. E. W. et al. Machine learning and decision support in critical care. Proc. IEEE Inst. Electr. Electron Eng. 104, 444–466 (2016).
Mohamadlou, H. et al. Prediction of acute kidney injury with a machine learning algorithm using electronic health record data. Can. J. Kidney Health Dis. 5, 1–9 (2018).
Pan, Z. et al. A self-correcting deep learning approach to predict acute conditions in critical care. Preprint at https://arxiv.org/abs/1901.04364 (2019).
Park, S. et al. Impact of electronic acute kidney injury (AKI) alerts with automated nephrologist consultation on detection and severity of AKI: a quality improvement study. Am. J. Kidney Dis. 71, 9–19 (2018).
Chen, I., Johansson, F. D. & Sontag, D. Why is my classifier discriminatory? Preprint at https://arxiv.org/abs/1805.12002 (2018).
Schulam, P. & Saria, S. Reliable decision support using counterfactual models. In Advances in Neural Information Processing Systems 30 (eds Guyon, I. et al.) 1697–1708 (2017).
Telenti, A., Steinhubl, S. R. & Topol, E. J. Rethinking the medical record. Lancet 391, 1013 (2018).
Department of Veterans Affairs. Veterans Health Administration: Providing Health Care for Veterans. https://www.va.gov/health/ (accessed 9 November 2018).
Razavian, N. & Sontag, D. Temporal convolutional neural networks for diagnosis from lab tests. In 4th Int. Conf. Learn. Representations (2016).
Zadrozny, B. & Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proc. 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (eds, Zaïane, O. R. et al.) 694–699 (ACM, 2002).
Zilly, J. G., Srivastava, R. K., Koutník, J. & Schmidhuber, J. Recurrent highway networks. In Proc. International Conference on Machine Learning (vol. 70) (eds Precup, D. & Teh, Y. W.) 4189–4198 (2017).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Collins, J., Sohl-Dickstein, J. & Sussillo, D. Capacity and trainability in recurrent neural networks. In International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) https://openreview.net/forum?id=BydARw9ex (2017).
Bradbury, J., Merity, S., Xiong, C. & Socher, R. Quasi-recurrent neural networks. In International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) https://openreview.net/forum?id=H1zJ-v5xl (2017).
Lei, T. & Zhang, Y. Training RNNs as fast as CNNs. Preprint at https://arxiv.org/abs/1709.02755v1 (2017).
Chung, J., Gulcehre, C., Cho, K. & Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modelling. Preprint at https://arxiv.org/abs/1412.3555 (2014).
Graves, A., Wayne, G. & Danihelka, I. Neural Turing machines. Preprint at https://arxiv.org/abs/1410.5401 (2014).
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D. & Lillicrap, T. Meta-learning with memory-augmented neural networks. In Proc. International Conference on Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 1842–1850 (2016).
Graves, A. et al. Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016).
Santoro, A. et al. Relational recurrent neural networks. In Advances in Neural Information Processing Systems 31 (eds Bengio, S. et al.) 7310–7321 (2018).
Caruana, R., Baluja, S. & Mitchell, T. in Advances in Neural Information Processing Systems (eds Mozer, M. et al.) 959–965 (1996).
Wiens, J., Guttag, J. & Horvitz, E. Patient risk stratification with time-varying parameters: a multitask learning approach. J. Mach. Learn. Res. 17, 1–23 (2016).
Ding, D. Y. et al. The effectiveness of multitask learning for phenotyping with electronic health records data. Preprint at https://arxiv.org/abs/1808.03331v1 (2018).
Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (vol. 9) (eds Tehand, Y. W. & Titterington, M.) 249–256 (2010).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) https://dblp.org/rec/bib/journals/corr/KingmaB14 (2015).
Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. In Proc. International Conference on Machine Learning (eds Precup, D. & Teh, Y. W.) 1321–1330 (2017).
Platt, J. C. in Advances in Large-Margin Classifiers (eds Smola, A. et al.) 61–74 (MIT Press, 1999).
Brier, G. W. Verification of forecasts expressed in terms of probability. Mon. Weath. Rev. 78, 1–3 (1950).
Niculescu-Mizil, A. & Caruana, R. Predicting good probabilities with supervised learning. In Proc. International Conference on Machine Learning (eds Raedt, L. D. & Wrobel, S.) 625–632 (ACM, 2005).
Saito, T. & Rehmsmeier, M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10, e0118432 (2015).
Efron, B. & Tibshirani, R. J. An Introduction to the Bootstrap (CRC, 1994).
Mann, H. B. & Whitney, D. R. On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18, 50–60 (1947).
Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) 6402–6413 (2017).
De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342–1350 (2018).
Abadi, M. et al. Tensorflow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1603.04467 (2015).
We thank the veterans and their families under the care of the US Department of Veterans Affairs. We thank A. Graves, O. Vinyals, K. Kavukcuoglu, S. Chiappa, T. Lillicrap, R. Raine, P. Keane, M. Seneviratne, A. Schlosberg, O. Ronneberger, J. De Fauw, K. Ruark, M. Jones, J. Quinn, D. Chou, C. Meaden, G. Screen, W. West, R. West, P. Sundberg and the Google AI team, J. Besley, M. Bawn, K. Ayoub and R. Ahmed. Finally, we thank the many physicians, administrators and researchers of the US Department of Veterans Affairs who worked on the data collection, and the rest of the DeepMind team for their support, ideas and encouragement. G.R. and H.M. were supported by University College London and the National Institute for Health Research (NIHR) University College London Hospitals Biomedical Research Centre. The views expressed are those of these author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
G.R., H.M. and C.L. are paid contractors of DeepMind. The authors have no other competing interests to disclose.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Peer review information Nature thanks Lui G. Forni, Suchi Saria and Eric Topol for their contribution to the peer review of this work.
Extended data figures and tables
All electronic health record data available for each patient were structured into a sequential history for both inpatient and outpatient events in six-hourly blocks, shown here as circles. In each 24-h period, events without a recorded time were included in a fifth block. Apart from the data present at the current time step, the models optionally receive an embedding of the previous 48 h and the longer history of 6 months or 5 years.
The best performance was achieved by a multi-task deep recurrent highway network architecture on top of an L1-regularized deep residual embedding component that learns the best data representation end-to-end without pre-training.
a, b, The predictions were recalibrated using isotonic regression before (a) and after (b) calibration. Model predictions were grouped into 20 buckets, with a mean model risk prediction plotted against the percentage of positive labels in that bucket. The diagonal line demonstrates the ideal calibration. Source Data
a, For prediction of any AKI within 48 h at 33% precision, nearly half of all predictions are trailing, after the AKI has already occurred (orange bars) or early, more than 48 h prior (blue bars). The histogram shows the distribution of these trailing and early false positives for prediction. Incorrect predictions are mapped to their closest preceding or following episode of AKI (whichever is closer) if that episode occurs in an admission. For ±1 day, 15.2% of false positives correspond to observed AKI events within 1 day after the prediction (model reacted too early) and 2.9% correspond to observed AKI events within 1 day before the prediction (model reacted too late). b, Subgroup analysis for all false-positive alerts. In addition to the 49% of false-positive alerts that were made in admissions during which there was at least one episode of AKI, many of the remaining false-positive alerts were made in patients who had evidence of clinical risk factors present in their available electronic health record data. These risk factors are shown here for the proposed model that predicts any stage of AKI occurring within the next 48 h. Source Data
Supplementary Sections A-K, including Supplementary Figures 1-12 and Supplementary Tables 1-12. Supplementary Section A: Supplementary figures showing the visual examples from five systematically selected success cases and five systematically selected failure cases from the predictive model. Supplementary Section B: Supplementary analysis of the auxiliary numerical prediction tasks. Supplementary Section C: Additional analysis from an experiment into the significance of individual features in our trained models based on occlusion analysis. Supplementary Section D: Supplementary results and methods from the comparison of broad comparison of available models on the AKI prediction task. Supplementary Section E: Comparison of our models performance to baseline models trained on features that have been chosen by clinicians as being relevant for modelling kidney function. Supplementary Section F: The results of literature reviews into risk prediction of AKI and machine learning on electronic health records. Supplementary Section G: Supplementary analyses and results of individual subgroups of the patient population studied. Supplementary Section H: Supplementary analysis of the influence of data recency on model performance. Supplementary Section I: Analysis of the contribution of the aspects of our model’s design to its overall performance through an ablation study that removes specific components of the model, training it fully, and then comparing the simplified model’s PR AUC on the validation set. Supplementary Section J: Supplementary methods and results from the hyperparameter sweeps described in the Methods section. Supplementary Section K: Additional analysis from an experiment into the relationship between model confidence and prediction accuracy.
This file contains Source Data for Supplementary Figure 1.
About this article
Nature Reviews Nephrology (2019)