Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Explainable machine-learning predictions for the prevention of hypoxaemia during surgery

Nature Biomedical Engineering volume 2, pages 749–760 (2018)Cite this article

Subjects

Abstract

Although anaesthesiologists strive to avoid hypoxaemia during surgery, reliably predicting future intraoperative hypoxaemia is not possible at present. Here, we report the development and testing of a machine-learning-based system that predicts the risk of hypoxaemia and provides explanations of the risk factors in real time during general anaesthesia. The system, which was trained on minute-by-minute data from the electronic medical records of over 50,000 surgeries, improved the performance of anaesthesiologists by providing interpretable hypoxaemia risks and contributing factors. The explanations for the predictions are broadly consistent with the literature and with prior knowledge from anaesthesiologists. Our results suggest that if anaesthesiologists currently anticipate 15% of hypoxaemia events, with the assistance of this system they could anticipate 30%, a large portion of which may benefit from early intervention because they are associated with modifiable factors. The system can help improve the clinical understanding of hypoxaemia risk during anaesthesia care by providing general insights into the exact changes in risk induced by certain characteristics of the patient or procedure.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Prescience integrates many data sources into a single risk, which is explained through a succinct visual summary.
Fig. 2: Patient and procedure characteristics.
Fig. 3: Pooled comparison of the prediction performance of five anaesthesiologists with and without the assistance of Prescience.
Fig. 4: Sample real-time prediction during a procedure.
Fig. 5: Comparison of averaged feature-importance estimates between Prescience and anaesthesiologists for both initial and real-time prediction.
Fig. 6: Effect of varying individual feature values for both preoperative and real-time features.

Similar content being viewed by others

Data availability

Owing to patient-privacy considerations, the operating-room datasets from participating hospitals are not publicly available. The raw data from the anaesthesiologist comparisons in Fig. 3 are available in Supplementary Tables 6 and 7, and data from Fig. 5 are available in Supplementary Tables 8 and 9.

References

  1. Weiser, T. G. et al. Estimate of the global volume of surgery in 2012: an assessment supporting improved health outcomes. Lancet 385, S11 (2015).

    Article  Google Scholar 

  2. Gawande, A. A., Thomas, E. J., Zinner, M. J. & Brennan, T. A. The incidence and nature of surgical adverse events in Colorado and Utah in 1992. Surgery 126, 66–75 (1999).

    Article  CAS  Google Scholar 

  3. Kable, A. K., Gibberd, R. W. & Spigelman, A. D. Adverse events in surgical patients in Australia. Int. J. Qual. Health Care 14, 269–276 (2002).

    Article  CAS  Google Scholar 

  4. Nair, B. G., Gabel, E., Hofer, I., Schwid, H. A. & Cannesson, M. Intraoperative clinical decision support for anesthesia. Anesth. Analg. 124, 603–617 (2017).

    Article  Google Scholar 

  5. Maier-Hein, L. et al. Surgical data science for next-generation interventions. Nat. Biomed. Eng. 1, 691–696 (2017).

    Article  Google Scholar 

  6. Dunham, C. M., Hileman, B. M., Hutchinson, A. E., Chance, E. A. & Huang, G. S. Perioperative hypoxemia is common with horizontal positioning during general anesthesia and is associated with major adverse outcomes: a retrospective study of consecutive patients. BMC Anesthesiol. 14, 43 (2014).

    Article  Google Scholar 

  7. Strachan, L. & Noble, D. W. Hypoxia and surgical patients--prevention and treatment of an unnecessary cause of morbidity and mortality. J. R. Coll. Surg. Edinb. 46, 297–302 (2001).

    CAS  PubMed  Google Scholar 

  8. Ehrenfeld, J. M. et al. The incidence of hypoxemia during surgery: evidence from two institutions. Can. J. Anaesth. 57, 888–897 (2010).

    Article  Google Scholar 

  9. Kooij, F. O., Klok, T., Hollmann, M. W. & Kal, J. E. Decision support increases guideline adherence for prescribing postoperative nausea and vomiting prophylaxis. Anesth. Analg. 106, 893–898 (2008).

    Article  Google Scholar 

  10. Garg, A. X. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 293, 1223–1238 (2005).

    Article  CAS  Google Scholar 

  11. ElMoaqet, H., Tilbury, D. M. & Ramachandran, S. K. Multi-step ahead predictions for critical levels in physiological time series. IEEE Trans. Cybern. 46, 1704–1714 (2016).

    Article  Google Scholar 

  12. Lipton, Z. C., Kale, D. C. & Wetzell, R. C. Phenotyping of clinical time series with LSTM recurrent neural networks. Preprint at http://arxiv.org/abs/1510.07641 (2015).

  13. Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 7, 299ra122 (2015).

    Article  Google Scholar 

  14. Saria, S., Rajani, A. K., Gould, J., Koller, D. & Penn, A. A. Integration of early physiological responses predicts later illness severity in preterm infants. Sci. Transl. Med. 2, 48ra65 (2010).

    Article  Google Scholar 

  15. Caruana, R. et al. Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In Proc. 21st ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. 1721–1730 (ACM, 2015).

  16. Deo, R. C. Machine learning in medicine. Circulation 132, 1920–1930 (2015).

    Article  Google Scholar 

  17. Memarian, N., Kim, S., Dewar, S., Engel, J. & Staba, R. J. Multimodal data and machine learning for surgery outcome prediction in complicated cases of mesial temporal lobe epilepsy. Comput. Biol. Med. 64, 67–78 (2015).

    Article  Google Scholar 

  18. Štrumbelj, E. & Kononenko, I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 41, 647–665 (2014).

    Article  Google Scholar 

  19. Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?” Explaining the predictions of any classifier. In Proc. 22nd ACM SIGKDD Int. Conf. Know. Disc. Data Min. 1135–1144 (ACM, 2016).

  20. Lundberg, S. & Lee, S.-I. A unified approach to interpreting model predictions. In Adv. Neural Information Processing 4765–4774 (Curran Associates, 2017).

  21. Lundberg, S. M., Erion, G. G. & Lee, S.-I. Consistent individualized feature attribution for tree ensembles. Preprint at http://arxiv.org/abs/1802.03888 (2018).

  22. Tarassenko, L., Hann, A. & Young, D. Integrated monitoring and analysis for early warning of patient deterioration. Br. J. Anaesth. 97, 64–68 (2006).

    Article  CAS  Google Scholar 

  23. Summers, R. L., Pipke, M., Wegerich, S., Conkright, G. & Isom, K. C. Functionality of empirical model-based predictive analytics for the early detection of hemodynamic instabilty. Biomed. Sci. Instrum. 50, 219–224 (2014).

    PubMed  Google Scholar 

  24. Current Procedural Terminology: CPT (American Medical Association, 2007).

  25. Friedman, J. H. Greedy function approximation: a gradient boosting machine. Ann. Stat. 29, 1189–1232 (2001).

    Article  Google Scholar 

  26. Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. In Proc. 22nd ACM SIGKDD Int. Conf. Know. Disc. Data Min. 785–794 (ACM, 2016).

  27. Lumachi, F. et al. Relationship between body mass index, age and hypoxemia in patients with extremely severe obesity undergoing bariatric surgery. In Vivo 24, 775–777 (2010).

  28. Kendale, S. M. & Blitz, J. D. Increasing body mass index and the incidence of intraoperative hypoxemia. J. Clin. Anesth. 33, 97–104 (2016).

    Article  Google Scholar 

  29. Defining Adult Overweight and Obesity (Centers for Disease Control and Prevention, 2016); https://www.cdc.gov/obesity/adult/defining.html

  30. Myles, P. S., Leslie, K., McNeil, J., Forbes, A. & Chan, M. T. V. Bispectral index monitoring to prevent awareness during anaesthesia: the B-Aware randomised controlled trial. Lancet 363, 1757–1763 (2004).

    Article  CAS  Google Scholar 

  31. Avidan, M. S. et al. Anesthesia awareness and the bispectral index. N. Engl. J. Med. 358, 1097–1108 (2008).

    Article  CAS  Google Scholar 

  32. Epstein, R. H., Dexter, F. & Patel, N. Influencing anesthesia provider behavior using anesthesia information management system data for near real-time alerts and post hoc reports. Anesth. Analg. 121, 678–692 (2015).

    Article  CAS  Google Scholar 

  33. Guay, J. & Ochroch, E. A. Intraoperative use of low volume ventilation to decrease postoperative mortality, mechanical ventilation, lengths of stay and lung injury in patients without acute lung injury. Cochrane Datab. Syst. Rev. J. 2018, CD011151 (2018).

  34. Pulse Oximetry Training Manual (World Health Organization, 2011).

  35. Dyagilev, K. & Saria, S. Learning (predictive) risk scores in the presence of censoring due to interventions. Mach. Learn. 102, 323–348 (2016).

    Article  Google Scholar 

  36. Roth, A. E. (ed.) The Shapley Value: Essays in Honor of Llloyd S. Shapley (Cambridge Univ. Press, Cambridge, 1988).

    Google Scholar 

  37. Health, United States, 2016: With Chartbook on Long-term Trends in Health 314–317 (National Center for Health Statistics, Hyattsville, 2017).

Download references

Acknowledgements

We thank G. Erion, M. T. Ribeiro, J. Schreiber and members of the Lee laboratory for feedback and suggestions that improved the manuscript and experiments. This work was supported by National Science Foundation grant nos. DBI-135589 and DBI-1552309, National Institutes of Health grant no. 1R35GM128638, NSF Graduate Research Fellowship grant no. DGE-1256082 and a UW eScience/ITHS seed grant Machine Learning in Operating Rooms.

Author information

Authors and Affiliations

  1. Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA

    Scott M. Lundberg & Su-In Lee

  2. Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, WA, USA

    Bala Nair, Monica S. Vavilala, Michael J. Eisses, Trevor Adams, David E. Liston, Daniel King-Wai Low, Shu-Fang Newman & Jerry Kim

  3. Center for Perioperative and Pain initiatives in Quality Safety Outcome, University of Washington, Seattle, WA, USA

    Bala Nair, Monica S. Vavilala & Shu-Fang Newman

  4. Harborview Injury Prevention and Research Center, University of Washington, Seattle, WA, USA

    Bala Nair & Monica S. Vavilala

  5. Veterans Affairs Puget Sound Health Care System, Seattle, WA, USA

    Mayumi Horibe

  6. Seattle Children’s Hospital, Seattle, WA, USA

    Michael J. Eisses, Trevor Adams, David E. Liston, Daniel King-Wai Low & Jerry Kim

Authors
  1. Scott M. Lundberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bala Nair
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Monica S. Vavilala
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mayumi Horibe
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael J. Eisses
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Trevor Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. David E. Liston
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Daniel King-Wai Low
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Shu-Fang Newman
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jerry Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Su-In Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.-I.L., S.M.L., B.N. and J.K. initiated the study. S.-I.L. and S.M.L. developed the Prescience algorithms and designed data analyses and experiments. S.M.L. performed data analyses, experiments and data preprocessing. B.N. and S.-F.N. provided the electronic medical record data. J.K. recruited anaesthesiologists and helped design the anaesthesiologist test and survey. M.H., M.J.E., T.A., D.E.L. and D.K.-W.L. performed the web-based anaesthesiologist experiments and provided survey data. M.S.V. provided clinical assessment, interpretation of feature importances and connections with anaesthesiologists’ workflow. S.-I L. and S.M.L. wrote the paper in conjunction with B.N., J.K. and M.S.V. who wrote the sections on clinical interpretation and integration with current practices. M.H. provided manuscript feedback.

Corresponding author

Correspondence to Su-In Lee.

Ethics declarations

Competing interests

B.N. is an advisor for Perimatics LLC and holds equity in the company. D.K.-W.L. is a Chief Medical Officer for MDmetrix, Inc. The other authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figures 1–12, Supplementary Tables 1–3 and Supplementary References 1–2.

Reporting Summary

Supplementary Table 4

Initial features used by Prescience. An enumeration of all the 3,797 features used for preoperative predictions.

Supplementary Table 5

Intraoperative features used by Prescience. An enumeration of all the 3,905 features used for intraoperative predictions.

Supplementary Table 6

Data for Fig. 3a.

Supplementary Table 7

Data for Fig. 3b.

Supplementary Table 8

Data for Fig. 5a.

Supplementary Table 9

Data for Fig. 5b.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lundberg, S.M., Nair, B., Vavilala, M.S. et al. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nat Biomed Eng 2, 749–760 (2018). https://doi.org/10.1038/s41551-018-0304-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-018-0304-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics