Article | Published:

Explainable machine-learning predictions for the prevention of hypoxaemia during surgery

Nature Biomedical Engineeringvolume 2pages749760 (2018) | Download Citation

Abstract

Although anaesthesiologists strive to avoid hypoxaemia during surgery, reliably predicting future intraoperative hypoxaemia is not possible at present. Here, we report the development and testing of a machine-learning-based system that predicts the risk of hypoxaemia and provides explanations of the risk factors in real time during general anaesthesia. The system, which was trained on minute-by-minute data from the electronic medical records of over 50,000 surgeries, improved the performance of anaesthesiologists by providing interpretable hypoxaemia risks and contributing factors. The explanations for the predictions are broadly consistent with the literature and with prior knowledge from anaesthesiologists. Our results suggest that if anaesthesiologists currently anticipate 15% of hypoxaemia events, with the assistance of this system they could anticipate 30%, a large portion of which may benefit from early intervention because they are associated with modifiable factors. The system can help improve the clinical understanding of hypoxaemia risk during anaesthesia care by providing general insights into the exact changes in risk induced by certain characteristics of the patient or procedure.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Data availability

Owing to patient-privacy considerations, the operating-room datasets from participating hospitals are not publicly available. The raw data from the anaesthesiologist comparisons in Fig. 3 are available in Supplementary Tables 6 and 7, and data from Fig. 5 are available in Supplementary Tables 8 and 9.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Weiser, T. G. et al. Estimate of the global volume of surgery in 2012: an assessment supporting improved health outcomes. Lancet 385, S11 (2015).

  2. 2.

    Gawande, A. A., Thomas, E. J., Zinner, M. J. & Brennan, T. A. The incidence and nature of surgical adverse events in Colorado and Utah in 1992. Surgery 126, 66–75 (1999).

  3. 3.

    Kable, A. K., Gibberd, R. W. & Spigelman, A. D. Adverse events in surgical patients in Australia. Int. J. Qual. Health Care 14, 269–276 (2002).

  4. 4.

    Nair, B. G., Gabel, E., Hofer, I., Schwid, H. A. & Cannesson, M. Intraoperative clinical decision support for anesthesia. Anesth. Analg. 124, 603–617 (2017).

  5. 5.

    Maier-Hein, L. et al. Surgical data science for next-generation interventions. Nat. Biomed. Eng. 1, 691–696 (2017).

  6. 6.

    Dunham, C. M., Hileman, B. M., Hutchinson, A. E., Chance, E. A. & Huang, G. S. Perioperative hypoxemia is common with horizontal positioning during general anesthesia and is associated with major adverse outcomes: a retrospective study of consecutive patients. BMC Anesthesiol. 14, 43 (2014).

  7. 7.

    Strachan, L. & Noble, D. W. Hypoxia and surgical patients--prevention and treatment of an unnecessary cause of morbidity and mortality. J. R. Coll. Surg. Edinb. 46, 297–302 (2001).

  8. 8.

    Ehrenfeld, J. M. et al. The incidence of hypoxemia during surgery: evidence from two institutions. Can. J. Anaesth. 57, 888–897 (2010).

  9. 9.

    Kooij, F. O., Klok, T., Hollmann, M. W. & Kal, J. E. Decision support increases guideline adherence for prescribing postoperative nausea and vomiting prophylaxis. Anesth. Analg. 106, 893–898 (2008).

  10. 10.

    Garg, A. X. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 293, 1223–1238 (2005).

  11. 11.

    ElMoaqet, H., Tilbury, D. M. & Ramachandran, S. K. Multi-step ahead predictions for critical levels in physiological time series. IEEE Trans. Cybern. 46, 1704–1714 (2016).

  12. 12.

    Lipton, Z. C., Kale, D. C. & Wetzell, R. C. Phenotyping of clinical time series with LSTM recurrent neural networks. Preprint at http://arxiv.org/abs/1510.07641 (2015).

  13. 13.

    Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 7, 299ra122 (2015).

  14. 14.

    Saria, S., Rajani, A. K., Gould, J., Koller, D. & Penn, A. A. Integration of early physiological responses predicts later illness severity in preterm infants. Sci. Transl. Med. 2, 48ra65 (2010).

  15. 15.

    Caruana, R. et al. Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In Proc. 21st ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. 1721–1730 (ACM, 2015).

  16. 16.

    Deo, R. C. Machine learning in medicine. Circulation 132, 1920–1930 (2015).

  17. 17.

    Memarian, N., Kim, S., Dewar, S., Engel, J. & Staba, R. J. Multimodal data and machine learning for surgery outcome prediction in complicated cases of mesial temporal lobe epilepsy. Comput. Biol. Med. 64, 67–78 (2015).

  18. 18.

    Štrumbelj, E. & Kononenko, I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 41, 647–665 (2014).

  19. 19.

    Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?” Explaining the predictions of any classifier. In Proc. 22nd ACM SIGKDD Int. Conf. Know. Disc. Data Min. 1135–1144 (ACM, 2016).

  20. 20.

    Lundberg, S. & Lee, S.-I. A unified approach to interpreting model predictions. In Adv. Neural Information Processing 4765–4774 (Curran Associates, 2017).

  21. 21.

    Lundberg, S. M., Erion, G. G. & Lee, S.-I. Consistent individualized feature attribution for tree ensembles. Preprint at http://arxiv.org/abs/1802.03888 (2018).

  22. 22.

    Tarassenko, L., Hann, A. & Young, D. Integrated monitoring and analysis for early warning of patient deterioration. Br. J. Anaesth. 97, 64–68 (2006).

  23. 23.

    Summers, R. L., Pipke, M., Wegerich, S., Conkright, G. & Isom, K. C. Functionality of empirical model-based predictive analytics for the early detection of hemodynamic instabilty. Biomed. Sci. Instrum. 50, 219–224 (2014).

  24. 24.

    Current Procedural Terminology: CPT (American Medical Association, 2007).

  25. 25.

    Friedman, J. H. Greedy function approximation: a gradient boosting machine. Ann. Stat. 29, 1189–1232 (2001).

  26. 26.

    Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. In Proc. 22nd ACM SIGKDD Int. Conf. Know. Disc. Data Min. 785–794 (ACM, 2016).

  27. 27.

    Lumachi, F. et al. Relationship between body mass index, age and hypoxemia in patients with extremely severe obesity undergoing bariatric surgery. In Vivo 24, 775–777 (2010).

  28. 28.

    Kendale, S. M. & Blitz, J. D. Increasing body mass index and the incidence of intraoperative hypoxemia. J. Clin. Anesth. 33, 97–104 (2016).

  29. 29.

    Defining Adult Overweight and Obesity (Centers for Disease Control and Prevention, 2016); https://www.cdc.gov/obesity/adult/defining.html

  30. 30.

    Myles, P. S., Leslie, K., McNeil, J., Forbes, A. & Chan, M. T. V. Bispectral index monitoring to prevent awareness during anaesthesia: the B-Aware randomised controlled trial. Lancet 363, 1757–1763 (2004).

  31. 31.

    Avidan, M. S. et al. Anesthesia awareness and the bispectral index. N. Engl. J. Med. 358, 1097–1108 (2008).

  32. 32.

    Epstein, R. H., Dexter, F. & Patel, N. Influencing anesthesia provider behavior using anesthesia information management system data for near real-time alerts and post hoc reports. Anesth. Analg. 121, 678–692 (2015).

  33. 33.

    Guay, J. & Ochroch, E. A. Intraoperative use of low volume ventilation to decrease postoperative mortality, mechanical ventilation, lengths of stay and lung injury in patients without acute lung injury. Cochrane Datab. Syst. Rev. J. 2018, CD011151 (2018).

  34. 34.

    Pulse Oximetry Training Manual (World Health Organization, 2011).

  35. 35.

    Dyagilev, K. & Saria, S. Learning (predictive) risk scores in the presence of censoring due to interventions. Mach. Learn. 102, 323–348 (2016).

  36. 36.

    Roth, A. E. (ed.) The Shapley Value: Essays in Honor of Llloyd S. Shapley (Cambridge Univ. Press, Cambridge, 1988).

  37. 37.

    Health, United States, 2016: With Chartbook on Long-term Trends in Health 314–317 (National Center for Health Statistics, Hyattsville, 2017).

Download references

Acknowledgements

We thank G. Erion, M. T. Ribeiro, J. Schreiber and members of the Lee laboratory for feedback and suggestions that improved the manuscript and experiments. This work was supported by National Science Foundation grant nos. DBI-135589 and DBI-1552309, National Institutes of Health grant no. 1R35GM128638, NSF Graduate Research Fellowship grant no. DGE-1256082 and a UW eScience/ITHS seed grant Machine Learning in Operating Rooms.

Author information

Affiliations

  1. Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA

    • Scott M. Lundberg
    •  & Su-In Lee
  2. Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, WA, USA

    • Bala Nair
    • , Monica S. Vavilala
    • , Michael J. Eisses
    • , Trevor Adams
    • , David E. Liston
    • , Daniel King-Wai Low
    • , Shu-Fang Newman
    •  & Jerry Kim
  3. Center for Perioperative and Pain initiatives in Quality Safety Outcome, University of Washington, Seattle, WA, USA

    • Bala Nair
    • , Monica S. Vavilala
    •  & Shu-Fang Newman
  4. Harborview Injury Prevention and Research Center, University of Washington, Seattle, WA, USA

    • Bala Nair
    •  & Monica S. Vavilala
  5. Veterans Affairs Puget Sound Health Care System, Seattle, WA, USA

    • Mayumi Horibe
  6. Seattle Children’s Hospital, Seattle, WA, USA

    • Michael J. Eisses
    • , Trevor Adams
    • , David E. Liston
    • , Daniel King-Wai Low
    •  & Jerry Kim

Authors

  1. Search for Scott M. Lundberg in:

  2. Search for Bala Nair in:

  3. Search for Monica S. Vavilala in:

  4. Search for Mayumi Horibe in:

  5. Search for Michael J. Eisses in:

  6. Search for Trevor Adams in:

  7. Search for David E. Liston in:

  8. Search for Daniel King-Wai Low in:

  9. Search for Shu-Fang Newman in:

  10. Search for Jerry Kim in:

  11. Search for Su-In Lee in:

Contributions

S.-I.L., S.M.L., B.N. and J.K. initiated the study. S.-I.L. and S.M.L. developed the Prescience algorithms and designed data analyses and experiments. S.M.L. performed data analyses, experiments and data preprocessing. B.N. and S.-F.N. provided the electronic medical record data. J.K. recruited anaesthesiologists and helped design the anaesthesiologist test and survey. M.H., M.J.E., T.A., D.E.L. and D.K.-W.L. performed the web-based anaesthesiologist experiments and provided survey data. M.S.V. provided clinical assessment, interpretation of feature importances and connections with anaesthesiologists’ workflow. S.-I L. and S.M.L. wrote the paper in conjunction with B.N., J.K. and M.S.V. who wrote the sections on clinical interpretation and integration with current practices. M.H. provided manuscript feedback.

Competing interests

B.N. is an advisor for Perimatics LLC and holds equity in the company. D.K.-W.L. is a Chief Medical Officer for MDmetrix, Inc. The other authors declare no competing interests.

Corresponding author

Correspondence to Su-In Lee.

Supplementary information

  1. Supplementary Information

    Supplementary Figures 1–12, Supplementary Tables 1–3 and Supplementary References 1–2.

  2. Reporting Summary

  3. Supplementary Table 4

    Initial features used by Prescience. An enumeration of all the 3,797 features used for preoperative predictions.

  4. Supplementary Table 5

    Intraoperative features used by Prescience. An enumeration of all the 3,905 features used for intraoperative predictions.

  5. Supplementary Table 6

    Data for Fig. 3a.

  6. Supplementary Table 7

    Data for Fig. 3b.

  7. Supplementary Table 8

    Data for Fig. 5a.

  8. Supplementary Table 9

    Data for Fig. 5b.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41551-018-0304-0

Further reading