Article | Open | Published:

Evaluating the impact of organisational digital maturity on clinical outcomes in secondary care in England

npj Digital Medicinevolume 2, Article number: 41 (2019) | Download Citation

Abstract

All healthcare systems are increasingly reliant on health information technology to support the delivery of high-quality, efficient and safe care. Data on its effectiveness are however limited. We therefore sought to examine the impact of organisational digital maturity on clinical outcomes in secondary care within the English National Health Service. We conducted a retrospective analysis of routinely collected administrative data for 13,105,996 admissions across 136 hospitals in England from 2015 to 2016. Data from the 2016 NHS Clinical Digital Maturity Index were used to characterise organisational digital maturity. A multivariable regression model including 12 institutional covariates was utilised to examine the relationship between one measure of organisational digital maturity and five key clinical outcome measures. There was no significant relationship between organisational digital maturity and risk-adjusted 30-day mortality, 28-day readmission rates or complications of care. In multivariable analysis risk-adjusted long length of stay and harm-free care were significantly related to aspects of organisational digital maturity; digitally mature hospitals may not only deliver more harm-free care episodes but also may have a significantly increased risk of patients experiencing a long length of stay. Organisational digital maturity is to some extent related to selected clinical outcomes in secondary care in England. Digital maturity is, however, also strongly linked to other institutional factors that likely play a greater role in influencing clinical outcomes. There is a need to better understand how health IT impacts care delivery and supports other drivers of hospital quality.

Introduction

Reducing variation in care quality, improving outcomes and lowering cost continue to be significant challenges for all healthcare systems. There is growing evidence that health information technology (IT) is a potential solution to these challenges, and hospitals increasingly rely upon electronic systems to support the delivery of high-quality, efficient and safe care.1 Electronic systems may have a number of potential benefits over traditional paper-based approaches and can help support patient safety, reduce adverse events, improve clinical outcomes, lessen error, strengthen the quality and availability of information, support better decision-making and communication and foster improvements in workflow and culture.2,3,4,5,6,7,8,9

The majority of digital technology evaluations focus on narrow operational or process led benefits of specific technologies, typically predate recent advances in health IT or are focused on single institutions; only a handful evaluate the impact of multi-functional commercially developed systems within complex organisations.4,10,11,12 In order to fully examine the impact of new digital technology, it is therefore important to understand its broader impact in real-world settings. The digital maturity of a healthcare organisation is the extent to which its health IT is an enabler of high-quality care through supporting improvements to service delivery and patient experience. Digital maturity is multi-faceted, and encompasses not only technology, resource and capability but also the digital literacy, ability and motivation of staff and patients to use new technologies.11,13 There is a relative paucity of evidence evaluating the impact of digital maturity on meaningful outcomes at an organisational or health system level. Despite this, there is a growing body of evidence to suggest that digitally mature hospitals with fully electronic health records and advanced order entry and clinical decision support systems have fewer complications, lower mortality rates and reduced costs of care.5,9,14

Much of the published literature looking at the impact of health IT on performance and outcomes is focused on the US healthcare system. The influence of centrally mandated incentives to improve the adoption and meaningful use of health IT such as the $30 billion HITECH Act,15,16,17 together with the significant additional expenditure on healthcare seen in the US compared with other developed countries (17.2% of GDP vs. 9%18), and significant variation in access, equity and outcomes,19,20 means that drawing meaningful comparisons with other healthcare systems is challenging. In addition, evaluations of digital technology to date have typically focused on specific products or functions in distinct settings. In practice, digital technology is used within complex sociotechnical systems that do not operate using discrete independent processes, meaning vital aspects of their impact may be missed.21 In this study, we therefore seek to further understand the impact of organisational digital maturity on clinical outcomes in secondary care in England.

Results

Population characteristics

Population characteristics and standard descriptive statistics for all outcome variables and covariates are presented in Table 1. There are significant differences in staffing, workload and infrastructure across each of the participating hospitals. The mean aggregate of the total CDMI score for each organisation was 797 (range 324–1253, SD 174). There was significant variation among the hospitals in all domains of CDMI with a mean infrastructure score of 68 (range 20–100, SD 16.4), mean capability score of 354 (range 115–671, SD 116) and mean readiness score of 376 (range 177–495, SD 67). Whilst there was significant variation in CDMI scores, as expected less variation was seen between the outcome variables of choice: mean SHMI 1.00 (range 0.67–1.17, SD 0.09, IQR 0.12), mean READ 99.54 (range 83.54–111.15, SD 5.58, IQR 7.60), mean LLOS 98.85 (range 72.23–119.89, SD 10.47, IQR 15.74), mean HFC 93.8% (range 87.78–98.76, SD 1.93, IQR 2.00) and mean COC 3.10% (range 1.54–5.80, SD 0.75, IQR 0.82).

Table 1 Characteristics and descriptive statistics for 136 non-specialist NHS Trusts in England

Organisational digital maturity and outcomes

For each selected outcome variable, the results from the respective univariate and multivariable analyses are presented in Table 2.

Table 2 Association between organisational digital maturity and selected clinical outcomes for 136 hospitals in England

Relative risk of long length of stay (LLOS)

There was a significant association between organisational digital maturity and the relative risk of a long length of stay. At the univariate level, both the readiness and capability domains and aggregate total CDMI score were significantly related. This relationship remained when the other institutional covariates were accounted for in the multivariable model with total CDMI score (B = 0.01, p = 0.014, 95% CI 2.78–3–0.02) significantly related to LLOS (R2 = 0.2934).

Harm-free patient care episodes (HFC)

At a univariate level, there was a clear underlying relationship between organisational digital maturity and the provision of harm-free care. HFC was significantly related to all aspects of the CDMI tool at the univariate level. Importantly, this relationship remained at a multivariable level when institutional covariates were accounted for with total CDMI score (B = 2.42–3, p = 0.033, 95% CI: 2.03–4–4.63–3) significantly related to HFC (R2 = 0.1499).

Summary hospital-level mortality index (SHMI)

There was no clear relationship between organisational digital maturity and risk-adjusted 30-day mortality for hospitals in England. At a univariate level, only the capability domain of the CDMI tool; however, this relationship was not present in the multivariable model (R2 = 0.3207).

Relative risk of readmission (READ)

There was no significant association between any aspect of organisational digital maturity and the relative risk of readmission and at either the univariate or multivariable level for hospitals in England.

Patient episodes featuring complications of care (COC)

There was no significant association between any aspect of organisational digital maturity and patient episodes featuring complications of care and at either the univariate or multivariable level for hospitals in England.

Discussion

Using nationally representative data from 136 hospitals in England from 2015 to 2016, we found that aspects of organisational digital maturity are significantly related to the relative risk of a long length of stay for patients receiving inpatient care and the provision of harm-free care. However, the relationships are not clear-cut, and whilst digital maturity inevitably does play a role in influencing clinical outcomes, other institutional variables that too are strongly linked to digital maturity also play a significant part and may have a far greater impact.

A strength of this study is the large, nationally representative data that have been used. It is a pragmatic examination of 13,105,996 inpatient admissions over a 12-month period across 136 acute care providers in England. This represents a larger and more representative cohort than previously evaluated in similar studies.5,9 Furthermore, this is the first study to examine impact of organisational digital maturity on clinical outcomes in the English healthcare system.

There remains widespread variation in organisational digital maturity across hospitals in England, despite the drive for a fully electronic NHS being more than a decade old.22 Determinants of clinical outcomes at an institutional level are numerous and complex, and encompass a wide range of inter-related factors with multi-directional cause and effect. For example, well-staffed,23,24 high-volume hospitals25,26 with manageable bed occupancy levels27,28 all have consistently better quality outcomes. In addition to considering other institutional factors that influence outcomes, it is also important to be cognisant of the high cost of new digital technology when evaluating its impact. The cost and quality repercussions of health IT investment have not been fully established, and the return on investment from expensive IT projects may take many years to come to fruition and be problematic to verify.29 The pursuit of better digital maturity has multiple drivers and many of it's perceived benefits are potentially achievable through alternative means. Therefore, determining the specific influence of digital maturity on clinical outcomes is a challenging endeavour. This study has nonetheless demonstrated the potential power and utility of administrative datasets to investigate crucial questions regarding the true impact of digital investment in secondary care.

This analysis has suggested that once other relevant organisational factors are considered, there appears to be no significant relationship between organisational digital maturity and risk-adjusted mortality, risk of readmission nor complications of care. This study did, however, demonstrate a significant relationship between LLOS and both the readiness domain and the total aggregate digital maturity score; a relationship that was remained significant, although less pronounced at the multivariable level. Digitally mature hospitals have a greater number of patients with a risk-adjusted long length of stay. Again, it is likely that in this instance, organisational digital maturity is confounded by other relevant institutional factors which are also likely associated with digital maturity. Large academic teaching hospitals are likely to treat patients with more problematic health needs and undertake more complex interventions. Although patients with common conditions who are treated at these major academic teaching hospitals have significantly lower mortality rates30 and lower rates of failure to rescue incidents,26 the presence of a high-risk complex patient cohort may act to negatively influence overall length of stay overall. These large academic hospitals, typically located in urban areas, are also more likely to be digitally mature and have better health IT than small, non-academic rural institutions that may lack the resource, knowledge and skill to embark on large, expensive and complex IT projects which can drive better performance and higher quality.15,31,32

This analysis has also demonstrated a relationship between HFC and organisational digital maturity; digitally mature hospitals deliver a greater number of harm-free care episodes to their patients. There is evidence that the use of electronic systems and other digital innovations can significantly improve compliance with prophylaxis33 and reduce the incidence of VTE events,34 reduce the duration of unneeded catheterisation35 and subsequent incidence of CAUTI,36 improve the prediction and identification of pressure ulcers37,38 and support better risk assessments and reductions in the incidence of inpatient falls.39,40 However, it is likely that this relationship may be confounded by more important determinants of safe care. It is plausible that hospitals with better resources buy better IT which in turn directly influences care, for example through better access to clinical decision support, improved surfacing of relevant clinical data and superior interoperability with other health and social care organisations. Alternatively, the digital maturity of an organisation may directly influence the quality and completeness of data capture which in turn affects outcome reporting; more mature organisations have higher quality data. However, digital maturity does also plausibly act as a proxy for the provision of safe care. Effective and well-led organisations are likely to have robust policies and procedures in place to reduce harmful events and deliver safe care, deploy more sophisticated approaches to quality improvement, and have better resources and a more mature and developed approach to health IT.

Given these findings, it is important that future works seeks to unpick the complex systems present in order to establish the true nature, direction and strength of the relationships between organisational characteristics, the use of digital technology and clinical outcomes. This evidence is currently lacking, but is crucial to support an evidence-based approach to digital technology in order to maximise its’ potential impact and provide value for money when making decisions about future investment.

This study has suggested that aspects of organisational digital maturity may play a role in influencing clinical outcomes for patients in hospitals in England; these results, however, must be qualified within the limitations of the study. Limitations of the study include the use of routinely collected data and the pragmatic approach to analysis; the key limitation being the inability to attribute direct causality and only infer association. Determinants of clinical outcomes at a hospital level are numerous and complex, and consider a wide range of inter-related variables. It is therefore inevitably challenging to establish causal inference and account for all confounders that may be present. Routinely collected administrative data were used to allow a detailed exploration of hospital performance and quality across a number of levels of care with a high degree of accuracy.41,42 Although a range of plausible covariates were selected to provide the best estimates for any underlying relationship between organisational digital maturity and clinical outcomes, we were unable to account for the full range of potential confounders. Although our outcome measures were risk-adjusted, our patient-level controls were limited. Quantifying organisational digital maturity is also challenging, and the CDMI Tool has some limitations. A number of varied approaches to evaluating organisational digital maturity have been proposed,43,44,45 but only a limited number use a comprehensive assessment framework,46 such as that used by the CDMI tool. The CDMI tool has an inherent risk of reporting bias, given it is self-reported. There is also a possibility that the CDMI tool fails to pick up the key aspects of digital maturity that have a significant impact on outcomes, whilst simultaneously measuring important aspects that have no influence, such as investment in security; the failure to find a persisting significant relationship between digital maturity and clinical outcomes may be due to gaps in the CDMI measure rather than the absence of a relationship. In addition, the benefits from health IT investment may only be seen at the higher end of the maturity spectrum, and so will only be achievable for a small number of hospitals studied limiting the overall findings. Finally, the return on investment from health IT may take many years to become apparent and so may not be detected in the time-limited data examined.

Using nationally representative data from 136 hospitals in England, this study has shown that routine administrative data have the potential to provide valuable information to inform an evidence-based approach to health IT investment in secondary care. A measure of organisational digital maturity—the NHS Clinical Digital Maturity Index—has been shown to be significantly associated with patients receiving harm-free care, but importantly is also significantly associated with an increased risk of a long length of stay. Digital maturity is however also strongly related to other institutional factors and measures of quality, and so ascertaining a direct causal relationship with clinical outcomes is challenging. There is no apparent relationship between digital maturity and complications of care, readmission rates nor overall risk-adjusted mortality. Whilst digital maturity has a significant role to play in the delivery of high-quality care, other institutional factors play a greater and more significant role in influencing clinical outcomes with digital maturity plausibly acting as a proxy for well-run, high-performing organisations delivering complex care. The effective use of health IT can undoubtedly lead to improvements in care quality, however, to maximise its potential we must develop a deeper understanding about how it impacts care delivery in holistic real-world settings and how it acts to support and enable other drivers of hospital quality.

Methods

Study population

The data were collected from all 136 non-specialist NHS Trusts in England that provide acute care over a 12-month period from January 2015 to January 2016. This 12-month period covered data for a total of 13,105,996 emergency and planned admissions. The data were collected from multiple sources covering the same 12-month period to ensure a temporal relationship between all included data. The data collected included Clinical Digital Maturity Index (CDMI) scores from 2016 for each organisation, five specified outcome variables of interest and a further 15 relevant variables pertaining to staffing, infrastructure and workload. The quality and completeness of the data were good with missing data accounting for <0.2% of all items included. The data is non-identifiable, and was collected from publicly available administrative datasets; therefore, no specific ethical approval or patient-level informed consent was required. The Dr Foster Unit at Imperial College London has approval from the UK Health Research Authority to hold and analyse the data for research purposes (Ref. 15/CAG/0005).

The NHS Clinical Digital Maturity Index

There are multiple ways to assess digital maturity.11 The Clinical Digital Maturity Index (CDMI) is a mandatory national benchmarking tool for NHS organisations in England that provides an objective assessment of their organisational digital maturity. Through a self-assessment framework run by NHS Digital—the national provider of IT and data services for the NHS in England—comprising 133 individual questions, with organisations reporting their digital maturity across three main themes—readiness, capability and infrastructure—together with an overall aggregate maturity score.47 Readiness scores refer to an organisation’s ability to plan, deliver and optimise its digital systems (e.g., leadership, governance and strategic alignment), the Capability score is an assessment of the digital capabilities that are available to an organisation (e.g., electronic ordering or medicines management) and the Infrastructure score evaluates the extent to which essential infrastructure is in place to support the delivery of their required digital capabilities (e.g., Wifi provision or single sign-on). Overall maturity scores are awarded on a scale of 0–1400 and vary from 324 to 1253 across the organisations studied with a mean score of 797. The CDMI tool, despite not having been formally validated is an integral part of the digital agenda in the English NHS and is an evidence-based, broad and holistic measure that is unique in its universal assessment of an entire health systems digital maturity compared with other methods.11 The tool provides a robust, evidence-based and standardised assessment that allows for the direct comparisons between different organisations to be made. The aim of the tool is to allow organisations to benchmark their performance against their peers, and identify key areas for development and improvement. The tool is summarised in the online Supplementary Information.

Outcomes measures

All outcome data used were based upon publicly available hospital episode statistics and routine administrative data obtained from NHS Digital48 and the Dr Foster Unit49 at Imperial College London, aggregated at the level of the hospital. All the data are collected, maintained and are publicly accessible through NHS Digital. The quality and completeness of the data were good, with <1% of data missing. As previously highlighted, the better use of health IT in secondary care has been linked to fewer complications and lower mortality rates.5,9,14 In addition, the adoption of widely used and well-understood outcome measures widens the generalisability and usefulness of any findings presented. Therefore, five clinical outcomes that may be plausibly associated with digital maturity and which are widely utilised in health services research were selected. Risk-adjustment is performed at a patient level and is derived from sets of logistic regression models that include the available case-mix factors based on complete national-level datasets; the national expectation of an event acting as the comparator50:

  • Summary Hospital-Level Mortality Index (SHMI)—a risk-adjusted measure of deaths in hospital or within 30 days of discharge.51,52

  • Relative risk of readmission (READ)—the number of emergency readmissions within 28-days of discharge from hospital.53,54

  • Relative risk of long length of stay (LLOS)—the number of hospital episodes associated with a risk-adjusted length of stay exceeding the upper quartile for all patients nationally with that episode type.55,56

  • Percentage of harm-free patient care episodes (HFC)—provision of harm-free care (the absence of pressure ulcers, falls, catheter associated urinary tract infections (CAUTI’s) and venous thromboembolism (VTE)) across each day of a hospital admission via the NHS Safety Thermometer.57

  • Percentage of patient episodes featuring complications of care (COC)—a composite quality measure based on the occurrence of potentially preventable safety events as defined by the Agency for Healthcare Research and Quality (AHRQ) risk-adjusted quality indicators.58,59

Hospital covariates

In total, 15 covariates pertaining to staffing, infrastructure and workload that were identified as potentially influencing outcomes were selected a priori in order to control for potential confounders of the relationship between digital maturity and the above outcome measures. There is a firm body of evidence that staffing levels—be that total staff numbers, variation in staffing levels, skills mix or access to senior clinicians—have a significant impact on clinical outcomes.23,24,26,60,61,62,63,64,65,66,67 Hospital characteristics such as bed numbers, academic or teaching hospital status and quality of non-digital infrastructure such as increased capital expenditure or access to better non-digital technology have also be shown to directly influence patient outcomes.26,30,32,67,68,69,70,71 Finally, the impact of workload, volume and bed occupancy across a range of settings has also been shown to impact outcomes.25,27,28,64,68,69,72,73,74,75,76 The data were collated from publicly available datasets published by NHS Digital.48 Academic hospital status was defined as membership of the Association of UK University Hospitals.77 The 15 covariates selected were: the total number of staff, total number of clinical staff, clinical/non-clinical staffing ratio, doctor/nurse staffing ratio, consultant/junior staffing ratio, manager/clinical staffing ratio, doctor/bed number ratio, nurse/bed number ratio, total number of adult inpatient beds, total number of adult critical care beds, academic hospital status, number of A&E attendances, number of inpatient admissions, mean inpatient bed occupancy and mean critical care bed occupancy; these are summarised in Table 1.

Statistical analysis

All variables were assessed for normality with no further transformations deemed to be required. All outcome variables were treated as continuous variables. Standard descriptive statistics for all variables were applied.

Each of the panel of 15 hospital-specific variables was examined in univariate regression against each of the hospital-level outcome measures and all domains of CDMI. Univariate regression was also performed for each hospital-level outcome measure against each of the three hospital-level domains of CDMI and hospital-level aggregate CDMI score. A summary of these univariate associations is provided in the online Supplementary Table 1.

All variables were subsequently examined for multicollinearity using the variance inflation factor (VIF), and three variables were removed from the final multivariable model due to high collinearity; nurse/bed ratio, the total inpatient admissions and total staff numbers. Twelve remaining variables from the panel with a significant relationship to either the outcome variables or CDMI scores and low collinearity were included in the final multivariable models as plausible confounders for the relationship between CDMI score and clinical outcomes. Multivariable linear regression featuring the panel of retained hospital-specific covariates was then performed for each of the selected outcome measures. Statistical significance was set at p = < 0.05. All statistical analyses were performed in Stata V15 (StataCorp LLC, College Station, TX, USA).

Data availability

The full dataset and other relevant material used and analysed in this study are available from the corresponding author upon request.

Code availability

All statistical analyses were performed in Stata V15 (StataCorp LLC, College Station, TX, USA). The relevant code is available from the corresponding author upon request.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Buntin, M., Burke, M., Hoaglin, M. & Blumenthal, D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff. 30, 464–471 (2011).

  2. 2.

    Committee on Improving the Patient Record & Medicine, I. of. The Computer-Based Patient Record: An Essential Technology for Healthcare. (National Academies Press, Washington, 1997).

  3. 3.

    Bates, D. & Gawande, A. Improving safety with information technology. N. Engl. J. Med. 348, 2526–2534 (2003).

  4. 4.

    Jones, S., Rudin, R., Perry, T. & Shekelle, P. Health information technology: an updated systematic review with a focus on meaningful use. Ann. Intern. Med. 160, 48–54 (2014).

  5. 5.

    Furukawa, M., Eldridge, N., Wang, Y. & Metersky, M. Electronic health record adoption and rates of in-hospital adverse events. J. Patient Saf. 1–6 (2016).

  6. 6.

    Poissant, L., Pereira, J., Tamblyn, R. & Kawasumi, Y. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J. Am. Med. Inform. Assoc. 12, 505–516 (2005).

  7. 7.

    Kawamoto, K. & Lobach, D. Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior. AMIA… Annu. Symp. Proc. 361–365 (2003). http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1480005&tool=pmcentrez&rendertype=abstract

  8. 8.

    Nguyen, L., Bellucci, E. & Nguyen, L. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int. J. Med. Inform. 83, 779–796 (2014).

  9. 9.

    Amarasingham, R., Plantinga, L., Diener-West, M., Gaskin, D. & Powe, N. Clinical information technologies and inpatient outcomes: a multiple hospital study. Arch. Intern. Med. 169, 108–114 (2009).

  10. 10.

    Appari, A., Carian, E. K., Johnson, M. & Anthony, D. Medication administration quality and health information technology: a national study of US hospitals. J. Am. Med. Inform. Assoc. 19, 360–367 (2012).

  11. 11.

    Flott, K., Callahan, R., Darzi, A. & Mayer, E. A patient-centered framework for evaluating digital maturity of health services: a systematic review. J. Med. Internet Res. 18, e75 (2016).

  12. 12.

    Parente, S. & McCullough, J. Health information technology and patient safety: evidence from panel data. Health Aff. 28, 357–360 (2009).

  13. 13.

    Gottschalk, P. Maturity levels for interoperability in digital government. Gov. Inf. Q. 26, 75–81 (2009).

  14. 14.

    Chaudhry, B. et al. Systematic review: impact of health information technology on quality, efficiency and costs of medical care. Ann. Intern. Med. 144, 742–752 (2006).

  15. 15.

    Neumeier, H., Berner, E. S., Burke, D. E. & Azuero, A. Hospital budget increase for information technology during phase 1 of meaningful use. Health Care Manag. (Frederick). 34, 157–165 (2015).

  16. 16.

    Slight, S. P. et al. Meaningful use of electronic health records: experiences from the field and future opportunities. J. Med. Internet Res. 3, e30 (2015).

  17. 17.

    Blumenthal, D. Wiring the health system - origins and provisions of a new federal program. N. Engl. J. Med. 365, 2323–2329 (2011).

  18. 18.

    OECD. Health at a Glance 2017. https://doi.org/10.1787/health_glance-2017-en (2017).

  19. 19.

    Bradley, E., Sipsma, H. & Taylor, L. American health care paradox - high spending on health care and poor health. QJM 110, hcw187 (2016).

  20. 20.

    Dickman, S., Himmelstein, D. & Woolhandler, S. Inequality and the health - care system in the USA. Lancet 389, 1431–1441 (2017).

  21. 21.

    Sittig, D. & Singh, H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual. Saf. Healthc. 19(Suppl 3), i68–i74 (2010).

  22. 22.

    Greenhalgh, T. & Keen, J. England’s national programme for IT. Br. Med. J. 346, f4130 (2013).

  23. 23.

    Tvedt, C., Sjetne, I., Helgeland, J. & Bukholm, G. An observational study: associations between nurse-reported hospital characteristics and estimated 30-day survival probabilities. BMJ Qual. Saf. 23, 757–764 (2014).

  24. 24.

    Chung, W. & Sohn, M. The impact of nurse staffing on in-hospital mortality of stroke patients in Korea. J. Cardiovasc. Nurs. 33, 1 (2017).

  25. 25.

    Amato, L. et al. Volume and health outcomes: evidence from systematic reviews and from an evaluation of Italian hospital data. Epidemiol. Prev. 37(Suppl 2), 3–10 (2013).

  26. 26.

    Ghaferi, A., Osborne, N., Birkmeyer, J. & Dimick, J. Hospital characteristics associated with failure to rescue from complications after pancreatectomy. J. Am. Coll. Surg. 211, 325–330 (2010).

  27. 27.

    Madsen, F., Ladelund, S. & Linneberg, A. High levels of bed occupancy associated with increased inpatient and thirty-day hospital mortality in Denmark. Health Aff. 33, 1236–1244 (2014).

  28. 28.

    Eriksson, C., Stoner, R., Eden, K., Newgard, C. & Guise, J. The association between hospital capacity strain and inpatient outcomes in highly developed countries: a systematic review. J. Gen. Intern. Med. 32, 686–696 (2017).

  29. 29.

    Himmelstein, D., Wright, A. & Woolhandler, S. Hospital computing and the costs and quality of care: a national study. Am. J. Med. 123, 40–46 (2010).

  30. 30.

    Burke, L., Frakt, A., Khullar, D., Orav, E. J. & Jha, A. Association between teaching status and mortality in US Hospitals. J. Am. Med. Assoc. 317, 2105–2113 (2017).

  31. 31.

    Jha, A. et al. Use of electronic health records in U.S. hospitals. N. Engl. J. Med. 360, 1628–1638 (2009).

  32. 32.

    Shahian, D. et al. Contemporary performance of U.S. teaching and non-teaching hospitals. Acad. Med. 87, 701–708 (2012).

  33. 33.

    Haut, E. et al. Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerised clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch. Surg. 147, 901–907 (2012).

  34. 34.

    Amland, R. et al. Computerized clinical decision support to prevent venous thromboembolism among hospitalized patients: proximal outcomes from a multiyear quality improvement project. J. Healthc. Qual. 37, 221–231 (2015).

  35. 35.

    Cornia, P., Amory, J., Fraser, S., Saint, S. & Lipsky, B. Computer-based order entry decreases duration of indwelling urinary catheterization in hospitalized patients. Am. J. Med. 114, 404–407 (2003).

  36. 36.

    Topal, J. et al. Prevention of nosocomial catheter-associated urinary tract infections through computerized feedback to physicians and a nurse-directed protocol. Am. J. Med. Qual. 20, 121–126 (2005).

  37. 37.

    Kaewprag, P. et al. Predictive models for pressure ulcers from intensive care unit electronic health records using Bayesian networks. BMC Med. Inform. Decis. Mak. 17, 65 (2017).

  38. 38.

    Jin, Y., Jin, T. & Lee, S. M. Automated pressure injury risk assessment system incorporated into an electronic health record system. Nurs. Res. 66, 462–472 (2017).

  39. 39.

    Lee, J. Y., Jin, Y., Piao, J. & Lee, S.-M. Development and evaluation of an automated fall risk assessment system. Int. J. Qual. Heal. Care 28, 175–182 (2016).

  40. 40.

    Cho, I. & Chung, E. Effect of automatic inpatient fall prediction using routinely captured EMR data: Preliminary results. Stud. Health Technol. Inform. 225, 828–829 (2016).

  41. 41.

    Zhan, C. & Miller, M. Administrative data based patient safety research: a critical review. Qual. Saf. Healthc. 12, ii58–ii63 (2003).

  42. 42.

    Campbell, S., Cambell, M., Grimshaw, J. & Walker, A. A systematic review of discharge coding accuracy. J. Public Health Med. 23, 205–211 (2001).

  43. 43.

    Vest, J. & Jasperson, S. How are health professionals using health information exchange systems? Measuring usage for evaluation and system improvement. J. Med. Syst. 36, 3195–3204 (2011).

  44. 44.

    Bassi, J. & Lau, F. Measuring value for money: a scoping review on economic evaluation of health information systems. J. Am. Med. Inform. Assoc. 20, 792–801 (2013).

  45. 45.

    Johnson, K. & Gadd, C. Playing smallball: approaches to evaluating pilot health information exchange systems. J. Biomed. Inform. 40, 21–26 (2007).

  46. 46.

    Hripcsak, G. et al. The United Hospital Fund meeting on evaluating health information exchange. J. Biomed. Inform. 40, 3–10 (2007).

  47. 47.

    N. H. S. England. N. H. S. England launches Clinical Digital Maturity Index to improve patient safety. https://www.england.nhs.uk/2013/11/cdmi/ (2013).

  48. 48.

    NHS Digital. NHS Digital Data and Publications Index. https://digital.nhs.uk/article/6676/Publications-by-topic (2016).

  49. 49.

    Unit, D. F. Dr Foster Unit. https://www.imperial.ac.uk/dr-foster-unit (2017).

  50. 50.

    Bottle, A., Jarman, B. & Aylin, P. Strengths and weaknesses of hospital standardised mortality ratios. Br. Med. J. 342, c7116 (2010).

  51. 51.

    Campbell, M., Jacques, R., Fotheringham, J., Maheswaran, R. & Nicholl, J. Developing a summary hospital mortality index: retrospective analysis in English hospitals over five years. Br. Med. J. 344, e1001 (2012).

  52. 52.

    NHS Digital. Summary Hospital-Level Mortality Indicator (SHMI). https://digital.nhs.uk/SHMI (2016).

  53. 53.

    Joynt, K. E. & Jha, A. K. Thirty-day readmissions – truth and consequences. N. Engl. J. Med. 366, 1366–1369 (2012).

  54. 54.

    Fischer, C. et al. Is the readmission rate a valid quality indicator? A review of the evidence. PLoS ONE 9, 1–9 (2014).

  55. 55.

    Bottle, A. & Aylin, P. Intelligent information: a national system for monitoring clinical performance. Health Serv. Res. 43, 10–31 (2008).

  56. 56.

    Borghans, I. et al. Unexpectedly long hospital stays as an indicator of risk of unsafe care: an exploratory study. BMJ Open 4, 4–7 (2014).

  57. 57.

    NHS Improvement. NHS Safety Thermometer. https://www.safetythermometer.nhs.uk/index.php (2017).

  58. 58.

    Agency for Healthcare Research and Quality (AHRQ). AHRQ Quality Indicators. http://www.qualityindicators.ahrq.gov/Default.aspx (2017).

  59. 59.

    Agency for Healthcare Research and Quality (AHRQ). PSI 90 Fact Sheet. http://www.qualityindicators.ahrq.gov/News/PSI90_Factsheet_FAQ_v1.pdf (2016).

  60. 60.

    Pronovost, P. et al. Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. J. Am. Med. Assoc. 288, 2151–2162 (2002).

  61. 61.

    Butler, M. et al. Hospital nurse staffing models and patient and staff-related outcomes. Cochrane Database Syst. Rev. https://doi.org/10.1002/14651858.CD007019 (2011).

  62. 62.

    Chau, J. et al. A longitudinal examination of the association between nurse staffing levels, the practice environment and nurse-sensitive patient outcomes in hospitals. BMC Health Serv. Res. 15, 1–8 (2015).

  63. 63.

    Needleman, J. et al. Nurse staffing and inpatient hospital mortality. N. Engl. J. Med. 364, 1037–1045 (2011).

  64. 64.

    Schilling, P., Campbell, D., Englesbe, M. & Davis, M. A comparison of in-hospital mortality risk conferred by high hospital occupancy, differences in nurse staffing levels, weekend admission, and seasonal influenza. Med. Care 48, 224–232 (2010).

  65. 65.

    Price, L. et al. UK National COPD Audit 2003: impact of hospital resources and organisation of care on patient outcome following admission for acute COPD exacerbation. Thorax 61, 837–842 (2006).

  66. 66.

    Bell, D., Lambourne, A., Percival, F., Laverty, A. & Ward, D. Consultant input in acute medical admissions and patient outcomes in hospitals in England: a multivariate analysis. PLoS ONE 8, 2–6 (2013).

  67. 67.

    Ozdemir, B. et al. Association of hospital structures with mortality from ruptured abdominal aortic aneurysm. Br. J. Surg. 102, 516–524 (2015).

  68. 68.

    Jarman, B. et al. Explaining differences in English hospital death rates using routinely collected data. Br. Med. J. 318, 1515–1520 (1999).

  69. 69.

    Hoehn, R. et al. Hospital resources are associated with value-based surgical performance. J. Surg. Res. 204, 15–21 (2016).

  70. 70.

    Juillard, C. et al. A national analysis of the relationship between hospital volume, academic center status, and surgical outcomes for abdominal hysterectomy done for leiomyoma. J. Am. Coll. Surg. 208, 599–606 (2009).

  71. 71.

    Judge, A., Chard, J., Learmonth, I. & Dieppe, P. The effects of surgical volumes and training centre status on outcomes following total joint replacement: Analysis of the Hospital Episode Statistics for England. J. Public Health (Bangk.). 28, 116–124 (2006).

  72. 72.

    McCusker, J., Vadeboncoeur, A., Lévesque, J. F., Ciampi, A. & Belzile, E. Increases in emergency department occupancy are associated with adverse 30-day outcomes. Acad. Emerg. Med. 21, 1092–1100 (2014).

  73. 73.

    Blom, M., Jonsson, F., Landin-Olsson, M. & Ivarsson, K. The probability of patients being admitted from the emergency department is negatively correlated to in-hospital bed occupancy - a registry study. Int. J. Emerg. Med. 7, 1–7 (2014).

  74. 74.

    Gonzalez, A., Dimick, J., Birkmeyer, J. & Ghaferi, A. Understanding the volume - outcome effect in cardiovascular surgery: the role of failure to rescue. JAMA Surg. 149, 119–123 (2014).

  75. 75.

    King, W. et al. Equity in healthcare for coronary heart disease, Wales (UK) 2004–2010: A population-based electronic cohort study. PLoS ONE 12, e0172618 (2017).

  76. 76.

    The UK Neonatal Staffing Study Group. Patient volume, staffing, and workload in relation to risk-adjusted outcomes in a random stratified sample of UK neonatal intensive care units: A prospective evaluation. Lancet 359, 99–107 (2002).

  77. 77.

    Association of UK University Hospitals. Association of UK University Hospitals. http://www.aukuh.org.uk (2016).

Download references

Acknowledgements

This paper represents independent research supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre (PSTRC) at Imperial College London and Imperial College Healthcare NHS Trust. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Affiliations

  1. Department of Surgery & Cancer, Imperial College London, London, UK

    • Guy Martin
    • , Jonathan Clarke
    • , Sonal Arora
    • , Dominic King
    •  & Ara Darzi
  2. School of Public Health, Imperial College London, London, UK

    • Felicity Liew
    •  & Paul Aylin
  3. DeepMind, London, UK

    • Dominic King

Authors

  1. Search for Guy Martin in:

  2. Search for Jonathan Clarke in:

  3. Search for Felicity Liew in:

  4. Search for Sonal Arora in:

  5. Search for Dominic King in:

  6. Search for Paul Aylin in:

  7. Search for Ara Darzi in:

Contributions

G.M., J.C., F.L., S.A., D.K., P.A. and A.D. all contributed to the article. G.M., J.C. and F.L. conducted the majority of the study. G.M. wrote the paper with contributions from J. C., F.L., S.A., D.K., P.A. and A.D. A.D. was responsible for overall supervision, and is the guarantor for the study. All authors have access to the data and can take responsibility for the integrity of the data and accuracy of the data analysis.

Competing interests

The authors declare no competing interests.

Corresponding author

Correspondence to Guy Martin.

Supplementary information

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41746-019-0118-9