Introduction

Acute kidney injury (AKI) is now recognized as an important outcome for pediatric patients, affecting short term morbidity and mortality and long term renal function.1,2,3,4,5,6,7,8,9 AKI incidence is estimated to be 27–42% in large pediatric intensive care unit (ICU) cohorts and at least 5% in non-ICU admissions, although accurate estimation of AKI incidence in children is limited by the lack of serial SCr measurements.10,11,12 There is enthusiasm to leverage electronic health records (EHRs) to increase detection and recognition of AKI.13,14,15,16,17 One strategy has been to implement AKI alerts to ensure that clinicians are aware when a patient meets diagnostic criteria for AKI, typically based on SCr criteria.18,19,20,21,22,23,24 While a subset of these tools have demonstrated efficacy, they are only relevant for patients with multiple SCr measurements. For pediatric patients, baseline or pre-hospitalization SCr levels are typically not available, and inpatient screening of SCr is often not done, particularly for non-critically ill patients.12 Thus, a second strategy for improving AKI outcomes has been to initiate targeted screening by SCr measurement in at-risk patients.25,26,27 An early effort to conduct SCr screening in children treated with nephrotoxic medications increased AKI detection and reduced AKI severity.25

We previously developed and validated risk prediction models to identify pediatric ICU and ward patients who are at risk of AKI even in the absence of SCr measurements.28 Here we report the results of a pragmatic, hospital-wide, randomized trial conducted to determine if an automated AKI risk alert based on our predictive models would increase AKI screening in pediatric inpatients. As secondary outcomes, we also assessed AKI incidence, AKI severity, length of stay (LOS) and mortality.

Patients and methods

Study design and power calculation

This study used a pragmatic randomized clinical trial design at the Monroe Carrell Jr. Children’s Hospital at Vanderbilt. This study was approved by the Vanderbilt University Medical Center (VUMC) Institutional Review Board with waiver of informed consent and registered on ClinicalTrials.gov (number NCT02660931). The datasets used to build and validate the predictive models were used to determine sample size. The AKI risk threshold was set at 50% to significantly enrich for AKI. At this risk threshold, 423 at-risk ICU admissions per arm resulted in 84% power to detect a 10% difference in screening and would accrue in ~12 months. For the pediatric ward, 445 at-risk admissions per arm provided 89% power to detect a 10% screening rate difference, again requiring ~12 months of accrual. AKI risk prediction data and randomization status were collected during an initial period from 10/3/2016 through 12/5/2016, but no interventions were delivered at that time due to technical issues causing failure of the alert to display in the EHR; these data were used as a study run-in period. The randomized trial with display of the alert was conducted from 12/6/2016 to 11/1/2017. An interim analysis, blinded to randomization group, was performed 6 months after the trial start and reviewed by the Vanderbilt Institute for Clinical and Translational Research Learning Healthcare System team. The interim analysis revealed no significant differences between groups in baseline characteristics, primary outcomes, or secondary outcomes, so the trial was continued to the planned end date.

Study populations and randomization

Inclusion criteria for the study were age 28 days through 21 years and admission beginning during the study period to the pediatric medical or cardiac ICU, or the inpatient ward. Exclusion criteria were the same as those used to develop the AKI risk prediction models,28 namely neonatal ICU admission or prior ICD or CPT codes indicating chronic kidney disease (Supplemental Table S1 (online)). Admissions were assigned to either the ICU cohort or the hospital ward cohort based on location, and the appropriate risk prediction model was applied (ICU or ward), as was done for risk model development and validation.28 Any admissions to the ICU subsequently transferred to the inpatient ward were retained in the ICU cohort through discharge to allow continued assessment of ICU-related risk factors. Any admissions to the inpatient ward subsequently transferred to the ICU were included in the ward cohort for the portion of the admission while they were on the ward (with risk calculated using the ward prediction model); the portion of the admission from the time of transfer to the ICU through discharge were included in the ICU cohort (with risk calculated using the ICU prediction model). Each admission was randomized to the intervention group or the control group in a ~1:1 ratio based on the patient’s medical record number (odd vs. even final digit); analysis of historical data from the risk model development cohorts indicated no differences in demographic variables based on this assignment. If a patient was admitted multiple times, their admissions were always assigned to the same study group, and each admission was included in the study. Inpatient care providers were not privy to randomization status or the randomization method. However, once an alert was displayed during an admission, the provider knew that the admission was randomized to the intervention group.

Acute kidney injury risk prediction models

The EHR-based risk prediction models for AKI have been described.28 In brief, using EHR data from pediatric ICU patients, we developed and validated a statistical model to predict AKI risk during ICU hospitalization based on 10 characteristics (age, high-risk nephrotoxins, moderate risk nephrotoxins, total number of medications, platelet count, red blood cell distribution width, phosphorus, transaminases, pH, and hypotension). A second model was generated for pediatric ward patients using eight characteristics (as above, without pH or hypotension). Throughout each admission in the intervention and control groups, the risk of AKI was calculated and incorporated into the EHR.

For this study, we developed an order entry advisor triggered when the risk score provided by the AKI prediction model was greater than the predetermined threshold. The AKI risk prediction model was developed into a custom clinical decision support (CDS) service integrated with VUMC’s Enterprise Service Bus (ESB). The ESB serves as an integration platform for data generated by all clinical systems and exposes data elements to be consumed by custom CDS applications. CDS implementation for AKI risk utilized a complex event-processing framework where clinical events, represented by discrete data flowing through the ESB, initiate surveillance and update CDS results in real-time as new data elements are available on the ESB. The surveillance process for AKI risk calculation was initiated at the time of patient admission and consumed all data elements defined as predictive model covariates as they were released to ESB (through web services or real-time HL7 interfaces). The AKI risk scores, calculated and updated in real-time, were subsequently populated into patients’ charts to trigger the order entry workflows for the intervention.

Clinical decision support

Pediatric ICU and ward admissions randomized to the intervention were eligible for a CDS alert to be displayed to providers (attending physicians, residents, fellows, and advanced practice nurses) at the time of completion of any order entry session. The CDS alert (Fig. 1) informed providers that the patient was at increased risk for AKI and displayed options of ordering a basic metabolic panel test (BMP, including serum sodium, potassium, chloride, carbon dioxide, glucose, blood urea nitrogen, calcium and creatinine) immediately or the following morning. Providers could also choose to override the CDS recommendations by choosing “Decline ordering BMP,” which silenced the alert for all providers for the following 24 h, or by clicking the “X” in the top right corner, which would close the alert but not silence the alert for future order entry sessions. Providers could state the reason for declining to order a BMP in an optional text box. These free-text responses were categorized by the study PI.

Fig. 1
figure 1

Acute Kidney Injury Risk Alert. The alert was shown at the end of order entry if the admission was randomized to the intervention group, the calculated acute kidney injury risk exceeded 50%, the alert had not been acknowledged in the prior 24 h, and no serum creatinine test result was available from the prior 24 h or scheduled to be obtained within 24 h. BMP—basic metabolic profile, including serum sodium, potassium, chloride, carbon dioxide, glucose, blood urea nitrogen, calcium, and creatinine

Full size image

The CDS alert was displayed if all the following conditions were met: (1) Admission randomized to the intervention group; (2) Current calculated risk of AKI > 50%; (3) No SCr results from the previous 24 h performed at VUMC; (4) No active SCr order scheduled within the next 24 h; and (5) No provider selected “Decline ordering BMP” in the past 24 h. Selection of “Order a BMP for NOW” or “Order a BMP for AM” brought providers to the laboratory test order entry screen, with pre-filled selections matching their choice. Further information about the randomized trial was available to providers via a link on the alert screen. All clinical providers in the pediatric ICU and ward were notified of the randomized trial prior to the start via brief in-person presentations and a summary via email.

Primary and secondary outcomes

The primary outcome was SCr testing within 48 h of the first instance of calculated AKI risk > 50%. The secondary outcomes of LOS and mortality were compared in the intervention vs control groups among these at-risk admissions, and among all randomized admissions. We also compared the incidence of AKI in the intervention group vs. controls for the subset of admissions with at least two SCr measurements. AKI was defined using Kidney Disease Improving Global Outcomes (KDIGO) SCr criteria (SCr increase of 1.5-fold or by 0.3 mg/dL from baseline during the admission), without requiring the increase to occur within 7 days or 48 hours, respectively.29 For all admissions with AKI, highest AKI severity was compared in the intervention vs. control groups, using KDIGO definitions for Stage 1 (1.5-fold or 0.3 mg/dL SCr increase), Stage 2 (two-fold increase), and Stage 3 (three-fold increase or SCr ≥ 4 mg/dL) AKI.

Statistical analyses

Continuous variables were compared using the Wilcoxon rank sum test, categorical variables using the Pearson χ2 test, and ordinal variables using the proportional odds likelihood ratio test. All p-values were based on two-sided tests, and statistical significance was determined at the level of 0.05. Statistical analyses were completed using R, version 3.3.0 (R Development Core Team, Vienna, Austria).

Results

Cohorts

Between 12/6/2016 and 11/1/2017, 1909 ICU admissions and 10,997 ward admissions were randomized for the trial (Fig. 2). Baseline data are shown in Table 1. There were no differences by intervention vs. control groups when comparing median or maximum AKI risk for each admission for the pediatric ICU or ward admissions cohorts. Characteristics of the 2470 admissions during the study run-in period are presented in Supplemental Table S2 (online).

Fig. 2
figure 2

Study Cohorts. Admissions to the PICU and inpatient ward were included. Individuals with known CKD were excluded. Admissions were assigned to the ward or PICU cohorts for AKI risk calculation and for study analysis based on patient location. Admissions that remained in ward units for the duration of the hospitalization were assigned to the ward cohort (n = 10,822). Those initially admitted to the PICU were assigned to the PICU cohort and retained in the ICU cohort for the duration of the admission, even if the patient was transferred to the inpatient ward (n = 1734). Admissions initially located in the inpatient ward but then transferred to the ICU (n = 175) were included in the ward cohort from admission until transfer to the ICU (the duration of their ward stay), then included in the ICU cohort from transfer until discharge. The primary outcome (SCr screening within 48 h of AKI risk > 50%) was evaluated among those identified as at risk for AKI (AKI risk > 50%). LOS and mortality were assessed among those at risk, and among all randomized patients. AKI incidence and AKI severity were assessed in those admissions with at least 2 SCr measurements, enabling determination of AKI status. aAdmissions to the ward later transferred to the PICU had risk calculation based on the ward model until transfer, and data through the time of transfer are included in the ward cohort; after transfer to the PICU, risk was calculated using the PICU model, and the portion of the admission from transfer through discharge is included in the PICU cohort. bAdmissions were included for assessment of AKI outcomes based on presence of 2 SCr measurements, regardless of AKI risk. CKD—Chronic Kidney Disease; PICU—Pediatric Intensive Care Unit; SCr—Serum Creatinine; LOS—Length of Stay; AKI—Acute Kidney Injury

Full size image
Table 1 Baseline characteristics of study cohorts
Full size table

Clinical decision support

The AKI risk alert was displayed 127 times for 68 admissions in the pediatric ICU. In 65 instances (51%), the provider chose to decline ordering a BMP. Reasons for declining were provided in 20 instances, with the most common being that the patient had a recent SCr result or had low risk for AKI (Fig. 3a). A BMP was ordered for the next morning in 31 instances (24%) and ordered immediately in 12 instances (9%); the BMP orders were in response to the first alert for the admission in 31 instances (72%), the second alert in 7 (16%), the third in 2 (5%), and the fourth or subsequent alert in 3 (7%). The alert was dismissed 19 times (15%). The AKI risk alert was displayed 435 times for 138 admissions in the pediatric ward. Providers declined ordering a BMP 282 times (65% of alert instances) and reasons for declining were provided in 97 instances, with recent results and perceived low risk again being the most common (Fig. 3b). Providers ordered a BMP for the morning 20 times (5%), ordered a BMP immediately 8 times (2%) and dismissed the alert 125 times (29%). BMP ordering was in response to the first alert in 11 instances (39%), the second in 8 (29%), the third in 2 (7%), and the fourth or subsequent in 7 (25%).

Fig. 3
figure 3

Reasons Given for Declining to Follow CDS Recommendation. The count of each type of reason given is shown in the graph. a For the pediatric ICU, reasons were entered for a total of 20 of CDS alert instances. b For the pediatric ward, reasons were given for a total of 97 CDS alert instances. *Other reasons included one instance each of “aware,” nephrology involved, and care per protocol

Full size image

Primary and secondary outcomes

Among pediatric ICU admissions with calculated AKI risk > 50%, SCr testing was obtained more often in the intervention group (418/606, 69.0%) than in controls (361/597, 60.5, Table 2). LOS and mortality were not different between groups (Table 2). When all randomized admissions to the ICU were analyzed, those in the intervention group had longer LOS than controls (median 3, interquartile range [1–6] vs. 3 [2–7] days, p = 0.05, Supplemental Figs. S1-S2 (online)), and there was no difference in mortality (21 (2%) vs. 18 (2%), p = 0.7). Among pediatric ICU admissions with at least two SCr measurements (enabling determination of AKI status) there was no difference in the incidence of AKI or AKI severity in the intervention vs. control groups (Table 3). Admissions with at least 1 inpatient SCr measurement were more likely to have an elevated risk of AKI than those with no inpatient SCr measurement for both the control and intervention groups (Supplemental Table S3 (online)). For admissions to the pediatric ward, there was no difference in SCr testing rates, LOS, mortality, AKI incidence, or AKI severity (Tables 2–3, Supplemental Figs. S1-S2 (online)).

Table 2 Primary and secondary outcomes among admissions ever exceeding 50% AKI risk
Full size table
Table 3 AKI incidence and severity among admissions with multiple SCr measurements
Full size table

Randomization was performed based on medical record number, and not based on providers, clinical services, or time. The same clinician may have cared for individuals in the intervention and control groups at the same time, and thus providers may change BMP ordering in the control group after seeing the alert in the intervention group (i.e. Hawthorne Effect).30 To determine the potential impact of the ongoing study on provider decisions to obtain SCr testing, we compared SCr testing for admissions randomized to the control group during the run-in period, when no alerts were displayed, to those during the trial. Among pediatric ICU admissions, there was no difference in SCr testing for controls during the run-in period vs. during the trial (117/210 (56%) vs. 539/936 (58%), respectively, p = 0.6). Similarly, there was no difference between these two groups for pediatric ward admissions (259/1,055 (25%) vs. 1283/5,505 (23%), p = 0.4).

Discussion

In this pragmatic, hospital-wide, randomized clinical trial, we demonstrate that the pop-up CDS alert for increased risk for AKI increased AKI screening in the pediatric ICU but not the inpatient ward. While there is often robust evidence supporting the content of CDS alerts, there are relatively little data on the efficacy of alerts to impact clinical decision making or clinical outcomes. Increasing frequency of CDS alerts can lead to increased alert override and “alert fatigue.”31,32,33 The pragmatic study design used here facilitated rapid assessment of the CDS alerts on the primary process outcome (SCr testing rate among those at-risk for AKI) and demonstrated one effective strategy to foster evidence-based CDS use in clinical settings. These strategies are necessary to promote maintenance of effective CDS and revision or retirement of ineffective CDS, as the latter contributes to alert fatigue without improving health outcomes.

Our data inform the future implementation of our AKI alerts and potentially other similar alerts. Specifically, among pediatric wards, where much of the prior work demonstrating improved outcomes with screening of at-risk patients has been performed, the CDS alert was not shown to be effective. The number of ward admissions exceeding the 50% risk of AKI was much lower than anticipated based on prior data, indicating a need for model re-calibration and limiting power to detect a difference between groups. In the small number of admissions identified as at-risk, we observed less SCr testing in the intervention group (without statistical significance); thus, there was no trend indicating that a larger sample size would demonstrate efficacy of the CDS to improve screening rates. Alternate approaches are indicated for identifying and screening those at risk for AKI in this setting, as our alert contributed only to alert fatigue and provided no benefit.

For pediatric ICU patients, where there are more robust data on AKI incidence but limited data on AKI screening, we show a marginal increase (9%) in SCr testing among those at risk for AKI in the intervention group, representing approximately 50 additional laboratory tests in the intervention group. Given this positive change in the process measure, the alert strategy may be effective in improving clinical outcomes for children. A much larger study is required to demonstrate that increased testing reduces AKI burden, LOS, or mortality.

The difference in effect of the CDS alert in the pediatric ICU vs. ward may be due to several factors. Patients admitted to the ICU have more severe illness than those on the ward and are at higher risk for AKI. Clinicians caring for these patients may be attuned to AKI risk and thus more likely to comply with the CDS guidance. The increased risk of AKI among ICU patients also increased our power to detect differences between the intervention and control groups. In addition, ICU patients undergo laboratory testing on a more frequent basis, increasing the likelihood of SCr testing for other reasons (e.g. with electrolyte testing) or to be “added on” to another clinical test (e.g. complete blood counts) without the need for an additional blood draw. For pediatric ward patients, the additional risk and cost associated with the laboratory test may have outweighed the potential risk of AKI in clinician decision making.

In both the ICU and ward cohort, the most common reason given for declining to order the recommended screening was that a recent result was available, followed by a clinical assessment that the patient was low risk. Additional targeted education of the providers and/or additional information in the alerts regarding the time course of AKI, the positive sequelae of early detection, and the validated AKI risk models may impact provider actions.

With respect to our secondary outcomes of AKI incidence and severity, we found no differences between intervention and control groups in the ICU cohort. Prior work has suggested that increased screening for AKI among at-risk pediatric patients can reduce AKI severity.25,34 Our study was not powered to detect differences in these secondary outcomes, and larger trials are required to determine the clinical impact of targeted screening programs. Of note, the time from admission to first instance of calculated risk > 50% was relatively late (median day 11–12 of admission for ICU and day 2–3 for ward admissions), with a wide distribution (Table 1). Much work has focused on risk prediction and AKI detection in the first week of hospitalization,11 but our data indicate that risk factors may accumulate later in the course of care.

A concern raised during study planning was the possibility of increased LOS due to abnormal laboratory values without clinical significance. Among admissions ever at > 50% AKI risk (i.e. admissions where the CDS may have been active), there were no differences in LOS. Among all randomized ICU admissions, the intervention group had longer stays than controls. Post-hoc analyses reveal that the difference is driven by those with the longest hospital stays (Supplemental Table S4 (online)), which are unlikely to be due to spurious laboratory values. If there is a small but real increase in LOS attributable to the CDS alert, widespread adoption of this strategy across health systems could result in a meaningful increase. Our data do not indicate whether the increased LOS seen across all hospital admissions randomized to the intervention is due to statistical chance, appropriate prolonged management of recognized renal disease, follow up of laboratory values, or some other factor(s).

This study has several limitations. This was a single institution study performed at an academic children’s hospital and may not be generalizable to other settings. Specifically, the AKI risk prediction models would require calibration based on local data, and specific implementation strategies would be dependent on the EHR system in place. This study excluded admissions to the neonatal ICU, due to the complexity of ascertaining AKI status in neonates. For admissions to the ward, calculated AKI risk was low, resulting in fewer than 400 ward admissions from which to assess our primary outcome. This not only indicates that the risk prediction model requires recalibration, since 5% of the ward patient population met AKI SCr criteria, but also resulted in insufficient power to assess the impact of the CDS intervention on the ward. We did not inform providers of the randomization status or randomization method, but providers could, in theory, determine the randomization scheme; in addition, once the CDS displayed for an admission, providers were aware of randomization status for that individual. The definition of AKI used to develop the risk prediction models and to define AKI as a secondary outcome relies only upon SCr values, and does not incorporate urine output, calculated glomerular filtration rate, or any other biomarker measurements. Furthermore, AKI was not prospectively assessed as a part of this study, thus the true incidence of AKI in the intervention and control groups is not known and our data may underestimate AKI incidence due to incomplete ascertainment.

Conclusion

These data indicate that a CDS alert may be effective in increasing screening for AKI among pediatric ICU patients, but was not effective for admissions to the pediatric ward. Larger studies are required to determine the impact of increased screening on clinical outcomes, such as AKI incidence and severity. This study also demonstrates the utility of pragmatic trials in prospectively assessing the effects and value of CDS across children’s hospital units, which may assist in improving alerts displayed to providers and eliminating alerts that are ineffective.

Disclosure

The authors have no financial relationships relevant to this article to disclose, and no potential/perceived conflicts of interest relevant to this article to disclose.