Main

Acute kidney injury (AKI) occurs in ~25% of children admitted to intensive care units (ICUs) and 5% of those admitted to non-ICU pediatric wards, although application of different diagnostic criteria results in broad incidence estimates (1, 2, 3, 4, 5, 6, 7, 8, 9, 10). Among pediatric ICU patients, AKI is associated with increased length of stay, and it demonstrates odds ratios >3 for mortality (1, 2, 8, 9, 11, 12). Seminal efforts to identify children with AKI through screening of at-risk populations have reduced AKI severity, likely due to early identification and modification of AKI risk factors (13, 14). Recently, the Acute Dialysis Quality Initiative Consensus Conference identified AKI as a target for risk prediction and recommended development of AKI alert systems that will “continuously and automatically monitor and assess a patient’s risk for developing AKI” (15, 16).

The primary goal of this study was to develop and validate AKI risk prediction models in pediatric ICU and non-ICU patients that use data from electronic health records (EHR) to identify patients in whom laboratory testing, namely measurement of serum creatinine to assess renal function, is indicated. Prediction models that can be calculated in real time in the EHR and those that do not rely upon prior serum creatinine values will enable AKI risk prediction across a broad pediatric population, including those not in well-described high-risk populations and for whom no serum creatinine data are available.

Methods

Population and EHR Data Extraction

This study was reviewed and approved by the Vanderbilt institutional review board who granted a waiver of the informed consent process. Inclusion criteria for the development cohort were age 28 days through 21 years, measurement of baseline and an additional inpatient serum creatinine as defined below, and admission between 1 January 2011 and 31 December 2012. Exclusion criteria were Neonatal ICU admission (based on location codes) and chronic kidney disease (based on ICD-9 and CPT codes, Supplementary Table S1 online), both of which can be applied prospectively in EHR systems (17). In addition, those not meeting the AKI definition but with persistent serum creatinine measurements above the normal range for age and sex were excluded, as it is unclear whether they represent AKI without a measured baseline or no-AKI controls. Admissions were assigned to the ICU cohort if the patient received ICU care at any time during the hospitalization; thus, a patient cared for in the ICU who subsequently was transferred to the floor was included in the ICU cohort for both the development and validation cohorts. Validation cohorts used admissions from 1 January 2013 to 31 December 2013 with identical inclusion and exclusion criteria. All study data, including demographic, laboratory, medication administration, and administrative data, were extracted from EHR data sources using Vanderbilt’s research data warehouse and were managed using REDCap (Research Electronic Data Capture) (18, 19). Medications were categorized as high-risk nephrotoxins if they are highly associated with renal injury (e.g., aminoglycosides, tacrolimus), moderate-risk nephrotoxins if they are associated with renal injury in some circumstances (e.g., loop diuretics, enalapril), or non-nephrotoxins (Supplementary Table S2). For each individual, the number of unique medications in each category was tallied, and three medication-related continuous variables were analyzed (high-risk nephrotoxins, moderate-risk nephrotoxins, and total medications). Peri-admission data included all the data available from 24 h before the inpatient admission order through 48 h after admission. For model development, covariate data were ascertained from 72 h before the serum creatinine measurement that defined AKI status. Further details are available in the Supplementary Materials.

Outcome Definition

Although serum creatinine is not a covariate in the risk prediction models, the AKI outcome is defined by change in creatinine (Supplementary Figure S1). Baseline creatinine was defined as the lowest measurement obtained 90 days before through the first week of admission. Inpatient creatinine was defined as any measurement obtained 24 h prior to the admission order through discharge. AKI was defined solely using the creatinine criteria, as reliable urine output data were not available for all pediatric inpatients (15), and accurate height measurements were not consistently available for estimated glomerular filtration rate. Patients were classified as having AKI, using the Kidney Disease Improving Global Outcomes (KDIGO) serum creatinine criteria, if any inpatient creatinine measurement was ≥0.3 mg/dl above the baseline or ≥1.5-fold more than the baseline, without the restriction that the increase in serum creatinine occurred within 48 h or 7 days, respectively, as defined in the guidelines (7). Those not meeting these criteria were classified as no-AKI controls.

Covariate Selection and Statistical Analysis

Predictor variables were selected from candidate variables to balance the following goals: (i) real-time availability; (ii) correlation to AKI; (iii) limited collinearity; (iv) either low missingness or ability to transform to a categorical variable. For ICU patients, predictor variables were age at admission, high-risk nephrotoxins, moderate-risk nephrotoxins, total medications, minimum platelet count, median red cell distribution width (RDW), phosphorus, serum transaminases, minimum pH, and hypotension. The same predictors were used for non-ICU patients, excluding pH and hypotension. Further details on the candidate and final variables are available in the Supplementary Materials.

The ICU and non-ICU cohorts were analyzed in parallel, generating two distinct AKI prediction models. Descriptive statistics were presented as median (interquartile range (IQR)) or frequency (%). Continuous variables were compared using Wilcoxon rank-sum test and categorical variables using Pearson chi-square test. We analyzed the association between AKI and the a priori selected covariates using multivariable logistic regression for ICU and non-ICU admissions. We internally validated and calibrated the models using bootstrapping (20). External validation of each model was performed by evaluating model performance in the temporally distinct validation cohorts. All analyses were performed with R, version 3.3.0 (R Development Core Team, Vienna, Austria). The level of statistical significance was set at P<0.05. Further details are available in the Supplementary Methods.

Results

Model Development and Internal Validation for ICU Patients

The ICU development cohort included 1,332 admissions (Figure 1a) of patients with care in the ICU during any portion of the hospitalization and creatinine measurements available for outcome determination. Median age was 4.4 (IQR 0.9–12.0) years, and the majority of patients were male (N=724, 54%) and White (N=1,025, 77%, Table 1). In this critically ill cohort, 791 patients (59%, Figure 1a) met the criteria for AKI, defined as a serum creatinine increase of ≥0.3 mg/dl or an increase of ≥1.5-fold from baseline (7). Of the 791 patients with AKI, 351 (44%) had stage 1 AKI, 299 (38%) had stage 2 AKI, and 141 patients (18%) had stage 3 AKI. The evaluated serum creatinine was obtained 0.3 (IQR 0.0–0.9) days after admission order for AKI cases and 0.9 (IQR 0.4–1.5) days for no-AKI controls (P<0.001). In the 72 h prior to the evaluated creatinine, those with AKI had lower minimum platelet counts, higher median RDW, lower minimum pH, and younger ages (Table 2). Those with AKI also had fewer total medication exposures and fewer high-risk nephrotoxic medication exposures, but more moderate-risk nephrotoxic medication exposures during that time.

Figure 1
figure 1

Identification of cases and controls for development and validation cohorts. (a) Patient encounters including intensive care (ICU patients) and without intensive care (non-ICU patients) identified for inclusion in the development cohort. (b) Patient encounters including intensive care (ICU patients) and without intensive care (non-ICU patients) identified for inclusion in the validation cohort. AKI, acute kidney injury; CKD, chronic kidney disease; ICU, intensive care unit; SCr, serum creatinine.

Full size image
Table 1 Patient demographic and predictive model variables 24 h prior to and 48 h after admission
Full size table
Table 2 Serum creatinine values and predictive model variables in development cohort in those with and without AKI
Full size table

The odds ratios and 95% confidence intervals for the selected predictors for AKI are shown in Figure 2a. Figure 3a shows the calibration plot from the development cohort, which demonstrates minimal overfitting and good calibration. Additional details including variable selection, the prediction rule as a linear predictor, and model performance and discrimination metrics from internal validation are given in the Supplementary Results, Supplementary Tables S3–S5, Supplementary Figure S2 and Supplementary File 1.

Figure 2
figure 2

Adjusted odds ratios for individual factors included in the acute kidney injury risk prediction for ICU (a) and non-ICU (b) patients. Odds ratios depicted are for age (15 vs. 3 years at admission), increase in one high-risk nephrotoxin, increase in one moderate-risk nephrotoxin, additional one total medication, minimum platelet count (278 × 103/μl vs. 206 × 103/μl), median RDW (13.5% vs. 13.2%), highest phosphorus (none checked vs. normal and one or more high value vs. all normal), transaminases (none checked vs. normal and one or more high value vs. all normal), minimum pH (7.3 vs. 7.2, ICU only), and presence vs. absence of hypotension (ICU only). Point estimates and 95% confidence intervals for each odds ratio are shown to the right of each plot. CI, confidence interval; ICU, intensive care unit; RDW, red cell distribution width.

Full size image
Figure 3
figure 3

Calibration curves for internal and external validation of acute kidney injury (AKI) models, with predicted probability based on the AKI model on the X axis and actual observed probability on the Y axis. (a) Internal calibration curve for intensive care unit (ICU) patients, generated via bootstrapping. (b) External calibration curve for ICU patients. (c) Internal calibration curve for non-ICU patients, generated via bootstrapping. (d) External calibration curve for non-ICU patients.

Full size image

Model External Validation for ICU Patients

The ICU external validation cohort included 866 patient admissions during the subsequent year, and 470 (54%) patients met the criteria for AKI (Figure 1b). This cohort was similar at admission to the development cohort with respect to demographics and AKI risk factors (Table 1). For external validation, the predictive rule derived from the development cohort was applied to the validation cohort to calculate predicted probabilities, which were then validated against true patient outcomes by fitting a logistic model. The calibration plot for the external validation data (Figure 3b) indicated a good model fit.

The frequency distributions of model-calculated AKI risk among ICU admissions with and without AKI in the validation cohort are shown in Figure 4a as an indication of model discrimination. Note the overlapping distributions, with those in the ICU who developed AKI with a tendency toward higher predicted probability scores. The receiver-operating characteristic (ROC) curve for the ICU model in the validation cohort is shown in Figure 4b; the c-statistic, which measures the area under the ROC curve, was 0.74 (95% confidence interval 0.71–0.77). The model performance varied by severity of AKI, with a c-statistic of 0.72 (0.68–0.76) for stage 1 AKI, and increased to 0.76 (0.72–0.8) for stage 2 or 3 AKI. Additional details of the external validation are included in the Supplementary Results, Supplementary Tables S5 and S6.

Figure 4
figure 4

Discrimination of acute kidney injury (AKI) models in the validation cohort. (a) Frequency distribution of calculated probabilities of AKI in the intensive care unit (ICU) validation cohort for those with (top panel) and without (bottom panel) AKI. (b) Receiver-operating characteristic (ROC) curve for the ICU validation cohort, with risk prediction model sensitivity on the Y axis and 1-specificity on the X axis. The area under the curve (AUC), or c-statistic, indicating model discrimination, is 0.74 (95% confidence interval 0.71–0.77). (c) Frequency distribution of calculated probabilities of AKI in the non-ICU validation cohort for those with (top panel) and without (bottom panel) AKI. (d) ROC curve for the non-ICU validation cohort, with risk prediction model sensitivity on the Y axis and 1-specificity on the X axis. The AUC (c-statistic) is 0.69 (95% confidence interval 0.66–0.72).

Full size image

Model Development and Internal Validation for Non-ICU Population

The non-ICU development cohort included 2,337 admissions with creatinine measurements available for outcome determination, of which 722 (31%) patients met AKI criteria (Figure 1). Of the 722 patients with AKI, 443 (61%) had stage 1 AKI, 199 (28%) had stage 2 AKI, and 80 patients (11%) had stage 3 AKI. Median age was 8.9 (IQR 2.6–14.8) years, and the majority of patients were male (N=1,264, 54%) and White (N=1,804, 77%, Table 1). The evaluated serum creatinine was obtained 0.0 (IQR −0.1 to 0.3) days after admission order for AKI cases and 0.6 (0.1–1.6) days for no-AKI controls (P<0.001). Similar to the ICU cohort, non-ICU patients with AKI were younger, had lower minimum platelet counts, and higher median RDW than those who did not develop AKI. Non-ICU patients with AKI had fewer total, high-risk, and moderate-risk nephrotoxic medications in the 72 h preceding their evaluated creatinine measurements as compared with no-AKI controls (Table 2).

The odds ratios and 95% confidence intervals for the selected AKI predictors are shown in Figure 2b. The internal calibration plot is shown in Figure 3c, which demonstrated minimal overfitting and good calibration. Additional results from the internal validation are presented in Supplementary Results, Supplementary Tables S3–S5, Supplementary Figure S2 and Supplementary File 1.

Model External Validation for Non-ICU Population

The non-ICU validation cohort included 1,474 admissions during the subsequent year, of which 469 (32%) patients met the criteria for AKI (Figure 1b). This cohort was similar at admission to the development cohort (Table 1). The calibration plot for the external validation data (Figure 3d) again indicated good model fit. The frequency distributions of model-calculated AKI risk among those with and without AKI in the non-ICU validation cohort are shown in Figure 4c. The ROC for the non-ICU model in the validation cohort is shown in Figure 4d; the c-statistic was 0.69 (95% confidence interval 0.66–0.72). Again, the model performance varied by disease severity, with a c-statistic for patients in the validation cohort for those with stage 1 AKI of 0.66 (0.63–0.7), and 0.74 (0.7–0.77) for those with more severe stage 2 or 3 AKI. Additional results from external validation are included in the Supplementary Results, Supplementary Tables S5 and S6.

Discussion

We present the development of accurate and reliable risk prediction models that can be used to screen for pediatric AKI in ICU and non-ICU patients based on variables readily available in the EHR. These models will be incorporated into the EHR as part of a randomized trial to test targeted AKI surveillance, a strategy that has been demonstrated to reduce AKI severity in other settings (13, 14). Given the frequency of AKI in children and the associated increases in length of stay, morbidity, and mortality, this work has the potential to save healthcare dollars and improve clinical outcomes for children. Furthermore, this approach demonstrates a strategy for utilizing EHR data to generate clinically implementable risk prediction rules that can be employed for alternative clinical outcomes.

The goal of this study was to build predictive models for pediatric AKI. Model covariates are only required to be informative in the model, and are not required to independently predict AKI or have a role in the pathogenesis of the outcome. Each covariate in the model was selected from an a priori set of candidate covariates, and, for many, the association and odds ratios were as expected. Lower platelet count, lower pH, higher RDW, and hypotension are known risk factors for AKI (1, 4, 6, 21, 22). One prior study found that younger patients experienced more AKI (6), whereas other studies have indicated higher risk in older patients; (1, 4) associations of young age to AKI are potentially confounded by the low baseline creatinine in the youngest pediatric patients, leading to AKI diagnosis via 1.5-fold creatinine increase without clinically meaningful reduction in renal function. Other results were unexpected, such as the stronger association of moderate-risk nephrotoxins than high-risk nephrotoxins to AKI, which may be due to clinicians avoiding the known nephrotoxic medications in children with suspected renal insufficiency. Fewer high-risk nephrotoxins and total medications may result from the very early presentation of many cases of AKI (e.g., on admission) prior to medication administration and the earlier measurement of creatinine in AKI cases vs. no-AKI controls; however, ICU patients with AKI had increased rate of exposure to moderate-risk nephrotoxins, indicating that we are capturing medication exposures. We also found that for the categorical variables (phosphorus and transaminases), unchecked or missing values were associated with an increased risk of AKI vs. normal/low values. This indicates that AKI was overrepresented in patients where these laboratory measurements were not performed as part of clinical care, relative to those with measured values within normal limits. This may indicate that patients with normal values are at lower risk for AKI due to more benign presentation of disease. Patients in whom these laboratory measurements are not routinely performed may be a group with relatively high incidence of AKI. However, these statistical correlations do not imply causation, and our study was not designed to identify independent risk factors or etiologies for pediatric AKI. Importantly, changes in clinical practice (e.g., a sharp increase in testing of transaminases or phosphorus) would require recalibration of the model, as calculated (but not true) AKI risk would be affected. A demonstration calculator incorporating model covariates is available (Supplementary File 2) and can be used to show the relative impact of changes in each predictor.

Although the AKI outcome is defined by change in creatinine, we intentionally designed these risk prediction models for AKI to perform well prior to the ascertainment of creatinine and associated results such as blood urea nitrogen, potassium, and calcium. Notably, a risk prediction model for early AKI in a pediatric ICU including blood urea nitrogen, pH, platelet count, total bilirubin, age, post-operative status, and pre-admission cardiac arrest had a c-statistic of 0.76–0.86 in validation sets, similar to the performance of our models (21). At our institution, pre-admission cardiac arrest and post-operative status are not available for incorporation into real-time prediction in our EHR, so these were not included. Furthermore, we developed a parallel predictive model for non-critically ill patients, an important advancement as these patients have less frequent laboratory monitoring and less rigorous assessment of urine output making early clinical detection of AKI more difficult.

Given the goal of early AKI detection, stage 1 AKI criteria was used, defined using the KDIGO creatinine criteria, the current clinical standard (23). The KDIGO criteria also include time windows for the increase in creatinine (i.e., increase by 0.3 mg/dl or more within 48 h or 1.5-fold increase in 7 days), which were not enforced in our cohort, as this restriction could introduce significant bias as only those with suspected renal injury would have frequent monitoring. In all, 187 of the 939 patients with AKI in the validation cohort had an increase of 0.3 mg/dl within 48 h, and 617 had 1.5-fold increase of creatinine within 7 days of the baseline, indicating that a majority of the cohort satisfied these criteria; however, the inclusion of patients who did not meet the time window requirements may impact the validity of our models in detecting AKI meeting formal KDIGO criteria.

Our development cohorts included data from the most recent admission for every patient hospitalized over a 2-year period in our tertiary referral children’s hospital pediatric ICU and non-ICU with baseline and an additional inpatient creatinine measurement. Since creatinine is measured as part of a panel obtained for a wide variety of reasons and not only because of suspicion of AKI, this cohort remains adequate to represent the patient population as a whole. However, patients with no creatinine measurements likely have systematically lower AKI risk, resulting in overestimation of AKI risk. Given our goal of using the EHR-based models as a screening tool to identify those at risk for follow-up testing via serum creatinine measurement, risk overestimation is preferred. However, the generalizability of our model cannot be proven without serial prospective assessment of serum creatinine in a representative cohort of pediatric patients.

The next step in translating this work is clinical implementation of the model to determine the impact of real-time AKI risk prediction. Previous prediction strategies have demonstrated benefits of early AKI detection. EHR-based screening for nephrotoxic medication exposures in pediatric inpatients at a tertiary pediatric hospital resulted in a 42% reduction in AKI intensity (13). In further work targeting cystic fibrosis patients exposed to aminoglycosides, daily serum creatinine measurement was associated with shorter aminoglycoside courses, fewer concomitant aminoglycosides, and earlier detection of AKI (14). In a randomized trial in adults, text paging providers and pharmacists, if their patient met AKI criteria, failed to demonstrate efficacy in improving the clinical outcomes of maximum change in serum creatinine, dialysis use, or mortality (24). These limited data indicate that early detection of AKI through targeted screening, not just clinician notification of AKI status, may be required to improve patient outcomes. Our risk prediction model can enable such early detection across all admitted pediatric patients, including those who are not exposed to nephrotoxic drugs or do not belong to a specific high-risk subset of patients. Given the early diagnosis of AKI in our cohort (often on admission), the risk prediction models may be most useful in identifying at-risk patients very early in their hospital course, providing the opportunity for early detection and intervention.

Our development of risk prediction models for incorporation into EHRs has several limitations. In building predictive models, covariates are not required to be independent or causative, so any associations with the outcome do not inform mechanistic insights into pediatric AKI. As discussed above, our defined outcome of AKI is limited to changes in creatinine measurement. This restricts our retrospective cohort to those patients with sufficient creatinine data available for assigning AKI status. Owing to the sparse availability of baseline creatinine measurement preceding the inpatient admission for pediatric patients, the KDIGO time windows for change in creatinine were not enforced, and the baseline measurement may have been measured after the AKI event (indicating return to normal renal function). This strategy was employed to reduce the potential bias from including only those individuals with serum creatinine measurement available prior to any injury. Furthermore, we did not differentiate between community-acquired and hospital-acquired AKI; indeed, many of our AKI cases were detected upon admission, and it is difficult to assess risk factors prior to admission. These issues may affect the validity of the models and the ability to accurately predict AKI that develops later in the hospital admission. Future work that includes more sensitive or specific biomarkers for renal injury, assayed across a broad spectrum of patients, may improve the precision and accuracy of AKI risk prediction modeling, as will sub-phenotyping the outcome (e.g., AKI on admission vs. early AKI vs. late AKI). We pre-specified that the patients admitted to the ICU at any time during their admission would be analyzed in the ICU cohort, but the AKI risk for patients subsequently transferred out of critical care units may be better predicted using the non-ICU model after transfer; additional data collection will enable testing of this hypothesis. Our external validation was performed using a distinct set of admissions (temporal validation), but did not assess model performance in another pediatric institution (geographic validation). Implementation in a different population should be preceded by validation and calibration in the target population. Finally, performance metrics indicate that while our model is stable across internal and external validation, model discrimination is modest. Cut points with high sensitivity for AKI have low specificity, especially for the non-ICU model (e.g., at 30% AKI risk, the non-ICU model is 74% sensitive but only 58% specific, Supplementary Table S6). To be an effective screening tool for AKI, high sensitivity is required, and low specificity is acceptable as a low-risk follow-up test is available. In this scenario, modest AKI risk prediction can trigger advice to check creatinine, the current diagnostic test for AKI.

Conclusions

Through analysis of admissions to this pediatric hospital, we generated and validated statistical predictive models for AKI, using the data routinely collected and available from EHRs. These risk prediction models can be incorporated into clinical practice from within EHRs as a screening tool to alert providers of increased risk for AKI among children in ICU and non-ICU cohorts, especially at the time of admission. Our approach demonstrates the use of EHR to promote a learning healthcare system by using EHR data in generating clinically implementable risk prediction.