Introduction

Febrile neutropenia (FN) is a common condition in children receiving chemotherapy and can be life-threatening1,2. An evidence-based guideline grounded in an understanding of which clinical characteristics and laboratory tests most accurately predict FN patients at high risk for severe illness could prompt more aggressive management and intensive monitoring. Conversely, it could help identify patients at low, or no, risk of serious clinical infection who may benefit from early discharge and less parenteral antibiotics during FN admission3,4,5.

Several prediction rules based on clinical and laboratory parameters have been developed6,7,8,9 for determining which FN patients are at risk for complications. However, these clinical decisions rules (CDRs) vary across populations and geographical locations, making it difficult to develop a single set of criteria to be used in clinical settings10,11. The available risk model studies have several limitations, including small study populations lacking independent validation, frequent missing values, and differences in the predictive factors considered. These CDRs have looked to predict severe sepsis, bacteremia, documented infection12, or need for critical care12,13,14 as outcomes.

To overcome the limitations of previous studies, efforts are under way to develop and validate risk models based on large studies in representative pediatric populations of patients receiving systemic chemotherapy. PICNICC (Predicting Infectious Complications in Children with Cancer) model was published as a mean of predicting complication in pediatric oncology patients15. This study was limited by its reliance on study-defined microbiologically documented infection (MDI) as the endpoint outcome, rather than a more comprehensive, patient-centered assessment of adverse outcomes such as transfer to intensive care. Additionally, MDI represent a large spectrum of severity of infections, which could range from mild skin infection to severe invasive infection. A recent study found that PICNICC risk stratification schema performed poorly in adolescents/young adults with cancer16. There are few studies of prediction models in pediatric cancer patients with FN in the USA. Hakim et al.17 from St. Jude hospital in Memphis, Tennessee, developed a model for predicting severe infections and adverse outcomes in FN based on a large sample size of pediatric cancer patients at St. Jude Children's Research Hospital. Authors in this study found initial clinical impression using subjective variable such as “sick appearance” as independent risk factor in their prediction model.

The aims of this study are to examine performance of Hakim et al. model in our FN cohort. We try also to define relevant variables at the time of FN presentation and develop and validate a risk prediction model for bloodstream infection (BSI) and transfer to intensive care unit (TIC) in pediatric and adolescent cancer patients. Both prediction models were built for both outcomes using logistic regression and random forest and compared with Hakim model.

Methods

Setting and study population

A retrospective cohort study was conducted at University of Chicago Medicine (UCM) Comer Children’s Hospital, a 172-bed acute care hospital located on Chicago’s south side that serves a diverse pediatric population. The medical center offers highly specialized cancer care, including stem cell transplant (SCT)18.

Study protocols were approved by the Clinical Trials Review Committee and the University of Chicago Institutional Review Board. We confirm all methods were carried out in accordance with relevant guidelines and regulations. The need for informed consent was formally waived by the approving committee. To identify appropriate patients for inclusion, the Clinical Research Data Warehouse, operated by the Center for Research Informatics, was queried for hospital admissions of patients 24 years of age or younger admitted to Comer Children’s Hospital from March 2009 to December 2016 for discrete clinical and laboratory values as well as diagnosis codes using International Classification of Disease, 9th Revision, Clinical Modification (ICD-9-CM) codes (ICD-10 after October 2015) to identify patients with FNE. Oncology patients were identified with ICD codes for malignancy or SCT diagnoses. Neutropenic patients were identified by ICD code for neutropenia OR absolute neutrophil count (ANC) < 500. Febrile patients were identified by temperature ≥ 38.0 °C (≥ 100.4℉) in a 24-h period (Fig. 1). A list of predictor variables is provided in Supplementary Table 2.

Figure 1
figure 1

Study flow diagram. Febrile neutropenia (FN) with and without BSI and need to PICU. FNE Febrile neutropenia episodes, BSI blood stream infection. PICU pediatric intensive care, BMT bone marrow transplant.

Full size image

Manual electronic health record (EHR) review was conducted to verify that FN episodes were appropriate for inclusion based on the above characteristics. All episodes not meeting the above-mentioned criteria, including febrile non-neutropenic episodes, were excluded. SCT recipients who develop FN after transplant (Day #0) were excluded. For patients with more than one admission for FN, each admission was counted as a separate episode. Data for all variables listed below were collected during manual EHR review.

Data collection

Study data were collected by EHR review and managed using RED Cap19. The risk prediction variables are a mixture of clinical findings, and basic laboratory studies, which are readily available at admission or at early reassessment, and which are reproducible across all other settings. Variables collected include: patient demographics (age, gender), oncologic diagnosis, history of BSI; clinical features (blood pressure, documented complaints [gastrointestinal symptoms, mucositis, upper respiratory symptoms and chills], and location at presentation); and laboratory data at presentation (white blood cell [WBC] count, absolute neutrophil count [ANC], absolute monocyte count [AMC], absolute lymphocyte count [ALC], platelet count, hemoglobin, duration of neutropenia before FN episode, and blood culture results.). For temperature, we report the maximum temperature between presentation and antibiotic initiation. To decrease observer bias, risk factors and outcomes of interest were recorded by two investigators in order to blind associated variables in each FNE. We did not evaluate other potential adverse events besides BSI or transfer to ICU (e.g., culture negative sepsis) so we could focus on discrete FN-related outcomes.

Outcome and predictor variables

The primary outcome of BSI was defined by a positive blood culture (within 7 days from the date of FN onset) with a pathogenic organism or with a contaminant (determined by the National Health Safety Network (NHSN) criteria for skin commensals) that the clinical team decided to treat as a pathogen2. The secondary outcome of TIC was defined by transfer to ICU within 14 days from the date of FN onset which is defined as the first timepoint with both documented fever and documented neutropenia regardless of patient location. A list of predictor variables is provided in Supplementary Table 1. Fever was defined as a single oral temperature of > 38.3ºC or temperature of > 38.0ºC sustained over a 1-h period or on more than one occasion in a 24-h period. If there were multiple temperature readings, we reported the maximum temperature between first symptoms and antibiotic initiation2. Neutropenia was defined as ANC < 500/mm3. Hypotension was defined as a systolic blood pressure < 5th percentile for age and sex20. Each FN episode was independently associated with outcomes using a variable indicating the number of prior FN episodes (categorized as 0, 1 and > 1) in order to control for recurrent episodes. We characterized underlying oncologic conditions in patients as belonging to one of four groups based on intensity of chemotherapy: (1) mixed leukemia and acute myeloid leukemia (AML); (2) acute lymphoblastic leukemia (ALL) and lymphoma (Hodgkin’s and non-Hodgkin’s); (3) neuroblastoma (NB); and (4) all other solid tumors.

Clinical patient management

Pediatric FN patients were managed per standard institutional practice, which did not undergo any major changes during the study period. Ceftazidime is an initial empiric antimicrobial for FN patients with vancomycin ± gentamicin added based on clinical presentation (i.e., concern for central venous catheter infection or septic shock). Cefepime was administered instead of ceftazidime for selected patients with high-risk FN such as acute myelogenous leukemia (AML). Empiric antifungal therapy (usually liposomal amphotericin B) was added if the patient remained febrile on day 5 of antibiotics and if neutropenia was expected to last longer than 5 to 7 days. Antibacterial prophylaxis was not routinely used; the small number of patients who did receive prophylaxis were excluded.

Prediction model

We utilized two machine learning techniques, logistic regression and random forest (RF), to predict BSI and TIC based on variables collected at time of FN presentation. While logistic regression models the association between the outcome and predictors in linear terms, RFs explore complex non-linear relationships between variables to further improve prediction accuracy. We split our dataset longitudinally into model derivation (years 2009–2014, n = 343 [68%]) and independent prospective validation (years 2015–2016, n = 162[32%]) cohorts, this step allows us to determine model performance if implemented in our hospital at the start of the validation period, providing a better real-world assessment of performance in comparison to a random derivation-validation data split. Hyper-parameter optimization for random forests was performed on the derivation cohort using fivefold cross validation. Final predictions were performed on the prospective validation dataset and the area under the receiver operating characteristic curve (AUC) was compared between the logistic regression model (LReg) the RF model, and Hakim et al.17 model for both BSI and TIC. Variable importance plots were created to determine the variables most crucial to the prediction of each outcome using the RF model. Analyses were performed using R version 3.3 (R Project for Statistical Computing), with two-sided P < 0.05 denoting statistical significance. We had no variables with > 10% missing values. For variables with < 10% missing values, we used imputation. Mixed-effects logistic regression models were fit for the PICU, and BSI outcomes with the inclusion of a random subject effect to account for the multiple observations per patient.

Results

Study population

A total of 505 FN episodes (FNEs) were identified in 230 patients. FN episodes during a unique admission ranged from 1 to 3 episodes with complete ANC recovery between these episodes.

The median age was 11.5 (SD = 5.5) years and 47% were female (Table 1). Among 505 FNEs, 106 (21%) developed BSI (Table 1). The most frequent underlying diagnosis was ALL/lymphoma. Rates of BSI were highest among patients with AML (Table 1). The majority of FN cases 420 (83%) were either admitted directly from the emergency department (ED) or from clinic.

Table 1 Characteristics of FN episodes among pediatric cancer (N = 505).
Full size table

In bivariable analysis, FNEs that resulted in BSI, compared with FNEs that did not result in BSI, were more likely to have underlying diagnosis of AML; history of a prior BSI; thrombocytopenia < 50 × 109/L; ALC < 300/mm3; hypotension; higher fever; chills, and inpatient status at the time of FNE (Table 1). A total of 115 organisms were recovered during 106 episodes of BSI in 77 patients. Gram-positive bacteremia was detected in 65/106 (56.5%) and Gram-negative bacteremia was detected in 46/106 (43.3%) episodes. Polymicrobial bacteremia was detected in 7/106 (6.6%).

TIC rate was 54/505 (10.6%), though the majority of TIC occurred in the first week of FN onset 46/54 (85%), but still 8/54 (15%) had clinical worsening after day 7–14 require admission in PICU.

Frequency of variables and outcomes in the derivation and validation groups are shown in Table 2 (Fig. 2, 3).

Table 2 Characteristics, variables and outcomes in FNEs among derivation and validation cohorts (N = 505).
Full size table
Figure 2
figure 2

Depicts a list of variables important to the prediction of bactremia using the Random Forest model. Temperature height and hypotension recorded at time of presentation of FN episode, prior positive blood culture, and AML diagnosis were variables that contributed the most to BSI prediction.

Full size image
Figure 3
figure 3

Depicts a list of variables important to the prediction of PICU admission with resampling of the multivariate analysis. The maximum temperature and hypotension recorded at time of presentation of FN episode were the most important variables for predicting transfer to PICU.

Full size image

Model performance

AUCs and statical performance for different prediction models using the prospective validation cohort are shown in Table 3. The logistic regression model (LReg) performed similarly to the Hakim model in predicting BSI (LReg AUC 0.65 vs. Hakim AUC 0.66), whereas the RF model predicted BSI much more accurately than the Hakim model (RF AUC 0.79 vs. Hakim AUC 0.66, P < 0.05) (Table 3) (Fig. 4). The RF model also performed best at predicting 14-day TIC as compared to the (LReg) model and the Hakim Model (RF AUC 0.88 vs LReg AUC 0.76, P < 0.05) and (RF AUC 0.88 vs Hakim AUC 0.65, P < 0.05).

Table 3 Statical Performance of 3 prediction models for BSI and TIC.
Full size table
Figure 4
figure 4

Receiver operating characteristic curve for random forest (RF) model in BSI prediction (AUC 0.79).

Full size image

For each outcome, a likelihood ratio (LR) was calculated with 95% credibility (post-test probability) to assess a probability of BSI or TIC and potential utility of prediction model in clinical practice The quantitative value of a calculated LR corresponds to the utility of a prediction tool to point towards or away from an outcome. LR+ is represented by a value 1 or greater, and the higher the value, the increasing ability to identify high risk patients. LR− is a value between 0 and 1, and the closer it is to 0, the better it can identify a low-risk patient. The RF model has a higher LR + compared with Hakim et al. in predicting high risk for BSI (3.5 vs 1.7, P < 0.05) and TIC (6.8 vs 2.6, P < 0.05), and a lower LR- in predicting low risk for both BSI and TIC (0.24 vs 0.6), and (0.12 vs 0.4) respectively. Based on these results, our model over performed Hakim et al. model in predicting FN outcomes in our cohort.

Because the RF utilizes a decision-tree type approach, the location of each variable across all trees is an approximate measure of the importance of that variable towards predicting the outcome. Figure 2 depicts the 10 variables most important to predicting BSI. Of these, temperature, low blood pressure, prior positive blood culture, and AML as underlying diagnosis are the most important variables used by the RF to predict positive blood culture. Temperature and low blood pressure are also highly important in the RF model for predicting TIC (Fig. 3).

Discussion

In this study, we derived and validated a prediction model for BSI and transfer to ICU in a large, diverse population of children with cancer that demonstrated better performance than previously published methods. Independent risk factors for BSI and TIC included high temperature and low blood pressure on admission. The use of such risk factors to identify the patients who are at greatest risk for BSI and risk for TIC could help providers identify the appropriate level of care.

There is currently no single risk stratification system in widespread use in pediatric practice and there are considerable variations in practice. A simple, reliable, and safe risk stratification system will have potential to significantly reduce hospitalization rates and hospital length of stay without increasing overall mortality.

Several previous investigations of pediatric cancer patients with FN have identified different hematologic laboratory results6,7,9,17,21 (e.g., ANC, platelet count, or absolute monocyte count), clinical factors related to a patient’s underlying cancer (e.g., diagnosis of AML or the presence of uncontrolled relapsed disease) and vital sign abnormalities (e.g., fever, hypotension)8,21,22,23 as risk factors for BSI, MDI and adverse outcomes.

Most previous studies have treated temperature as a dichotomous variable (temperature > 39 or < 39 degrees)8,12,17,22. In our study, including maximal temperature as a continuous variable increased model performance in predicting BSI and TIC. We also identified hypotension as an independent risk for both BSI and TIC, like other studies12,23.

We identified prior BSI, regardless underlying diagnosis, as an additional variable important to predicting BSI. Subsequent BSI don’t represent relapse or inadequate treatment of a previous BSI, since subsequent BSI were caused by different pathogens than the index BSI. In addition, prior BSI remained a significant risk factor regardless of whether the central line was retained or removed. Interestingly, model performance doesn’t change even after exclude BSI caused by the common skin contaminant Coagulase-negative staphylococci (CoNS).

Similar to previous studies, we found that AML patients with FN are more likely to develop BSI compared with patients with other underlying diagnoses. This is not surprising since patients with AML receive more intensive chemotherapeutic regimens (i.e., Cytarbine containing regimen) than do other oncologic patients, leading to longer durations of neutropenia and thus a higher risk of infectious complications24,25. The rate of bacteremia within our cohort, 21%, is on the high end of the range reported in the pediatric FN literature26,27. We hypothesize that our limited use of antibacterial prophylaxis in high-risk FN patients could be contributing to this.

Strengths of our study include the relatively large patient population and number of events of the outcomes of interest (BSI and ICU transfer). Data from our data warehouse was supplemented by extensive manual review of the electronic medical record. We performed simultaneous assessment of potential risk factors allowing for the identification of independent factors predictive of BSI or ICU transfer, each of which we evaluated separately. Also, it is important to note that there are no current validated schemas for defining those patients at high risk of developing complications from FN. Our prediction model n not only identifies low risk, but also high-risk FN patients. Furthermore, most published prediction studies in FN include a have limited number of variables, while our study has analyzed a broader variety of elements including patient-specific factors (including age, malignancy type), treatment-specific factors (GCSF), episode-specific factors (including height of fever, hypotension, mucositis, blood counts).

The prediction and post-test probability of the model provides a robust method of determining pediatric cancer patients with high risk of BSI and transfer to ICU. The model performed better than other published models.

The current study has several limitations. First, it is a retrospective analysis at a single academic medical center and the results may not be generalizable to other institutions with different practices of antimicrobial prophylaxis and different empiric management of neutropenic fevers. While we did stratify patients into four groups based on cancer type, we did not look at the impact of chemotherapy phase and intensity (i.e. ALL) on outcomes. The retrospective nature of our study limited our ability to collect accurate objective clinical variables from the medical record such as mucositis, chills, and upper respiratory tract symptoms. Because overall mortality among our patients was low (2.3%), we lacked the power to perform subgroup analyses related to mortality as an outcome.

Conclusion

In this study, we derived and validated prediction rules for BSI and transfer to ICU in pediatric cancer patients who have FN. Children and adolescents with higher fever and hypotension at presentation are at increased risk of BSI and transfer to ICU. Having a prior BSI is an additional risk factor for developing a subsequent BSI. To the best of our knowledge, our study is one of the few done in the United States in last decade assessing the risk factors predictive of an adverse outcome in pediatric patients with FN. The information gained from this study will help in formulating a risk prediction model that is easy to use, widely applicable, and clinically relevant. Prospective, external validation of this model is essential prior to implementation to risk stratify pediatric FN patients. After external validation, our next step to is to use this tool to facilitate antibiotic stewardship and early hospital discharge among pediatric cancer patients with FN.