Main

Fever is among the most common presenting signs of illness in children (1). Identifying those febrile children with a serious infectious disease among the large majority who will have a benign, self-limiting infectious disease continues to be challenging. Many clinical signs and symptoms and biomarkers have been proposed as diagnostic predictors of serious bacterial infections (SBIs), but not one has yet proven useful for ruling in or ruling out the presence of SBI as a solitary predictor (2, 3, 4, 5). Prediction models that combine predictors of SBI appear more useful (6). A systematic review included the Feverkidstool, a clinical prediction model including both clinical signs and symptoms and the biomarker C-reactive protein (CRP), as a potential tool for supporting physicians in their decision-making for safely discharging febrile children from an emergency department (ED) (7). This Feverkidstool assists in distinguishing children with pneumonia and children with other SBIs from children without SBI (8). However, external validity of such tools needs to be confirmed to ensure their usefulness elsewhere, a step that is often lacking in pediatric research (9). In addition, procalcitonin (PCT), another biomarker that has been linked with the presence of SBI, had not been considered in the original study (3, 4, 10, 11, 12). Hence, questions on the diagnostic value of PCT in addition to clinical signs and symptoms as well as the comparability of PCT and CRP remain unanswered. In this study, we aimed to validate the Feverkidstool, and to determine the incremental diagnostic value of PCT in a prospective observational cohort of children with fever at risk of having an SBI at the ED.

Methods

Design

First, we performed a prospective observational study to validate the Feverkidstool, a previously presented prediction model consisting of clinical signs and symptoms and CRP, (8) in a cohort of febrile children who were included from two Dutch EDs. Next, we determined the incremental value of PCT to this prediction model. In addition, we looked at the diagnostic properties of a model with clinical signs and symptoms and PCT, and compared these with the diagnostic properties of the Feverkidstool.

Setting and Participants

This study was conducted at two EDs in Rotterdam, the Netherlands, and it included children aged 1 month to 16 years. Children were included from the Erasmus MC—Sophia Children’s Hospital between February 2009 and May 2012, and from the Maasstad Hospital between May 2011 and May 2012 (ref. 13). The Erasmus MC is a university hospital and its pediatric ED is visited by ~9,000 children annually, of whom 90% require basic pediatric care (14). The Maasstad Hospital is a teaching hospital with a mixed adult and pediatric ED that has some 10,000 pediatric medical emergency consultations yearly. At both study sites a study protocol was introduced that involved routinely recruiting patients at the moment of triage, measuring PCT and CRP during their stay at the ED, recording standardized clinical signs and symptoms in all eligible patients, and standardized follow-up after discharge from the ED. Fever was defined as a body temperature ≥38.5 °C at triage, fever recorded at home within 24 h before consultation, fever as reason for referral by the general practitioner, or fever as positive discriminator in the Manchester Triage System. Children with an underlying chronic condition requiring specialist pediatric follow-up that is expected to last for at least 1 year were excluded (15). Well-appearing febrile children with a clear focus of an upper airway infection not undergoing additional investigations were also excluded as they were at such a low overall risk of SBI that diagnostic tools would not contribute to clinical management; this concerned with only well appearing children where the nurse felt there was a clear focus of an upper respiratory tract infection, and no other warning signs present, while being in the ED; children with any persisting diagnostic uncertainty were eligible for the study. Additional diagnostic tests were performed at the discretion of the attending physician. Children who revisited the ED within 5 days after their initial visit were only eligible with their initial visit, although data from all visits were considered in establishing final diagnosis.

Ethics Approval

The study protocol was approved by the institutional ethical committees of both the Erasmus MC (MEC-2007-066) and the Maasstad Hospital (2010/64). Informed consent was required and written informed consent was obtained from all children who had CRP and/or PCT tests performed.

Data Collection

In the Erasmus MC, trained nurses recorded clinical signs and symptoms of all children with fever in a standardized electronic patient form that was tailored specifically to this study’s purposes, and which was incorporated in the patient’s digital hospital record. Triage data and patient characteristics were also recorded as a part of the ongoing data collection (16). Simultaneously, physicians reported on predictor variables by completing a predefined electronic proforma. In the Maasstad Hospital, patient characteristics, data on vital signs, triage data, and data on diagnostics and follow-up were collected prospectively in a standardized digital patient file by the triaging nurse. Additional clinical data were recorded by the attending physician on a paper research form. Nurses and physicians were trained in data collection and informed of the study, but were blinded for the specific outcome measures. Regular feedback on data quality and inclusion rates was provided by the researchers for both settings.

CRP and PCT

CRP was measured either by a bedside test (84%) or by a traditional laboratory test (16%). CRP values were available to the physician for further clinical evaluation. In the Erasmus MC laboratory CRP was determined in heparin plasma with an immunoturbidimetric CRP assay using a Modular system (Roche, Mannheim, Germany). In the Maasstad laboratory CRP was determined in heparin plasma with a nephelometric CRP assay using a Dimension VISTA 1500 system (Siemens, Erlangen, Germany). The Afinion AS100 (Axis-Shield PoC AS, Oslo, Norway, distributed by Clindia Benelux BV) was used for CRP bedside testing, and was used at both EDs. Before the study was carried out, validity and comparability of these tests were assessed and approved (13). PCT samples from both study sites were analyzed in the Erasmus MC using the Brahms Kryptor Assay (13). PCT values were available for research purposes only and were not available to the physician for further clinical evaluation.

Predictor Variables

The original Feverkidstool included clinical variables and CRP (Table 1); variables were defined as previously described (8). That is, age was modeled piece-wise linear, which accounted for continuously differentiating risk profiles in children <1 years old and of children aged 1 year or older. Duration of fever was truncated to the 97th percentile at a maximum of 6 days. Body temperature was measured in degrees Celsius and was measured rectally in children of up to 5 years, and tympanic in older children at the Erasmus MC—Sophia, and either rectally or tympanic at the Maasstad at the discretion of the nurse. Heart rate (beats per minute) and respiratory rate (breaths per minute) were recorded according to standards of the Advanced Pediatric Life Support group (APLS), and categorized using APLS age-specific thresholds (17). Oxygen saturation was dichotomized at <94% O2, and, when not recorded, the oxygen saturation was assumed to be normal at ≥94% O2. Ill appearance was defined on a 2 point scale according to the nurse’s impression at triage. If missing, ill appearance was completed by the physician’s impression (47%) and ultimately (in 5% of the eligible children) by using the Young infant (children aged ≤3 months) or Yale Observational scale (children aged >3 months) score of ≥8 (refs 18, 19). A prolonged peripheral capillary refill was defined as ≥3 s. Chest wall retractions were defined as present if subcostal retractions, or intercostal retractions, or tracheal tug were observed. We truncated CRP at a value of 225 mg/l and PCT at 6.0 ng/ml, both at the 97th centiles, and we used the natural logarithms of CRP and PCT (Ln(CRP) and Ln(PCT)).

Table 1 General characteristics of children with C-reactive protein and procalcitonin available
Full size table

Outcome Measures

We defined SBI according to a predefined reference standard based on positive cultures from otherwise sterile locations, focal abnormal radiologic findings, and expert clinical consensus by the investigators (R.O., E.K., and R.N.; Supplementary Table S1 online; (8, 20)). We subsequently categorized SBI into children with pneumonia and children with other SBI vs. children with no SBI to account for the heterogeneity of presenting signs and symptoms similar to the definitions used previously (8). Other SBIs consisted of urinary tract infections, septicemia, meningitis, orbital cellulitis, erysipelas, bacterial gastroenteritis, bacterial arthritis, bacterial upper airway infection, and bacterial osteomyelitis. Final diagnoses were coded by trained medical students, who were blinded for the study’s specific aims and PCT values, but not for clinical data or CRP, and all diagnoses were reviewed for consistency by the investigators. Consensus diagnoses were mandatory for all children not completely fulfilling the reference standard, children with an atypical clinical course, and for children with a prolonged hospital admission or a prolonged course of antibiotics. These included children who received antibiotics before cultures were taken, most importantly in children with abnormal urinalyses and negative cultures, children with pleiocytosis in cerebral spinal fluid and negative cultures (i.e., aseptic meningitis), and children with radiographic evidence of musculoskeletal infection and negative cultures. Follow-up of all eligible children included a standardized telephone call after 72 h or follow-up visit at the ED of discharged patients, and checking for revisits in the medical system to ensure that no cases of SBI were missed, thus minimizing verification bias (21).

Missing Values

Children with values for both CRP and PCT were available for this study. Missing variables for clinical signs and symptoms were imputed 10 times using the MICE algorithm, R statistical software ( Supplementary Table S2 ; (ref. 22). The imputation model included the Feverkidstool variables, PCT, the outcome measure SBI, and descriptors of case-mix, such as arrival time, season of presentation, triage urgency, antibiotics prescription, hospitalization, and a cohort variable (23). Multiple imputation techniques enabled analyzing all available data, without adding information to the original available data, and thus not adding information on the incremental value of PCT (24). Analyses were performed separately in the 10 imputed data sets and were combined using Rubin’s rules (25).

Statistical Analysis

First, we validated the original prediction model by determining discriminative ability and calibration, thus validating the diagnostic performance of the Feverkidstool in this new population. Next, we refitted the Feverkidstool variables and added PCT to the model. In addition, we refitted a model containing clinical signs and symptoms and PCT, but without CRP. For all analyses we used polytomous regression analyses differentiating pneumonia vs. no SBI and other SBIs vs. no SBI. Polytomous regression analyses allow for distinguishing more than two outcome categories while considering the total population of febrile children and their potential final diagnoses. Discriminative ability was expressed by pairwise c-statistics and the polytomous discrimination index (PDI). Pairwise c-statistics were estimated as one outcome category (i.e.. pneumonia or other SBIs) vs. one other outcome category (here: no SBI), equaling receiver operating characteristic curves. The PDI assesses discriminative ability considering all possible outcome categories with a value of 0.33 (1/number of outcome categories, here 1/3), indicating a non-informative test and a value of one indicating a perfect discriminating test. The PDI is the average of category-specific polytomous discrimination indices each of which separately reflects the ability to predict a case better than non-cases for all outcome categories of interest (26). Bootstrapping (100 iterations) was used to determine 95% confidence intervals (CIs) of the PDI. Calibration reflects the predicted risks vs. the true observed risks for children with either pneumonia or other SBIs vs. no SBI. Incremental diagnostic value of PCT was determined by its predictive strength in the model with clinical signs and symptoms and CRP (27), and by differences in the discriminative ability between the models assessed as the differences in pairwise C-statistics and PDI (26, 28, 29). We estimated sensitivities, specificities, and positive and negative likelihood ratios for several risk thresholds of the refitted prediction models (adjusted for model optimism) by comparing one outcome category vs. the rest. All analyses were performed using R statistical software (22).

Results

Population Characteristics

In all, 2,703 febrile children were included in this study, of whom 1,085 had both CRP and PCT performed and were available for analysis (Table 1 and Figure 1); their median age was 1.6 years (interquartile range: 0.8–3.4), and 73 children (7%) had pneumonia, and 98 children (9%) had other SBIs (Table 2). Overall, children with CRP and PCT measures had a slightly higher incidence of SBI, were hospitalized more often, and were prescribed antibiotic therapy more frequently compared with the total included population, ( Supplementary Table S3 ). Seventy-four percent of the children had their CRP measured: characteristics of children with both CRP and PCT were similar to the children with only CRP.

Figure 1
figure 1

Flowchart of included population.

Full size image
Table 2 Feverkidstool: predictor distribution for the three outcome categories
Full size table

Validation of the Original Prediction Model

The Feverkidstool showed good external validity: using the original coefficients of the Feverkidstool ( Supplementary Appendix S4 ), pairwise c-statistics were 0.84 (95% CI: 0.74–0.94) for the prediction of pneumonia and 0.82 (95% CI: 0.73–0.91) for other SBI; PDI was 0.66 (95% CI 0.60–0.70). The predicted risks for having pneumonia were clearly higher for children with pneumonia than for those without pneumonia; similar findings were observed for the risk of having other SBIs vs. children without other SBIs (Figure 2). Calibration was reasonable with an underestimation of the predicted risks in the highest risk quartile, mostly because of sparsity of cases in higher-risk categories (Figure 3).

Figure 2
figure 2

External validation of the original prediction model with clinical signs and symptoms and C-reactive protein. Predicted risks of pneumonia (a) and of other SBI (b) for children diagnosed with pneumonia, other SBI, or no SBI. SBI, serious bacterial infection.

Full size image
Figure 3
figure 3

External validation of the original prediction model with clinical signs and symptoms and C-reactive protein. Calibration plots for the risk of having pneumonia (a) and risk of having other SBIs (b). Calibration plot of the predicted risk (x axis) and the observed frequency of pneumonia (a) and other SBIs (b) (with 95% CI, y axis). The triangles represent the mean (predicted vs. observed) risk estimates by quintiles of predicted risks. The dashed diagonal line represents ideal calibration. The distribution of the predicted risks of patients with pneumonia (n=73 (a)), other SBIs (n=98, (b)), and other patients (n=1,012 (a) and n=987 (b)) is shown in the bottom of the graph, parallel to the x axis. CI, confidence interval; SBI, serious bacterial infection.

Full size image

Incremental Value

The predictive strength of PCT when added to a refitted model was reflected by regression coefficients of Ln(PCT) of 0.34 (SE 0.13) for pneumonia and 0.45 (SE 0.11) for other SBIs; ( Supplementary Table S5 ). The likelihood ratio test of adding PCT to the model was significant with a P value <0.001 (difference in deviance between the two polytomous models was 23.4, with 2 degrees of freedom). The addition of PCT did not alter regression coefficients of the other variables substantially, but lessened the predictive effect of CRP ( Supplementary Table S5 ). The model that included both CRP and PCT had similar discriminative ability as the refitted Feverkidstool: for the prediction of pneumonia the pairwise c-statistic increased from 0.85 (95% CI: 0.76–0.94) to 0.86 (95% CI: 0.77–0.94) and for other SBI from 0.81 (95% CI: 0.73–0.90) to 0.83 (95% CI: 0.75–0.91). The PDI increased from 0.65 (95% CI: 0.59–0.70) to 0.66 (95% CI: 0.61–0.71; Table 3). A refitted model with clinical signs and symptoms and PCT, but without CRP, achieved similar model performance (Table 3). Distributions of predicted risks were similar for the three refitted models (Figure 4). All models had similar strength for ruling in and ruling out SBI (Table 4): risk thresholds ≥10% were, based on high specificities and positive likelihood ratios, useful for ruling in SBI, and risk thresholds ≤2.5% were, based on high sensitivities and negative likelihood ratios, useful for ruling out SBIs. A single value of CRP or PCT was not as strong for predicting SBI as the prediction model and lacked predictive value for all types of SBI ( Supplementary Table S6 ; (ref. 13)). Overall, models including a biomarker performed notably better than a model including just clinical signs and symptoms, in particular for other SBIs (Table 3).

Table 3 Discriminative ability of the refitted prediction models
Full size table
Figure 4
figure 4

Distribution of predicted risks for pneumonia and other SBIs. Predicted risks of pneumonia (a) and other SBIs (b) for children diagnosed with pneumonia, other SBIs, or no SBI for the three refitted models. SBI, serious bacterial infection.

Full size image
Table 4 Diagnostic performances of prediction models including clinical signs and symptoms and either C-reactive protein or procalcitonin and a model including clinical signs and symptoms and both C-reactive protein and procalcitonin
Full size table

Discussion

Principal Findings

This is the first study to confirm the external validity of the Feverkidstool, a clinical prediction model with clinical signs and symptoms and CRP that calculates the risk of having SBI in febrile children at the ED. Among those children with both CRP and PCT available, the incremental value of PCT was significant; however, it showed only marginal improvement of the discriminative ability of the Feverkidstool. Similarly, when looking at the diagnostic performance at several risk thresholds, a model including both CRP and PCT did not outperfom such models containing either CRP or PCT. Models with clinical signs and symptoms and one of the two biomarkers performed similar, but better than a model with clinical signs and symptoms alone.

Comparison with Existing Literature

In contrast to our findings, a systematic review claimed that PCT as a single marker was a better biomarker for detecting SBI in febrile children compared with CRP, and was especially useful for ruling out the presence of SBI (4). Other recent papers also concluded that PCT was superior to CRP in identifying SBI in febrile children, in particular for detecting more invasive bacterial infections in young infants (10, 11, 30). However, these studies were mostly performed in highly selected patient populations, such as well-appearing young infants with fever, ill-appearing children, or hospitalized children, as opposed to our broader population of febrile children at the ED with diagnostic uncertainty (3, 10, 11, 30). In addition, many of the studies that evaluated the diagnostic value of PCT looked at the predictive ability of PCT as a single predictor not considering the clinical context or value of CRP (3, 4). One prediction model that combined CRP and PCT together with urinalysis, the Lab-score, showed promising diagnostic value for detecting SBI in febrile children (31, 32). However, clinical signs and symptoms were not incorporated and the additional value of one biomarker in relation to the other was unclear. Moreover, two validation studies failed to replicate the diagnostic performance for ruling out SBI (13, 33). Furthermore, although in most previous research SBIs are questionably traditionally grouped in a single homogenous outcome category, we assessed the diagnostic value of PCT in a polytomous manner estimating the value of PCT to predict pneumonia and other SBIs simultaneously. Next, some of the clinical signs and symptoms considered in the prediction model, most importantly age and duration of fever, have been associated with differentiated predictive ability of CRP and PCT. For example, concentration levels of PCT are elevated at an earlier stage of disease than levels of CRP (11). Likewise, PCT appears to be a better predictor of invasive bacterial infections in young infants, in particular in those with a short onset of fever (10). Although our model combined age and duration of fever together with CRP and PCT, thus accounting for potential confounding, our data were not specifically tailored to look at the group of young febrile infants with a short duration of fever. In our cohort only 70 children were aged <3 months with 15 children having SBI (21%); of these children 49 (70%) had fever for ≤24 h, with 11 children having an SBI. In a previous study we showed that duration of fever had no significant effect on the interpretation of diagnostic values of CRP and PCT in a general population of febrile children up to 16 years of age in pediatric emergency care (13).

Clinical Implications

Altogether, our data suggest that routinely determining both CRP and PCT in the evaluation of febrile children at the ED does not appear to be warranted. The near identical diagnostic performances at the different risk thresholds across the models do not seem to justify potential differences in clinical management for most children at present. However, PCT was still a significant predictor of SBI and an additional PCT value might yet be useful in individual patients with persisting diagnostic uncertainty. This would depend on individualized clinical risk thresholds that, for example, would imply starting antibiotic therapy or performing additional diagnostic tests. Yet, these thresholds currently remain elusive for the topic of febrile children and should be the focus of future research (23, 34). Our study adds to the conclusions of a systematic review that the Feverkidstool can be used as a validated tool for supporting the decision of safely discharging febrile children from the ED (7). As always, debate will continue on how reliable a diagnostic test needs to be to account for the potentially severe consequences of missing a diagnosis of SBI, but our findings imply a guiding diagnostic role for the Feverkidstool at least. This study also provides more evidence that CRP and PCT can be used interchangeably in an emergency care setting, depending on local preferences for one or the other.

Strengths and Limitations

This study is the first to address the diagnostic value of PCT in combination with clinical signs and symptoms and CRP: our suggested approach of using a validated prediction model and evaluating the incremental value of a new diagnostic test in its clinical context makes a strong case for the validity of our findings. A strength is the large, prospective cohort of febrile children of whom we collected standardized clinical signs and symptoms, and in whom we attempted to routinely perform CRP and PCT tests. Nevertheless, a significant amount of the total included children did not have PCT measured, and our findings might represent a selected population of febrile children at higher risk of SBI, requiring future studies in other settings to confirm our findings. Missing PCT values were largely because of prospective sampling difficulties, possibly indicating a restricting factor for the routine use of this test. If anything, it seems that the presented population is at higher risk for SBI than the total included population, with sampling being more successful likely due to a higher proportion of venesections rather than capillary sampling methods. Yet, we feel our cohort is still more representative of a population of febrile children at risk for SBI in the ED than most previous studies (10, 11, 12, 35, 36, 37). The children with only CRP approximated the characteristics of children with both CRP and PCT closely, diffusing any selection bias. Another strength is the strong discriminative ability and reasonable calibration of the Feverkidstool in this new population, supporting its external validity. Previously, in a validation study conducted at several European emergency care departments, prediction rules for detecting SBIs that comprised only clinical signs lacked discriminative value to be useful in clinical practice (38).

Differences in data collection and population characteristics between the two settings were a limitation, but this heterogeneity also forms a strong test of external validity. Next, as is inherently related to large prospective observational studies, some data on clinical signs and symptoms were missing. The amount of missing data was limited (Table 1), and by multiplying imputing missing values the observed effects of all available data could be used, estimating the incremental value of PCT more precisely (39). In addition, CRP levels, as well as other clinical data, were available for the clinician, whereas PCT values were not. Although CRP was not among the main criteria of the reference standard, it might have influenced clinical management and requests of additional diagnostic tests, as well as final coding of the outcome categories. Another limitation was posed by the absence of a valid, and ethically justifiable, gold standard for bacterial pneumonia (40, 41). Focal infiltrates on chest radiographs in combination with suiting clinical features, as used in this study, arguably constitute the best proxy (42), at least including all children with definite lower respiratory tract infections (43). Other limitations associated with the development of the original prediction model were addressed previously (8). For example, the lack of certain reference tests was substituted with follow-up reducing the risk of any missed SBI, as such dealing with the ethical dilemma of not being able to perform all reference tests in all patients. The effect of missing reference tests remains important to discuss. In our departments, antibiotics were typically only started after senior clinical review and after performing reference tests following local and national guidelines. In addition, the standardized follow-up tackled the problem of false-negatives, i.e., children with SBI who were discharged without antibiotic treatment or additional diagnostic testing. Still, some children with SBI might have gone undetected. Noteworthy, although carefully reviewed for their impact in children with serious infections, the use of antibiotics before presentation to the ED was not used as an exclusion criteria. This might have influenced the coding of the outcome diagnosis of children with negative culture results. However, antibiotics use in the Netherlands for uncomplicated infections is fairly low (44), and reference tests, for example, for children suspected of UTI, will mostly be taken before starting antibiotics in both primary and secondary care. In addition, children were only included if the research laboratory markers were taken at the same time as the blood cultures and before intravenous antibiotics were given. In addition, children on antibiotic prophylaxis for an underlying chronic illness were not eligible for the study. Another limitation was inter-rater variability, while still needing to be taken into account, was reduced by considering mostly objective clinical parameters and was considered of relatively limited importance for this robust validation study. Finally, a number of other performance measures have been developed to report on incremental diagnostic values, most notably the net reclassification index, decision curve analysis, and net benefit (23). However, at present these measures for dichotomous outcomes have not been fully developed for polytomous models.

Conclusion

The Feverkidstool can now be considered as a validated tool for supporting clinical decision-making, such as to safely discharge febrile children from the ED. Our data suggest that CRP and PCT were equally valuable predictors in addition to clinical signs and symptoms for predicting SBI in our population of febrile children. Using CRP and PCT in the diagnostic management of febrile children at the ED appears useful, but our findings do not support routine dual use of both CRP and PCT.