A setting-specific asthma prediction score for preschool children with wheezing and/or dyspnoea presenting in primary healthcare is needed since existing indices are mainly based on general populations.
To find an optimally informative yet practical set of predictors for the prediction of asthma in preschool children at high risk who present in primary healthcare.
A total of 771 Dutch preschool children at high risk of asthma were followed prospectively until the age of six years. Data on asthma symptoms and environmental conditions were obtained using validated questionnaires and specific IgE was measured. At the age of six years the presence of asthma was assessed based on asthma symptoms, medication, and bronchial hyper-responsiveness. A clinical asthma prediction score (CAPS) was developed using bootstrapped multivariable regression methods.
In all, 438 children (56.8%) completed the study; the asthma prevalence at six years was 42.7%. Five parameters optimally predicted asthma: age, family history of asthma or allergy, wheezing-induced sleep disturbances, wheezing in the absence of common colds, and specific IgE. CAPS scores range from 0 to 11 points; scores <3 signified a negative predictive value of 78.4% while scores of ≥7 signified a positive predictive value of 74.3%.
We have developed an easy-to-use CAPS for preschool children with symptoms suggesting asthma who present in primary healthcare. After suitable validation, the CAPS may assist in guiding shared decision-making to tailor the need for medical or non-medical interventions. External validation of the CAPS is needed.
Childhood asthma usually starts during the preschool years. Unfortunately, it is currently not possible to test for or alter the natural history of asthma accurately in young children,1,2 so asthma during the preschool years is frequently underdiagnosed. About 10 asthma risk prediction indices for young children have been developed, although only one was developed in a primary care setting where most of these children present with respiratory symptoms for the first time and where most are treated. Risk stratification using prediction indices may be useful for clinical and research purposes. For example, risk stratification supports the tailoring of treatment strategies such that benefits and harms may be properly weighed.3 It may therefore improve the follow-up of children at risk by decreasing the disease burden and improving management, resulting in fewer respiratory symptoms, exacerbations, and emergency medical visits.4,5
Prediction indices are valuable because the information itself may contribute to patients' quality of life.6 Patients' emotional response to medical testing is a natural effect. The information involved in a high predicted probability of a young child having asthma at six years of age may enable them (or their parent/guardians) to make sense of the symptoms or to adjust their lifestyle and living conditions in light of the results, while a low predicted probability may alleviate their concerns.6,7
In addition, various research groups are investigating the underlying pathogenic mechanisms and preventive strategies for asthma,8 and as soon as effective therapy that is either costly and/or has adverse effects emerges, risk stratification will be useful.3
Existing predictive indices have an uncertain value for the primary healthcare setting because the majority of the cohort studies were based on parental questionnaires in a general population,9,
In 2004 we started a prospective cohort of young children at high risk of developing asthma. The objective was to construct a clinical index for predicting asthma in preschool children at risk of asthma presenting in primary care. Here we describe the development and performance of a novel clinical asthma prediction score (CAPS).
AppendixDesign and subjects
The subjects were children aged one to five years in the Netherlands who were participating in the ARCADE (AiRway Complaints and Asthma Development) prospective cohort study.4 They had presented at one of the pertinent 14 primary care clinics in the previous 12 months with complaints of recurrent coughing (≥2 visits), wheezing (≥1 visit), and/or shortness of breath (≥1 visit). The study was approved by the Netherlands Central Committee on Research Involving Human Subjects (CCMO/P04.0098C) and the details of the study have been published elsewhere.4 Written informed consent was obtained from the parents before all measurements.
For this study we derived demographic data and data with respect to the child's clinical history from questionnaires obtained by the Core Questionnaire of the International Study of Asthma and Allergy in Children (ISAAC), which was administered at baseline and at the age of six years.
Total immunoglobulin E (IgE) and specific IgE directed against cat, dog, and house dust mite were determined by radioallergosorbent test (RAST) in all the children at baseline. IgE positivity to cat, dog, and house dust mite was defined as >0.35kU/L.4
Spirometry and bronchial hyper-responsiveness
At six years of age, spirometry and bronchial hyper-responsiveness (BHR) assessments were obtained in children with wheezing, shortness of breath, recurrent coughing, or use of asthma medication (β2-agonists, inhaled corticosteroids) during the previous 12 months. BHR to increasing doses of methacholine was measured according to Cockcroft et al.(online supplement reference 3).
Outcome: asthma diagnosis at age six
The diagnosis of asthma at six years of age was defined as: having persistent symptoms of asthma and/or using asthma medication in the last year in combination with BHR (defined as the methacholine dose (<8.0mg/ml) causing a 20% reduction in forced expiratory volume in one second (PC20-FEV1)) or reversibility (>10% increase in FEV1 after short-acting β2-agonists).4 Persistent symptoms were defined as self-reported complaints during the previous 12 months of wheezing and/or shortness of breath and/or recurrent coughing.
Children who had not experienced asthma-related symptoms in the previous year or who had not used asthma medication in the previous year were not invited for lung function measurement. They were considered as not having asthma.
Probability of asthma at the age of six
Depending on the direction and magnitude of the selected predictors as measured by regression coefficients, we developed the Clinical Asthma Prediction Score (CAPS) as a points system which is easy to use, without requiring a computer or calculator.16 To use the CAPS in clinical practice, thresholds are needed to either rule in or rule out asthma at the age of six years. Therefore we set two thresholds — a lower limit at 30% below which we assumed that many physicians may well choose a ‘wait and see’ policy, and an upper limit at 60% above which it was assumed that physicians may pursue a more active management such as a trial of asthma treatment. These thresholds imply that children with a predicted risk of asthma risk between 30% and 60% will be assigned to the area of clinical indecision in which a ‘watchful waiting’ policy may be defensible.
Details of the statistical analysis are described in Appendix 1, available online at www.thepcrj.org. Briefly, we analysed whether dropout was selective and missing predictor data were imputed using multiple imputation.17 Twelve candidate predictors of all available data were preselected based on clinical expert opinion after removing five due to collinearity (Table 1). Bootstrapped (1000x) backward selection was used to select the final important predictors whose regression coefficients were shrunk to facilitate better performance when the model is applied to other data or future patients.18 The predictive performance of the prediction index was assessed by its calibration and discrimination abilities.
Furthermore, the additional value of specific IgE was calculated by the additional area under the ROC curve (AUCdiff) and the net reclassification improvement (NRI).19 All statistical analyses were carried out in Stata/SE Version 10.1 (Stata Corp, College Station, TX, USA).
Of the 771 children, 438 (56.8%) had complete information at baseline and an asthma diagnosis at the endpoint (Figure 1). Of these, 187 (42.7%) had asthma at the age of six. Table 1 shows the univariable frequencies of the preselected candidate predictors. The mean (SD) age at baseline was 3.00 (1.29) years and 189 (43.1%) were female. In 156 children (35.6%), asthma was present in either the parents or siblings. Missingness analyses showed no strong indications for selective drop out (Appendix 1 available online at www.pcrj.org).
Five predictors were included (Table 2): age, family history of asthma or allergy, wheezing-induced sleep disturbances in the previous 12 months, wheezing in the absence of a cold in the previous 12 months, and specific IgE test. The discrimination of the model expressed as area under the ROC curve was 0.73 (SE=0.024). Calibration was satisfactory (Table 2).
IgE and no IgE in the model
The model performance was calculated for the model including age, family history, wheezing-induced sleep disturbances, and wheezing in the absence of a cold but excluding IgE. The discrimination of the model expressed as area under the ROC curve was 0.71 (SE=0.024). Calibration was good (Table 2). Adding specific IgE to the model increased the AUC by 2.8 percentage points (AUCdiff= 0.028, 95% CI 0.0047 to 0.051, SE=0.012). Net reclassification analysis showed that 11.23% (p=0.09) of the children were correctly reclassified when IgE was added to the model based on history items only (Table 3).19
Probability of asthma at the age of six using CAPS
A points system ranging from 0 to 11 was developed. Thresholds to accept or reject the presence or development of asthma were set at 30% and 60%, respectively (Table 4).16 The positive predictive values (PPVs) of the CAPS (≥7=probability of asthma of ≥60%) were 74.3% (CAPS=7) and 100% (CAPS=11) and the negative predictive values (NPVs) of the CAPS (<3=probability of asthma of ≤30%) were 78.4% (CAPS=2) and 88.2% (CAPS=0) (see online supplement). Figure 2 shows asthma risks as a function of CAPS scores.
The clinical asthma prediction score (CAPS) we developed in a prospective cohort study facilitates the risk stratification of preschool children who attend primary healthcare with wheezing, dyspnoea, or recurrent coughing. The predictive ability of the CAPS may assist physicians to distinguish between a relatively low risk (14%) and a very high risk (90%) of developing asthma. The CAPS uses easily accessible information on age, family history of asthma or allergy, wheezing-induced sleep disturbances, and wheezing in the absence of a cold, and an additional blood sample to obtain specific IgE.
The use of the CAPS may enable physicians to reduce the number of children unnecessarily labelled with asthma (low CAPS). For this group, the prior probability went from 42.7% to 21.6%; this helps to reduce unnecessary testing, treatment, and uncertainty. On the other hand, CAPS may allow physicians to identify preschool children who may need proper asthma management (high CAPS). For this group, the prior probability went from 42.7% to 74.3%; a high CAPS may therefore reduce the burden of asthma in these children through better symptom control and fewer exacerbations and hospital admissions. In both situations there will be a reduction in costs. Children with an intermediate CAPS (3>7) retain a probability similar to the probability before applying the CAPS. In this group a watchful waiting policy seems justified, in line with the British guideline,20 which subsequently suggests a watchful waiting approach with review in the majority of cases with intermediate risk predictions.
Strengths and limitations of this study
Our study has the following strengths. First, we selected children from a relevant population — namely, those children in whom an attending primary care physician had considered asthma. Second, all predictors collected are easily accessible in primary care. Third, we used a stringent definition of asthma to ensure a reliable reference standard diagnosis for research purposes. Fourth, we followed the relevant STARD checklist criteria — a strict protocol with well-defined inclusion and exclusion criteria for the setting and population and a generally accepted reference standard.21 Fifth, shrinkage of the magnitude of the regression coefficients was used to counteract over-optimism of our model, thus enhancing its validity outside our data set.17,18 Sixth, we used NRI to measure the additional value of specific IgE to the model. NRI is a marker to express the model performance and to compare models, focusing on the correct reclassification of patients with and without asthma; it appears to be a clinically more sensible approach than the differences between AUCs.
We see the following limitations to our work. First, our study is a model development study without external validation. We did, however, use bootstrapped variable selection and shrinkage of regression coefficients to temper an over-optimistic performance of CAPS in external populations. Second, we have a high dropout rate. Missingness analyses showed that dropout was higher in children of parents who smoked (OR=1.88) and those with persistent (>1 month) coughing (OR=1.62). Parental smoking may be explained as a psychological cause for dropout; parents who smoke may feel guilty or embarrassed that smoking might have a negative influence on their child's physical condition. We assumed that persistent coughing as a cause of dropout was a coincidental finding. The high dropout rate may also be explained by the fact that, due to an extension of follow-up, participants had to explicitly renew their informed consent. Third, although some combinations of predictor values yield low (<30%) or high (>60%) asthma probabilities, almost 58% of the children in our study population had a predicted probability between 30% and 60%, close to the overall prevalence of 42%. For such children, CAPS results are not likely to change clinical management. Finally, the choice of different thresholds may lead to different results. As far as we are aware, there is no evidence-based advice on thresholds for this type of decision-making in the asthma literature. We decided to use thresholds as a clinically intuitive alternative to more formal methods of reclassification and as an attractive alternative for classical methods of predictive performance such as the area under the ROC curve.22 We chose a lower limit of 30% to rule out asthma and an upper limit of 60% to rule it in, which implies an area of indecision between the probabilities of 30% and 60%. We think that these limits — or limits close to these — may be relevant in daily primary care practice.23 However, evidence-based thresholds would require formal benefit-harm or cost-utility analyses that incorporate treatment effects, potential harms (and costs), and children's (and their parents') valuation of all different health outcomes. Although utility measurement in children is not simple, instruments to measure quality of life in children do exist.24,25
Interpretation of findings in relation to previously published work
The API represents innovative research and is the most investigated index. This index was developed in the Tucson general population cohort9 and has also been tested in other populations (e.g. in the Leicester Respiratory Cohort, n=1,954). Unfortunately, one of the predictors — a blood sample for eosinophilia at one year of age — was omitted from the Leicester cohort.26
Several predictive indices for preschool children have been published in the past years.9,
Most research groups used different populations to develop their indices and did not focus on the target population (i.e. preschool children presenting with respiratory complaints to their primary care physician). Eysink et al. included children aged one to four years visiting their primary care physician for persistent coughing of at least five days, and followed them to develop a predictive index for persistent asthma.29 Vial Dupuy et al. retrospectively selected wheezing infants from the medical files of an outpatient department of a paediatric pulmonology and allergy centre. The present study resembles the earlier Dutch study in the primary care setting except that the previous study was retrospective and the children were younger.31 Frank et al. selected their patients by postal survey on parent-reported wheeze, although this matched the clinicians' findings in fewer than 50% of cases.11,32 In the seven remaining studies, the participants were selected from general population birth cohorts,9,10,12,13,27,28,30 which makes these indices of doubtful applicability to most clinical settings unless major adaptations are made.
The predictive abilities of existing indices are, in general, not high. For example, Castro-Rodriguez et al. claim that the API is a ‘very useful tool’ to predict the long-term outcome of preschool wheeze ‘in every healthcare setting worldwide’. However, it was calculated that the performance of the API in more high-risk populations would be poorer.15 A negative API result in the Tucson cohort (‘asthma’ prevalence at age six of 22.2%) yielded a 10% probability of asthma (NPV=0.90). Overall, earlier predictive indices also had relatively high NPVs but disappointing PPVs, which makes them more useful to rule out asthma than to rule it in.26
Implications for future research, policy and practice
The implementation of a simple risk score in daily care is attractive to both physicians and parents. It allows physicians to provide tailored medical care and follow-up in children at high risk of developing asthma, and to apply a wait-and-see policy in those with a low or intermediate risk of asthma. The increased likelihood of predicting which children will develop asthma will help physicians to explain treatment and prognosis to parents, reducing uncertainties and improving adherence in their children.
Our study was a model development study, extended with internal validation using bootstrapped variable selection and shrinkage of regression coefficients, with the aim of developing a multivariable prediction model. Before the model can be confidently applied in daily clinical practice, external validation is needed.18 Also, further research should focus on new markers or tests to decrease the number of patients remaining inside the area of indecision (i.e. to increase discrimination).
Risk stratification serves three distinct functions in controlled trials: (1) assembling the appropriate spectrum of patients and allowing more efficient subgroup analysis14 and better assessment of baseline comparability; (2) it may also distinguish persistent and transient complaints early on and therefore increase the accuracy of the RCT results;3 and (3) the CAPS may be used to re-analyse early intervention trials to assess treatment response variability as a function of the CAPS score.3 Such trials should, of course, allow the calculation of a CAPS score.
We have developed an easy-to-use CAPS for preschool children at high risk of asthma in a primary healthcare setting. After suitable validation, the CAPS may assist in guiding shared decision-making to tailor the need for medical or non-medical interventions.
Handling editor Niels Chavannes
Statistical review Gopal Netuveli
We would like to thank the general practitioners at HAG-net-AMC, Zorggroep Almere and Prinsenhof, Leemhuis, and Buitenhuis for their time and effort. We also thank all parents and children for participating in the study. Our thanks also go to Pauline van Steenwijk, Machteld IJff, Alice Karsten, and Albertien Buijs for their invaluable help with data collection and other logistics.
Funding This study was financially supported by the Netherlands Asthma Foundation (3.4.06.078) and Stichting Astma Bestrijding (2008/027).
Appendix 1. Supplementary material
The subjects were one- to five-year-old children in an urban agglomeration in the western part of the Netherlands who were participating in the ARCADE (AiRway Complaints and Asthma Development) prospective cohort study,1 and had presented at one of the pertinent 14 primary care clinics in the previous 12 months with complaints of recurrent coughing (≥2 visits), wheezing(≥1 visit), and/or shortness of breath (≥1 visit).
The study was approved by the Netherlands Central Committee on Research Involving Human Subjects (CCMO/P04.0098C).
Eligible for participation were children aged one to five years who visited their primary care physician (in the Netherlands always a primary care physician) with the respiratory complaints of wheezing and/or dyspnoea in the previous 12 months or following 6 months after the start of ARCADE. Children who visited their primary care physician at least twice with the symptom of coughing were also eligible. Exclusion criteria were a confirmed diagnosis of asthma at inclusion (based on spirometry and challenge test), known temporary stay in the region, or parents unable to read or understand Dutch/English. The primary care physician sent postal information about the study to the parents (or caregiver, from now on called ‘parent’) of eligible children, including a reply card, which they could return if they considered participation. After returning the card, further detailed written information and an informed consent form was sent.
At baseline (T0) and at age 6 years (Tend) the parents received a questionnaire on (changes in) housing conditions, family history of allergy, asthma and eczema, presence of pets, breastfeeding, and asthma-related symptoms. Also, information about wheezing, rhinitis, eczema, cough, and phlegm was obtained by the Core Questionnaire of the International Study of Asthma and Allergy in Children (ISAAC).2
Total immunoglobulin E and specific IgE directed against cat (C), dog (D), and house dust mite (HDM) were determined by radioallergosorbent test (RAST) in all the children at baseline.3 IgE positivity to house dust mite, cat and/or dog dander was defined as >0.35kU/L.1 The blood samples were investigated by independent chemical analysts.
Spirometry and bronchial hyper-responsiveness
At six years of age spirometry and bronchial hyper-responsiveness (BHR) assessments (using Pulmoassist 2 spirometer; Jaeger, Würzburg, Germany) was obtained in children with symptoms (wheezing, shortness of breath or recurrent coughing) or use of asthma medication (β2-agonists, inhaled corticosteroids (ICS)) during the prior 12 months. BHR to increasing doses of methacholine was measured according to the modified method of Cockcroft et al. (using DeVilbiss 646 nebuliser; DeVilbiss, Somerset, MA, USA, with an output of 0.13mL/min).4 Tests were conducted by independent lung function assistants. The parents of children using asthma medication were asked to withhold all bronchodilators for 48 hours prior to the test. In the case of shortness of breath, children received ongoing short-acting β2-agonists up to 8 hours before the test.1
Outcome: asthma status at age six
The diagnosis of asthma (masked for predictor values) at six years of age was defined as having persistent symptoms of asthma and/or using asthma medication in the last year in combination with BHR (defined as the methacholine dose (<8.0mg/ml) causing a 20% reduction in FEV1 (PC20-FEV1)) or reversibility (>10% increase in FEV1 after short-acting β2-agonists).1 Persistent symptoms were defined as self-reported complaints during the previous 12 months of wheezing and/or shortness of breath and/or recurrent coughing.
Children who had not experienced the asthma-related symptoms in the previous year or who had not used asthma medication were not invited for lung function measurement. They were considered as not having asthma.
Probability of asthma at the age of six
Depending on the direction and power of the selected predictors measured by regression coefficients, we developed a clinical asthma prediction score (CAPS) as a points system that is easy to use and does not require a computer or calculator.5 To use the CAPS in clinical practice, thresholds are needed to either rule in or rule out asthma at the age of six years. We set two thresholds — a lower limit at 30% below which we assumed that many physicians may well choose a “wait and see” policy and an upper limit at 60%, assuming that physicians may pursue more active management such as a trial of asthma treatment. These thresholds imply that children with a predicted asthma risk of between 30% and 60% were assigned to the area of clinical indecision in which a “watchful waiting” policy may be defensible.
Analyses of dropouts
Multivariable logistic regression was used to analyse selective dropout (see Table A1). First we analysed the dropout on inclusion criteria (n=730), coughing, wheezing or shortness of breath.6 Second, we analysed for selective dropout using the other candidate variables (n=597).
Handling of missing data
Multiple imputations using all predicting covariates by iterative chained equations (ICE) to replace missing values were used. Ten imputed datasets were generated.7
Development of the model
Candidate predictors were selected based on clinical expert opinion. In the case of highly correlated candidate predictors (if Spearman >0.7), only one was analysed. Further, predictors were selected using bootstrapped backward selection (BBS method) with a significance level α of 0.157 (Akaike information criterion; AIC), by fractional polynomial multivariable modelling according to Royston and Sauerbrei.8 In each imputed dataset, backward variable selection was used in 1,000 bootstrapped samples; if the bootstrap inclusion fraction (BIF) of a covariate was more than 667/1,000 in each imputed dataset, it was selected as a predictor. After selecting the predictors, the coefficients of the predictors were multiplied by a fixed heuristic shrinkage factor.9
Performance of asthma prediction model
The predictive performance of the asthma prediction model was assessed by calibration and discrimination. Calibration (Hosmer-Lemeshow test) describes the accuracy of the measurements between the observed probabilities and the predicted probabilities in independent datasets. Discrimination represents the model's ability to discriminate between patients with and without asthma during our study. The most familiar measurement for discrimination is the area under the receiver operating characteristic (ROC) curve or concordance (c) statistic: a value of 0.5 represents chance and a value of 1 represents perfect discrimination.
The additional value of the most invasive predictor — namely, blood testing on specific IgE — was calculated using the area under the ROC curve (AUCdiff) and the NRI, an instrument that focuses on the reclassification of patients with and without asthma. The thresholds were <30% for a small probability of asthma at the age of six years and >60% for a very high probability of having asthma in later life. In the light of the upward movement for events and the downward movement for non-events, an index could be calculated.10 In addition, the results (including SE and 95% CI) were bootstrapped 1000 times in order to test the AUCdiff.11
Probability of asthma at age six by clinical asthma prediction score (CAPS)
To make complex statistical models useful to practitioners, Sullivan et al. developed a points system that is easy to use and does not require a computer or calculator5 (see Table A2). After organising risk factors into categories and determining reference values, referent risk factor profiles (WiREF, i=1,…,4) were selected for each risk factor. A constant (B) was defined, reflecting the increase in risk associated with a 1-year increase in age (B=1*(0.333)=0.333). Points associated with each of the categories of the risk factors were computed by: Pointsij=βi(Wij-WiREF)/B. Points were rounded to the nearest integer.5 Further, the estimates of the risk associated with each point total was determined by calculating the linear predictor and probability using the multiple logistic regression equation (Figure A1). Furthermore, the positive predictive and negative predictive values of CAPS (≥7 and <3=probability of asthma of ≥60% and ≤30%, respectively) were calculated (see Table A3).
To estimate the risk of developing asthma based on the statistical model, the simplified CAPS could be used in daily practice.
All statistics were carried out in Stata/SE Version 10.1 (Stata Corp, College Station, TX, USA).
van Wonderen KE, van der Mark LB, Mohrs J, et al. Prediction and treatment of asthma in preschool children at risk: study design and baseline data of a prospective cohort study in general practice (ARCADE). BMC Pulm Med 2009;9:13. http://dx.doi.org/10.1186/1471-2466-9-13
Asher MI, Keil U, Anderson HR, Beasley R, Crane J, Martinez F, et al.: International Study of Asthma and Allergies in Childhood (ISAAC): rationale and methods. Eur Respir J 1995;8:483-91. http://dx.doi.org/10.1183/09031936.95.08030483
Stapel SO, Eysink PE, Vrieze J, et al. IgE testing in capillary blood. Pediatr Allergy Immunol 2004;15:230-3. http://dx.doi.org/10.1111/j.1399-3038.2004.00142.x
Cockcroft DW, Killian DN, Mellon JJ, et al. Bronchial reactivity to inhaled histamine: a method and clinical survey. Clin Allergy 1977;7:235-43. http://dx.doi.org/10.1111/j.1365-2222.1977.tb01448.x
Sullivan LM, Massaro JM, D'Agostino RB, Sr. Presentation of multivariate data for clinical use: The Framingham Study risk score functions. Stat Med 2004;23:1631-60. http://dx.doi.org/10.1002/sim.1742
Allison P. Missing data (Quantitative Applications in the Social Sciences). 1st ed. 2001.
White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 2011;30:377-99. http://dx.doi.org/10.1002/sim.4067
Royston P, Sauerbrei W. Multivariable Model-building: a pragmatic approach to regression analysis based on fractional polynomials for modelling continuous variables. 1st ed. West Sussex,UK: John Wiley and Sons, 2008. ttp://dx.doi.org/10.1002/9780470770771
van Houwelingen JC. Shrinkage and penalized likelihood as methods to improve predictive accuracy. Statistica Neerlandica 2001;55:17-34. http://dx.doi.org/10.1111/1467-9574.00154
Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72. http://dx.doi.org/10.1002/sim.2929
Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148:839-43.