Systematic review of clinical prediction models to support the diagnosis of asthma in primary care

Diagnosing asthma is challenging. Misdiagnosis can lead to untreated symptoms, incorrect treatment and avoidable deaths. The best combination of clinical features and tests to achieve a diagnosis of asthma is unclear. As asthma is usually diagnosed in non-specialist settings, a clinical prediction model to aid the assessment of the probability of asthma in primary care may improve diagnostic accuracy. We aimed to identify and describe existing prediction models to support the diagnosis of asthma in children and adults in primary care. We searched Medline, Embase, CINAHL, TRIP and US National Guidelines Clearinghouse databases from 1 January 1990 to 23 November 17. We included prediction models designed for use in primary care or equivalent settings to aid the diagnostic decision-making of clinicians assessing patients with symptoms suggesting asthma. Two reviewers independently screened titles, abstracts and full texts for eligibility, extracted data and assessed risk of bias. From 13,798 records, 53 full-text articles were reviewed. We included seven modelling studies; all were at high risk of bias. Model performance varied, and the area under the receiving operating characteristic curve ranged from 0.61 to 0.82. Patient-reported wheeze, symptom variability and history of allergy or allergic rhinitis were associated with asthma. In conclusion, clinical prediction models may support the diagnosis of asthma in primary care, but existing models are at high risk of bias and thus unreliable for informing practice. Future studies should adhere to recognised standards, conduct model validation and include a broader range of clinical data to derive a prediction model of value for clinicians.

various means, though certain limitations were shared by several studies.

Participant selection
Participant selection was at low risk of bias for three studies. 2,3,4 Choi et al 5 gave little detail about inclusion criteria and was rated unclear. Three studies were judged high risk of bias; Hall et al 6 included a high proportion (40%) of children aged between 6 months and 5 years, a group notoriously difficult to make a diagnosis of asthma. Hirsch et al 7 conducted analyses on a small, selected sample drawn from a cross-sectional survey of the general population.
Lim et al 8 also used an overly selected sample, by only including adults who had an FEV1 greater than 70% of their predictive value on spirometry.

Predictors
Three studies were rated at low risk of bias for the selection and measurement of predictors used in their models. 2,3,7 The remaining four models had unclear risk of bias as there was insufficient information to judge if predictor assessments had been made without knowledge of outcome data. [4][5][6]8 Outcome Five studies were rated at high risk of bias for the outcome used. 2,4,[6][7][8] In two studies the outcome was assessed without being blind to predictors; Lim et al 8 stated that responses to the questionnaire used as their predictor variables were considered when deciding the outcome. Metting et al 2 used spirometry measures to inform both predictor variables and the outcome.
The timing of assessments introduced bias in one study. Tomita et al 4 started inhaled corticosteroid treatment after the first assessment of participants (when the predictor variables were measured) which is likely to have influenced the results of their outcome measure, methacholine bronchial provocation, performed up to eight weeks later.
The outcome measure used by Hall et al 6 had several potential sources of bias. Their outcome was based on the judgement of healthcare providers at each recruitment site, yet it was unclear how many providers were used, their medical background or if any training was provided. Providers based their decision on reversible airflow obstruction, measured in two ways; either clinically by the relatively subjective resolution of symptoms, or objectively by spirometry. Yet, spirometry is difficult to achieve in young children, 9 and in this study, was only attempted in children aged seven or above. Therefore, data for spirometry was only available for 80 of the 211 participants.
Hirsch et al 7 combined the assessment of three experts to assign a probability of asthma for each of the 180 participants based on a clinical assessment which included reversibility and bronchial challenge tests. Rather than base their outcome on the result of an objective test, participants were categorised with asthma if the consensus probability of asthma was 50% or greater, introducing bias into their measure.
Two studies were at unclear risk of bias for outcome based on a lack of information regarding the timing of predictor and outcome measurement. 3,5 In addition, Choi et al 5 did not report if the outcome measure was assessed blind to the predictor variables and the same outcome was not used for every participant as asthma could be classified based on reversibility or bronchial provocation.

Analysis
Five studies were rated at high risk of bias due to the methods used in analysis. [2][3][4][5]7 Seven participants were excluded from the model building of Schneider et al 3 with a further 81 missing from the analysis in the combined model. In addition, selection of candidate predictors in the model was based on univariate analysis, which is known to introduce bias when completing multivariable modelling. 10 Similarly, Tomita et al 4 initially used univariate analysis in selecting their candidate predictors and did not include all the enrolled participants in their analysis. Choi et al 5 also did not explain their handling of missing data, despite 56 participants not having data for six questions. Additionally, the score (weights) attached to each predictor in their model did not match the regression output. 5 Hirsch et al 7 excluded 21 participants from their derivation sample due to missing data, leaving 180 participants and only 84 with the outcome. The events per variable was small (5.6), and at high risk of bias. Metting et al 2 categorised continuous variables and excluded 135 individuals who had data missing at random.
Two studies were rated at unclear risk of bias for analysis, as information regarding the handling of missing data, selection of predictor variables, model overfitting and the weights assigned to each predictor was not reported. 6,8 Applicability Overall, one study had low overall applicability concern, 3 four had high 2,4,6,7 and two were rated unclear. 5,8 The selection of participants was the major reason for studies not closely matching the review question.

Participant selection
High concern for the participants or selection method not matching the review question was found in four studies. 2,4,6,7 Hirsch et al 7 conducted a postal survey, inviting all adults to take part, therefore the initial sample was not confined to those with symptoms suggesting asthma. Hall et al 6 included children below the age of five years, which made up 40% of the primary and secondary care samples they used. Metting et al 2 included participants who had been referred by their GP to receive further assessment, although the asthma/COPD referral service is situated at the interface between primary and secondary care. 39% of the 4129 participants recruited by Tomita et al 4 had abnormalities on x-ray, indicating that a large number of participants presented with symptoms suggesting an alternative diagnosis to asthma.
Two studies at unclear applicability concern, recruited from hospital outpatient settings in South Korea. 5,8 Each sample was judged to be equivalent to primary care, given the limited availability of primary care services in the country. 11 However, it is likely that patient characteristics would be different from those presenting to primary care in a country where it is established.

Predictors
The definition or assessment of the predictors matched the review question closely in all included studies.

Outcome
The outcome in four studies closely matched the systematic review question and were rated low applicability concern. 2-5 The applicability of one study was unclear due to the timing of the outcome assessment in relation to initial testing being unreported. 8 Two studies were at high applicability concern; the inclusion of children below the age of five years by Hall et al 6 meant that making an accurate diagnosis of asthma for all participants was unlikely; Hirsch et al 7 categorised individuals as having asthma if their probability was 50% or above, which did not closely match the review question. Reason for exclusion Bansal 2001 Present a method of identifying asthma patients from a sample of patients prior to entering a trial, not for use in clinical consultation. Barnes 1999 Validates algorithm of Panhuysen. Use in identifying asthma patients in an epidemiological study. Not for use in clinical consultation Bicherakhov 1994 Outcomes for asthma are not separate or data relating to the asthma outcome is not extractable.

Bonner 2006
Case detection in a pre-school sample, not for use in clinical consultation.
Burge 1999 Investigate the interpretation of peak expiratory flow measurements using neural network; the predictive value of more than one variable was evaluated but not combined to produce a diagnostic estimate.

Carroll 2012
The predictive value of more than one variable was evaluated but not combined to produce a diagnostic estimate

Cave 2016
Algorithms for identifying patients in electronic health record, not for use in clinical consultation.

Deng 2010
The focus is on case identification in population based studies, not for use in clinical consultation.

Eysink 2005
The CPM was derived to predict future risk of asthma and over half of the participants included were children less than five years old Fukuhara 2011 Patients included were not from primary care or equivalent setting. The reference standard used is not based on an internationally recognised definition of asthma.

Grassi 2003
Population based screening questionnaire, not a CPM.

Holleman 1993
Patients included were not from primary care or equivalent setting. Outcomes for asthma are not separate or data relating to the asthma outcome is not extractable.

Jamrozik 2009
The CPM was derived to predict future risk of asthma Jones 2004 School based case detection, not for use in clinical consultation.

Kable 2001
Variables included in the model are not clearly reported (doesn't allow the probability of asthma to be calculated for other individuals) Lee 2015 Patients included were not from primary care or equivalent setting. Li 1998 The algorithm was created based on expert opinion, not a CPM. Liebhart 1998 Patients included were not from primary care or equivalent setting.

Ma 2017
Outcomes for asthma are not separate or data relating to the asthma outcome is not extractable.

Menezes 2015
Does not present a CPM that could be used in clinical practice and the reference standard used is not based on an internationally recognised definition of asthma.

Murray 2017
Not a CPM -the original algorithm tested in this study was derived by economic modelling with expert recommendation Panhuysen 1998 Algorithm designed for identifying those with asthma in a sample recruited for a clinical trial, not for use in clinical consultation.

Pralong 2013
Patients included were not from primary care or equivalent setting.

Redline 2003
Develop a score to screen school children (with and without symptoms) likely to have asthma so that they can be referred for a diagnostic assessment, not for use in clinical consultation.

Redline 2004
Develop a score to screen school children (with and without symptoms) likely to have asthma so that they can be referred for a diagnostic assessment, not for use in clinical consultation.

Remes 2002
Variables used in the model are not clearly reported

Rother 2015
Patients included were not from primary care or equivalent setting.

Schneider 2003
The algorithm was created based on expert opinion, not a CPM.

Schneider 2012
Does not present a CPM that could be used in clinical practice.
Sistek 2001 The reference standard used is not based on an internationally recognised definition of asthma.

Sunyer 2007
Does not present a CPM that could be used in clinical practice.

Thiadens 1998
Data relating to the asthma outcome is not extractable.

Thiadens 2000
The population analysed is not representative of a primary care population Thorat 2017 Patients included were not from primary care or equivalent setting.

Topalovic 2017
Investigate algorithms for lung function test interpretation.

Torchio 2005
Patients included were not from primary care or equivalent setting and data relating to the asthma outcome is not extractable.

Tyagi 2014
Does not present a CPM that could be used in clinical practice.

Vandenplas 2005
Patients included were not from primary care or equivalent setting.

Wahn 2000
Non-original study -review article

Xi 2015
Describe algorithms for searching electronic databases not for clinical use.

Yu 2004
The reference standard used is not based on an internationally recognised definition of asthma.

Zolnoori 2012a
Outcomes for asthma are not separate or data relating to the asthma outcome is not extractable and variables used in the model are not available in routine clinical practice.

Zolnoori 2012b
Outcomes for asthma are not separate or data relating to the asthma outcome is not extractable and variables used in the model are not available in routine clinical practice.

Participants with any missing data:
Only those who answered 'yes' to question 1 were asked questions 1-1. to 1-6. (n = 246) Therefore 56 had missing data for 6 of the variables used.

Methods for handling missing data:
Not reported

Strengths:
Primary and secondary care populations included.

Limitations:
Development of model is unclear. Reporting of the performance of the model is also poor.
Inclusion of children below the age of 5 years.

Generalisability:
"In almost any population, a negative response to all 4 questions will mean that the child is very unlikely to have asthma."

Model building Model Evaluation
Modelling method: Logistic regression

Model assumptions met:
Not reported

Selection of predictors:
Not clear -they only use 4 questions from the survey -why they choose these is not reported.
They state: (in the abstract) "Four questions on the survey were shown to be sensitive and specific for asthma."

Recruitment:
Method: Initially postal questionnaire sent to all adults (≥16 years) registered at two GP practices.
Then selected individuals to achieve an asthma enriched sample. Participants were stratified for the likelihood they would have asthma based on responses to six questions.
Individuals from one practice were selected from each stratum to provide a stratified sample with equal numbers of asthma / no asthma who then attended for clinical assessment.
Setting: General Practice.

Eligibility:
Inclusion: For the postal questionnaire -all adults registered at two GP practices.
Clinical evaluation of patients was drawn from the responders Exclusion: Age < 16 years.
Study Dates: Derivation occurred in the 1995 dataset.
Validation in the 2001 dataset.

Definition:
Probability of asthma: <50%, 50-90%, >90% "Those reviewed individuals in whom the consensus estimate of probability of asthma was 50% or more were designated clinically asthmatic and the remainder were designated clinically non-asthmatic."

Measured:
3 experts were provided with information from the clinical assessment (not the questionnaire data) and asked to categorise each patient into one of the three probability categories The diagnostic value of the questionnaire was evaluated by ROC analysis. The AUC of the ROC curve was 0.610 ± 0.029" COI: None declared.

Strengths:
"elucidate the clinical validity of a selectively chosen questions recommended by GINA for diagnosing asthma in the general adult population" Limitations: "no healthy control group" No validation.
The recruiting hospital was in a city with "relatively severe" air pollution.