Introduction

Objective assessment of current clinical control and future risk is the foundation of a routine asthma review, inviting consideration of the reasons for poor control and underpinning the subsequent development of management plans.1,2 Asthma control can be defined in a number of ways3 but, in primary care, important components are the presence of symptoms (assessed by asking specific morbidity questions), use of relief medication, and occurrence of exacerbations (normally recorded in the patient's healthcare record). Routinely assessing and recording the results of standardised questions (Patient Reported Outcome Measures, PROMS) has therefore been promoted by initiatives to improve standards of clinical practice,4,5 including within the Quality and Outcome Framework (QOF) a ‘pay-for-performance scheme’ in the UK.6

PROMS in research and clinical practice

The well-defined process for developing and validating PROMs7 is exemplified by the widely used Asthma Control Questionnaire (ACQ)8,9 which has defined scores that represent well and poorly controlled asthma to aid clinical interpretation.3,10 Validity, however, needs to be assessed in relation to a specific purpose and setting.7 In a research context, instruments validated for the relevant context are chosen11 and administered by trained researchers using techniques which maximise the response rate and reduce completion errors.12 Use in the context of clinical care, however, introduces a range of poorly understood variables including the influence of diverse modes of administration and potential under- or over-reporting of symptoms by patients anticipating the response from their clinician. It cannot therefore be assumed that even the best validated PROM will deliver valid answers when used in routine clinical care.

The Royal College of Physicians three questions (RCP3Q): a PROM designed for use in clinical practice

The RCP3Q (Table 1) emerged from a UK consensus meeting of representatives of primary and secondary care and patient organisations as a clinical tool which ‘made sense to both clinician and patient’ and would be ‘widely used to improve standards of care’.13,14 Precursors of the RCP3Q (the Tayside Asthma Stamp and the Jones morbidity index) had been used in audits of asthma management15 by postal questionnaire to predict asthma exacerbations16,17 and, more recently, during telephone consultations to screen for patients who might benefit from a face-to-face review.18 When administered formally within a battery of questionnaires in a prospective observational study of 20 adults assessed 2-weekly for 12 weeks, there was a highly significant correlation between the RCP3Q and the ACQ (0.79, p<0.001) and a strong relationship between changes in both the scores.19

Table 1 The Royal College of Physicians three questions13
Full size table

In the UK, annual recording of the RCP3Q has been introduced as an essential component of an annual asthma review within the QOF.6 This multicentre study aimed to investigate the performance of the RCP3Qs in a real-life practice setting compared with the gold standard of the ACQ administered by self-completed questionnaire, and to examine the additional predictive value of other contextual data.

Methods

Our validation study was undertaken during 2009–2011 in practices throughout the UK. Ethical approval was granted by North West 4 Research Ethics Committee (Reference 09/H1001/103) and governance approval was provided by the participating National Health Service (NHS) Trusts.

Practice recruitment

We approached the 30 practices participating in the QOF pilot (in which all potential QOF indicators are tested for reliability, validity, and feasibility)20 and, with the support of the local Primary Care Research Networks, extended recruitment to other practices in these areas. Practices were eligible if they provided proactive asthma care (usually led by an asthma trained nurse) during which they routinely recorded the RCP3Qs. We specifically asked nurses not to change their normal practice.

Patient recruitment

Participating practices were given between 20 and 40 questionnaires to distribute (according to list size), with additional questionnaires provided on request. Adults (18 years or over) with asthma were recruited as they attended the practice asthma clinic. The only exclusion criteria were people with other significant respiratory disease, those unable to complete a questionnaire in English, or at the discretion of the clinician for significant medical or social reasons.

Data collection and handling

All data were obtained anonymously.

ACQ

At the end of the consultation, eligible patients were invited (by the asthma nurse or other member of the practice team) to complete the study questionnaire which was returned in a prepaid envelope to the research team. The questionnaire included:

  • The validated ACQ (6-question version).9 The ACQ measures clinical goals of asthma management including use of reliever medication on a scale of 0 (good control) to 6 (poor control).9 The threshold between ‘well controlled’ and ‘not well controlled’ asthma is close to 1.00.10

  • Identifiers: name of practice, demographic details (age and sex), date of completion.

RCP3Qs

All the routinely recorded answers to the RCP3Q recorded in the practice during the time they were participating in the study were extracted using an automated search of the practice database (Optimum Patient Care (OPC), Norfolk, UK www.optimumpatientcare.org). The individual patient level data included:

  • Identifiers: practice code, demographic details (age and sex), date of completion.

  • The RCP3Qs as recorded on the practice computer. In the absence of standardised coding, practices were asked how they normally coded the responses to the questions and the search modified appropriately.

  • Asthma drugs prescribed in the previous year.

  • Acute exacerbations coded in the records.

Data handling

The RCP3Q and data from records were extracted electronically to a database held by OPC. The ACQ was entered onto a separate database at the University of Edinburgh. The RCP3Q and ACQ entries were matched if the practice, age, and sex were identical and if the dates of completion were within 24 hrs. In the event of two pairs of records with the same identifiers, both were discarded.

Sample size calculation

Based on the potential availability of up to 30 practices from the QOF pilot we estimated, using simulated data, that for a sensitivity and specificity of approximately 0.8 with confidence intervals of ±0.075, we would require a sample size of approximately 250 patients.

Statistical analysis

We calculated the sensitivity and specificity of each level of the RCP3Q scale compared with the validated ACQ threshold score of ≥1 and plotted a receiver operating characteristic (ROC) curve, taking an area under the curve (AUC) of ≥0.8 to indicate good discrimination by the RCP3Q. We undertook additional analyses using ACQ thresholds of ≤0.75 (the optimal cut-off point to be confident that a patient has well controlled asthma6) and ≥1.5 (the optimal cut-off point to be confident that the patient has inadequately controlled asthma10). We examined variation between practices by making scatter plots of RCP3Q against ACQ for each practice. Statistical analysis was carried out in R 2.11 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Twelve practices took part in the study; their details are shown in Table 2. During the study period these practices recorded items from the RCP3Q for a total of 916 patients. We provided a total of 360 questionnaires to the practices although it is likely that not all were distributed; 291 questionnaires were returned of which 139 could be paired with RCP3Q responses extracted from the practice computers. The flow diagram shown in Figure 1 gives details of the process of pairing. Ten patients were excluded because either the ACQ or RCP3Q was incomplete. The remaining 129 patients from the 12 practices had valid ACQ and RCP3 scores and were thus included in the subsequent analysis. The number of patients per practice ranged from 2 to 28. The mean (SD) age of the patients was 54 (17) years and 55 (43%) patients were male. The age, sex, and ACQ score of the 152 patients whose questionnaires could not be reliably paired were similar to those included in the analysis.

Table 2 Characteristics of participating practices
Full size table
Figure 1
figure 1

Flow diagram of Asthma Control Questionnaires (ACQ) received and the process of pairing with the Royal College of Physicians three questions (RCP3Q) scores downloaded from practice computers

Full size image

Questionnaire responses

The ACQ scores ranged from 0 to 5 with a mean (SD) of 1.08 (1.03). The RCP scores ranged from 0 to 3 with a mean (SD) of 1.14 (0.86) and median of 1. Table 3 compares the distribution of scores for the two scales and shows that 25/27 patients (92.5%) with an RCP3Q score of zero had an ACQ score <1 indicating ‘well controlled asthma’. In contrast, an ACQ score of ≥1 indicating ‘not well controlled asthma’ was recorded by 26/70 patients (37.1%) with an RCP3Q score of 1 and by 28/32 patients (87.5%) with an RCP3Q score of 2 or 3.

Table 3 Distribution of RCP3Q and ACQ scores
Full size table

Diagnostic accuracy of RCP3Q and ACQ

The diagnostic accuracy of the RCP score was measured against the ACQ threshold value of 1.0 (the defined crossover point between ‘well controlled’ and ‘not well controlled’ asthma).10 With a cut-off RCP3Q score of ≥1, the sensitivity was 0.96 (95% CI 0.88 to 0.99) and specificity 0.34 (95% CI 0.24 to 0.46) while, with a cut-off RCP3Q score of ≥2, the sensitivity was 0.50 (95% CI 0.37 to 0.63) and the specificity was 0.94 (95% CI 0.87 to 0.98). These values are summarised in the ROC curve shown in Figure 2; the AUC was 0.79 (95% CI 0.75 to 0.81). The sensitivity and specificity of the RCP3Q compared with three different ACQ thresholds are shown in Table 4.

Figure 2
figure 2

Receiver operating curve for the Royal College of Physicians three questions (RCP3Q) score with the threshold of Asthma Control Questionnaire (ACQ) score of ≥1

Full size image
Table 4 Sensitivity and specificity of the RCP3Q compared with three different ACQ thresholds10
Full size table

Relationship of RCP3Q score to good and poor control

An RCP3Q score of zero was recorded for 27 patients: only one of these patients (4%) had an ACQ score of ≥1.5 (the threshold above which patients can be confidently assumed to have inadequately controlled asthma10). In contrast, an RCP3Q score of 2 or 3 was recorded for 32 patients of whom only three (9%) had an ACQ score of <0.75 (the threshold below which patients can be confidently assumed to have well controlled asthma10). Thus, in practice, a zero RCP3 score effectively rules out poor control and an RCP3 score of 2 or 3 rules out good control. However, an RCP3Q score of 1 is less able to discriminate: 15 (21%) of the 70 patients with an RCP3Q score of 1 had an ACQ score of ≥1.5 (poor control), 37 (54%) had an ACQ score of <0.75 (good control) and 18 (25%) had scores between 0.75 and 1.5.

Patterns of RCP3Q response

Table 5 lists each of the possible permutations of symptoms on the RCP3Q and shows the mean ACQ score of patients who reported that pattern. In patients reporting only one symptom (RCP3Q=1), those with symptoms interfering with sleep had a higher mean ACQ than those who only experienced daytime symptoms. Of the 14 patients who only reported interference with sleep, six (43%) had an ACQ score of ≥1.5 indicating poor control compared with nine of 56 (16%) of those whose only symptom was in the daytime or during activity. Conversely, applying the lower ACQ threshold of <0.75, 32 of 56 patients (57%) with daytime symptoms or activity limitation were well controlled compared with five of 14 patients (36%) with disturbed sleep.

Table 5 Patterns of response to individual RCP3Q questions: prevalence and association with ACQ score
Full size table

Additional records-based indicators of asthma severity

Both the number of reliever inhalers prescribed and the number of recorded asthma exacerbations in the previous year were associated with the ACQ score (Spearman's rho 0.30, p=0.001 and Spearman's rho 0.35, p<0.001, respectively) and with the RCP3Q score (Spearman's rho 0.28, p=0.003 and Spearman's rho 0.51, p<0.001). In view of this, we examined these record-based indicators of severity in order to see if they added predictive value. Given that an RCP3Q score of zero effectively ruled out poor control and a score of ≥2 made it highly likely, we limited this additional analysis to patients with an RCP3Q score of 1. These patients were categorised as meeting the record-based criterion for poor control if, in the last year, they had either been prescribed ≥4 reliever inhalers or had ≥1 recorded exacerbations. The ACQ score was ≥1 in 11/25 patients (44%) who met the record-based criterion and in 15/45 (27%) of those who did not, suggesting that it added relatively little additional value.

Variation between practices

Figure 3 shows the correlation between RCP3Q and ACQ scores among patients in each practice. The appearance suggests that, in two practices (J and L), almost all patients are rated as RCP3Q=1 despite reporting a range of ACQ scores, in some cases indicating poorly controlled asthma.

Figure 3
figure 3

Scatter plots of the Royal College of Physicians three questions (RCP3Q) score against the Asthma Control Questionnaire (ACQ) score for each practice

Full size image

Discussion

Main findings

When used and recorded in routine practice, the RCP3Q delivered results that were broadly correlated with the ACQ although there was considerable variation between practices. In particular, an RCP3Q of zero indicated good control as assessed by the ACQ and an RCP3Q score of 2 or 3 indicated poor control. An RCP3Q score of 1, which was the most commonly reported score, was less helpful: only 34% had poor control (defined by ACQ ≥1), although those with daytime symptoms only were unlikely to have unequivocally poor control (ACQ ≥1.5). Recorded exacerbations and use of reliever inhalers added little to the predictive value of the RCP3Q when compared with the ACQ, which suggests that these parameters reflect different aspects of control. The range of suitable codes and substantial variability in the coding strategies between practices made data extraction very difficult.

Strengths and limitations of this study

The participating practices may not have been representative of the full range of clinical practice, although the QOF pilot20 recruited a representative sample of UK practices and additional practices were recruited within these demographically diverse areas.

The difficulty in matching questionnaire responses to data extracted from the electronic health record led to less than half of returned questionnaires being suitable for analysis, which reduced the statistical power of the study. As a result, confidence intervals were wider than anticipated and it was not possible to quantify variation between practices using linear mixed effects modelling. It may also have introduced bias, although patients' demography and ACQ scores were similar in both matched and unmatched cases.

In the absence of a standardised coding for the three questions, the practices had developed their own protocols for recording the RCP3Q. This may mean that we missed or misinterpreted some coded responses, although we asked practices how they coded the data and customised our automated search accordingly (see Appendix 1, available online at www.thepcrj.org).

Practices invited patients to complete the ACQ after an asthma review in order to ensure that administration did not influence the patients' assessment of their control and how they answered the RCP3Q during the consultation. We specifically asked practice staff not to change their normal procedure for asking and recording the RCP3Q in order to reflect real-life practice, although awareness that they were participating in a study may have influenced their actions. In keeping with the concept of a phase IV study21 which aims to influence the ‘usual’ process as little as possible, we collected anonymous routinely recorded data from the practices. This meant that we are unsure how many ACQs were actually handed out, and cannot account for ACQs that we could not pair with RCP3Qs in the patients' electronic health record. Had we dictated procedures more tightly, we would probably have increased the rate of pairing but would not have been able to observe real-life practice.

Interpretation of findings in relation to previously published work

Our data illustrate the importance of real-life validation of questionnaires. Compared with the study by Thomas et al. in which the RCP3Q was administered under research conditions,19 we observed similar sensitivity (RCP3Q extracted from the routine records detected 96% of patients with ACQ ≥1 compared with 94%) but specificity was halved (34% compared with 67%). This may reflect the explicit 1-week timescale applied in the research context compared with a relatively open-ended approach used to assess control in real life which would be more likely to capture a recent episode of poor control.

The observed heterogeneity between practices probably reflects procedural variations. Although some clinicians may administer the questions precisely as described in the literature, others may paraphrase the questions or introduce them informally into the conversation and record the answers at the end of the consultation. Some practices may adopt practical strategies such as sending out questionnaires some weeks in advance with an invitation to attend for a review (potentially misleading in a variable condition such as asthma). Clinicians will find pragmatic ways to administer questionnaires to people (often excluded in research studies) with poor literacy, reduced vision, or who ‘cannot answer a questionnaire in English’. The variation is further compounded when choice exists in coding systems. Similar variation in other disease contexts (e.g. the nine-item Patient Health Questionnaire which assesses severity of depression22) raised concerns that the questionnaire performed inconsistently in real-life practice.23

Qualitative research after the PHQ9 was included in QOF in 2006 suggested that, despite general practitioners' caution about using formal questionnaires, patients' confidence in their general practitioners' assessment and management of their mental health problems increased.24 In contrast, the RCP3Q was generally well accepted by professionals in the pilot work,6 reflecting the emphasis — recently prioritised in the International Primary Care Respiratory Group's research needs prioritisation exercise — on simple questionnaires (or ‘questions’) to assist with clinical assessment in the typically low technological environment of primary care.25

Previous studies of the validity of the RCP3Q in the context of research concluded that ‘any positive responses may indicate suboptimal control’ whereas our findings suggest that the presence of daytime symptoms was compatible with good control as judged by an ACQ score of <0.75. This resonates with definitions of control in international and national guidelines which distinguish between occasional daytime symptoms (e.g. on ≤2 days a week) which may be ‘acceptable’ in a well-controlled patient whilst any nocturnal waking should be considered as less than well controlled and management adjusted accordingly.26,27 Modelling of similar morbidity questions in the context of a large-scale audit of asthma management concluded that the RCP3Q provided an effective assessment of control but that a version which incorporated frequency of symptoms was more sensitive to poor control.15

Including exacerbations and use of reliever inhalers as indicators of poor control added little to the predictive power of the RCP3Q in relation to the ACQ. This may be because both the questionnaires measure morbidity at one point in time whereas exacerbation and use of short-acting β2-agonists could reflect previous episodes of poor control which may have resolved at the time of a review. Exacerbations and prescribing history are normally readily available to primary care clinicians, and our data reinforce recommendations that all aspects of present control and future risk should be considered when assessing asthma control.1,3

Implications for future research, policy and practice

The clinical implications of our findings are summarised in Table 6. The RCP3Q scores have clinical utility although they should be interpreted in the light of the broader clinical context — especially scores of 1. Many developers of questionnaires express the hope that their instrument will be of value in the clinical context,9,28 and the potential of alternative instruments for measuring asthma control in routine asthma reviews has been summarised by the International Primary Care Respiratory Group.29 However, clinical utility is not the same as validation in a research context, and the significant variation we observed between practices emphasises the need for real-life phase IV assessment before tools are introduced into routine clinical practice.30 Provision of a single suitable coding system could remove one source of variability.

Table 6 Interpretation of the Royal College of Physicians three questions (RCP3Q) scores in clinical practice
Full size table

Conclusions

Our data support the promotion of the RCP3Q as a practical tool for detecting suboptimal current control when used in routine asthma reviews and offers advice on how scores obtained in real-life practice should be interpreted. Taken in conjunction with exacerbations and use of reliever inhalers, routine adoption of these simple morbidity questions could contribute to addressing the recognised challenge of improving asthma control.