Introduction

COPD is a major disease burden in terms of morbidity, disability, mortality, and health-care cost1,2,3,4. Although COPD is a progressive and life-threatening disease, it is manageable. With appropriate management, most people with COPD can achieve good symptom control and quality of life, as well as reduced risk of other related conditions.

High-quality care for patients with COPD is important, mainly due to serious consequences and the growing number of affected people5. Multiple international COPD guidelines for primary care exist and are routinely updated6. However, general practitioners do not always adjust their practice according to the latest recommendations to ensure the most appropriate care for patients with COPD7,8,9,10. To measure these limitations and to improve the quality of COPD primary care, quality indicators (QIs) are needed11. Quality assessment focuses on health-care outcomes, structures and processes, and coded data in the electronic health record (EHR) can be used to evaluate the quality of care12,13,14,15,16. Moreover, EHR-extractable QIs drastically increase the number of patients whose quality of care can be assessed12. As a result, feedback to health-care professionals can make them reflect on their own practice and assist them in improving the quality of care17.

Although QIs for COPD care are available in the literature, few cover all aspects of COPD care18,19,20. In addition, they are not specifically intended for primary care or can be extracted out of the EHR. The aim of this study was therefore to develop an up-to-date set of evidence-based and EHR extractable QIs that cover all aspects of COPD primary care, thus enabling the monitoring and improvement of the quality of COPD primary care.

Methods

We conducted our study from December 2019 to April 2021.

Study design

To develop QIs for COPD in primary care, we used a RAND-modified Delphi method11,13,21,22. The following steps were included: (1) Selection of recommendations and existing QIs from evidence-based guidelines; (2) rating procedure by an expert panel in three rounds: (a) individual written questionnaire, (b) online consensus meeting, and (c) final appraisal; and (3) translation of the recommendations into QIs.

Ethical approval

The study was approved on December 9, 2019, by the Research Ethics Committee UZ/KU Leuven with reference MP012007.

Panel selection

Multiple health-care providers and patients with COPD were contacted via e-mail or phone call: pulmonologists of three hospitals via e-mail invitations, general practitioners via local general practitioner circles and patients with COPD via a non-profit patient association. The selection of possible participants was based on their expertize with COPD. All patients needed to have the diagnosis of COPD for >10 years and all professionals needed to treat patients with COPD on a daily basis to ensure this inclusion criterion was met. In total, 12 participants were included to construct a multidisciplinary panel based on profession, patient involvement, and expertize. They received information about the aim of the study and the methodology in an informed consent letter. We asked them to sign the consent form before participation.

Selection of recommendations and QIs from evidence-based guidelines

First, we searched for national and international guidelines on the World Wide Web and by reference tracking20,23,24,25,26,27,28. These guidelines had to be evidence-based and developed by internationally renowned institutions. Second, a literature review using the electronic database MEDLINE was performed until January 4, 2020, for additional guidelines or recommendation statements (see Fig. 1). The search strategy combined MeSH terms and free text: “chronic obstructive pulmonary disease” OR “chronic bronchitis” OR “pulmonary emphysema” OR “COPD” AND “guideline” OR “standard of care” OR “clinical protocols” OR “health planning guidelines”. One investigator (S.D.) screened all titles and abstracts to exclude irrelevant references. References were retained if they were clinical guidelines or recommendation statements aimed at clinicians in primary care in which all aspects of the care for COPD, including definition, screening, diagnosis, treatment, management, comorbidities, referral, follow-up, pulmonary rehabilitation or palliative and end-of-life care, was recommended. Then, full texts of all potentially relevant guidelines or recommendation statements were read and reviewed by two investigators (S.D. and L.G.) for their eligibility. In case of disagreement, a third author (S. VdB.) was consulted.

Third, a list of existing sets of QIs for COPD developed by international health organizations was offered by the Belgian Centre for Evidence-Based Medicine (www.cebam.be).

The entire selection process was based on language (English and Dutch) and year of publication (after 2012). Furthermore, an instrument developed by the Appraisal of Guidelines for Research and Evaluation (AGREE) Collaboration was used independently by two investigators (S.D. and L.G.) to ensure that only guidelines of high quality were implemented29. Guidelines with scores of <50% were excluded.

One investigator (S.D.) exhaustively extracted all recommendations from the included guidelines and listed them in (sub)categories regarding their domain (see Supplementary Table 2). Thereafter, QIs for COPD from the Cebam list were added if they met the aforementioned eligibility criteria and if they were not already covered. Then, the EHR extractability and the applicability to primary care of the recommendations and QIs were judged based on the SMART principle30. Finally, the selected recommendations and QIs were included in a questionnaire that was provided to the panel (see Supplementary Table 3). An alternative copy of the questionnaire was provided to the patients, excluding the recommendations specifically aimed at health-care providers.

Rating procedure

The questionnaire was sent by e-mail to the panel members accompanied by a clear explanation of the objectives of the study and specific instructions to fill in the questionnaire (round a). They were asked to score each recommendation and QI by giving a number on a 9-point Likert scale, with 1 being the lowest score and 9 the highest score for capability to measure the quality of COPD primary care. If a panel member was unable to assess the recommendation or QI, then it was possible to designate “not assessable”. The EHR extractability could also be denoted. Per category, the panel members were asked to prioritize the recommendations and QIs with a maximum top-5 rank. In addition, panel members were given the opportunity to motivate their score, to give remarks or to suggest new recommendations.

All scores and remarks were entered into an Excel database and analyzed. The recommendations and QIs were assigned as having a high, uncertain or low potential to measure the quality of COPD primary care. These three criteria were based on the median Likert scale score, prioritization and degree of agreement among the panel members. First, the median Likert scale score was the calculation of the median score for each recommendation and QI. Hereby, non-assessable scores were omitted from the calculation. Second, prioritization meant that, for example, in a top 3, the first one in the ranking received 3 points, the second received 2 points and the last received 1 point. These points were then converted into percentages using the number of panel members that scored that recommendation or QI and the highest possible score. Third, agreement was defined as ≥70% of the panel members giving a median Likert score of ≥7, and disagreement was defined as ≥30% of the panel members giving a median Likert score ≥7 AND ≥ 30% giving ≤3. Finally, we refer to Table 1 for the categorization into high, uncertain or low potential to measure the quality of COPD primary care.

Table 1 Classifying recommendations and QIs into categories of high, uncertain and low potential.
Full size table

The online consensus meeting was held via video conferencing (round b). Prior to the meeting, the panel members were informed of the results from the questionnaire in a feedback report. The report contained for each recommendation and QI their individual score, the median score, percentage of prioritization and agreement. Additionally, the assigned criterion of high, uncertain or low potential for quality of COPD care measurement was indicated. At the meeting, panel members first reviewed the recommendations and QIs with a low or high potential for exclusion or inclusion, respectively. They were encouraged to openly discuss their views of each recommendation and QI to reach consensus. Panel members were also allowed to modify the included recommendations and QIs and create new recommendations, keeping in mind the SMART principle to evaluate EHR extractability and appropriateness in primary care. Finally, the recommendations and QIs marked as uncertain potential were discussed until agreement was reached to exclude or include, possibly with modification. If reaching consensus was elusive, then a formal vote was held, and recommendations and QIs with >50% panel member support were included.

The list of included recommendations and QIs obtained from the consensus meeting was sent by e-mail to the panel members for their final appraisal (round c).

Translation of recommendations into QIs

In the last step, the recommendations were translated into QIs as a percentage. For example, the recommendation “COPD patients should be offered annual vaccination with the influenza vaccine” was translated into “The percentage of patients with COPD who are vaccinated with an influenza vaccine annually”.

Results

Panel selection

A multidisciplinary expert panel (n = 12) consisted of four pulmonologists, three general practitioners, one physiotherapist, one nurse and three COPD patients. The panel members mainly originated from two regions in Flanders (province of West Flanders and province of Flemish Brabant). The health-care providers worked either in hospitals or in general practices. We did not account for factors such as age, gender, education level, or disease severity.

Selection of recommendations and QIs from evidence-based guidelines

The selection process for recommendations and QIs from evidence-based guidelines is visualized in Fig. 1. First, seven guidelines were selected. Second, 389 references were identified through the database search. In total, 382 irrelevant references were excluded based on title and abstract. Subsequently, seven full texts were retrieved and assessed for eligibility. Of these, four references were excluded: three did not meet the quality criteria, and one was a duplicate. Together, ten guidelines fulfilled the eligibility criteria20,23,24,25,26,27,28,31,32,33. Third, six sets of existing QIs (n = 93) were added.

Fig. 1: Flowchart of selection of recommendations and QIs from evidence-based guidelines.
figure 1

*Titles, abstracts, and quality indicators excluded for one of the following reasons: published before 2012, no English or Dutch, no full text access, incorrect subject (e.g., lung cancer screening), incorrect target users (e.g., pulmonologists), no guidelines or duplicates. **Full texts excluded because three did not sufficiently meet quality criteria and one was a duplicate.

Full size image

Initially, 404 recommendations were extracted from the included guidelines. They were classified into ten main categories (definition, screening, diagnosis, treatment, management, comorbidities, referral, follow-up, pulmonary rehabilitation and end-of-life care). After applying the SMART principle and merging similar recommendations, 37 recommendations were used in the questionnaire. In addition, five QIs from the sets were added to the questionnaire.

Rating procedure

The rating procedure included three rounds. All panelists (12/12) completed the individual written questionnaire (round a). The questionnaire for health-care providers consisted of 37 recommendations and five QIs. The COPD patients were asked to fill in the same set without the pharmacological part (18 recommendations). Based on the median score, prioritization and degree of agreement, 27 recommendations were assigned as high, ten as low and five as uncertain potential to measure the quality of COPD primary care (see Supplementary Table 1).

Thereafter, eight panel members (three pulmonologists, three general practitioners, one nurse, and one physiotherapist) participated in the online consensus meeting (round b). One out of ten recommendations and QIs with low potential was included. Then, 23 out of 27 high-potential recommendations and QIs were retained. No recommendation or QI of uncertain potential was accepted. In total, 20 recommendations and four QIs were included, and 17 recommendations and one QI were excluded (see Supplementary Table 1).

Although the SMART principle was used, the expert panel decided to not exclude two recommendations and one preexisting QI that are currently not extractable from the EHR. Because of their importance for the quality of COPD primary care, some panel members proposed having these recommendations and QI made codable by computer experts in the nearest future.

The expert panel emphasized a maximum duration of seven days for both oral corticosteroids and antibiotics to treat an acute exacerbation. Therefore, at the consensus meeting, the panel’s pulmonologists introduced one new recommendation expressing the maximum duration of antibiotic treatment in case of an acute exacerbation. Finally, this resulted in a list of 21 recommendations and four QIs.

After the consensus meeting, the core set of 21 recommendations and four QIs was sent by e-mail to the expert panel for final appraisal (round c). The four panel members who were unable to attend the consensus meeting agreed without any adjustments. All panel experts approved the final set.

Translation of recommendations into QIs

The recommendations were finally translated into 19 QIs, and two preexisting QIs were added, resulting in a final set of 21 QIs. The final set (Table 2) of QIs was made up of two QIs on definition, two QIs on screening, two QIs on diagnosis, three QIs on prevention and nonpharmacological treatment, nine QIs on pharmacological treatment, of which one was added by the expert panel, two QIs on referral and one QI on end-of-life care.

Table 2 Final set of 21 quality indicators (QIs) for COPD.
Full size table

The two recommendations and one preexisting QI that are currently not extractable from the EHR could not yet be translated into measurable QIs and were listed separately from the final set (Table 3).

Table 3 Recommendations and QI on COPD care, belonging to the core set of recommendations and QIs as defined by the expert panel, but not yet translatable into measurable QIs.
Full size table

Discussion

This study, using a RAND-modified Delphi method, developed a set of 21 evidence-based and EHR extractable QIs to assess the quality of COPD primary care. The set contains different aspects: definition, screening, diagnosis, prevention, nonpharmacological treatment, pharmacological treatment, referral and end-of-life care. In addition, these QIs can be used in an automated audit and feedback intervention as a framework to facilitate and improve the quality assessment of COPD primary care. Furthermore, this set of QIs is based on ten international guidelines and approved by a multidisciplinary expert panel including patients.

In the current study, a number of indications for referral were drawn up by the pulmonologists. Moreover, a prominent role in the follow-up and end-of-life care of COPD patients was assigned to the general practitioner. To date, these two categories have received too little interest. Other studies have already mentioned that a large proportion of patients with COPD receive inappropriate end-of-life care. Timely initiation of care planning could lead to a reduction in avoidable inappropriate end-of-life care34,35. No recommendations on pulmonary rehabilitation were selected. However, this does not mean that pulmonary rehabilitation is not important, but our expert panel stated that this therapy is mostly initiated by a pulmonologist. In Belgium, pulmonary rehabilitation services are usually connected to hospitals involving a team of different medical and paramedical disciplines. Therefore, these recommendations were not specific for primary care in Belgium. In addition, to date, the initiation of pulmonary rehabilitation is not extractable out of the EHR in Belgium and thus these recommendations were also not SMART applicable.

This study differs from the available literature in two main respects. First, in previous literature, for example the ACOVE-3 study or the HEDIS program, QIs were specifically aimed to specialists or only applied to elders19,36. Additionally, COPD QIs developed by large health organizations, such as the National Institute for Health and Care Excellence (NICE), did not cover the broad spectrum of COPD primary care20,23. Also, the focus of the existing QIs was mainly on diagnosis and management. Therefore, we designed a set of QIs exclusively for general practitioners, including all aspects of the medical process. Second, in our study, much attention was given to the EHR extractability of the QIs, which was not stated in another study18. One of our panel members was an expert in the use of the EHR as a member of the Intego network (www.intego.be). However, two recommendations and one preexisting QI were retained because of their great importance but are currently not extractable from Belgian EHR systems. More specifically, the registration of smoking cessation advice and functioning scores (CAT and mMRC) were estimated to be very important. This shortcoming in EHR extractability should be resolved in the near future so that these recommendations and QI should not be regarded as inappropriate.

The current study can serve as a first step to assess and improve the quality of COPD primary care. Automated extraction from the EHR will enhance the usability of these QIs. As this depends on coded data, challenges facing EHR systems are to transform uncoded data into structured data for subsequent analyses. Therefore, new methodologies for full integration of data and unification in EHR systems are required to optimize data management and extraction16. The continuous development of EHRs holds promise that quality assessment will grow and lead to improvements in COPD health-care. Additionally, to assess the implementation of these QIs, they must be validated by operational testing. A clinical practice test in multiple EHR systems is the best way to judge the validity of our QIs before integrating them in an automated quality measurement. This set is currently being tested in daily practice in Belgium with regard to validity and reliability.

One of the strengths of this study is the use of a multidisciplinary expert panel representing different disciplines involved in COPD care from diverse settings and regions within Flanders. In addition, our study paid attention to the patient’s point of view by including patients in the expert panel, according to other studies describing the development of QIs11,13,21. This panel diversity allowed us to take into account all aspects of COPD care. Furthermore, we used a robust methodology (RAND-modified Delphi method) to develop the QIs for COPD37 and applied the SMART principle, which has proven its usability for the development of QIs13,21,38. Because of the broad international base, this study has high validity and generalizability. Therefore, the final set of QIs can be used to evaluate quality assessment across countries, but the EHR extractability needs to be confirmed when these QIs would be used in other countries11. Also, QIs may be associated with the health-care system in which they were developed. In Belgium, patients have free choice of health-care professionals and have direct access to them. Hence, for example, QIs for referral may differ between countries.

However, there are some limitations. Our panel did not include COPD educators or pharmacists, and it is known that another constitution of the panel could have led to a different set of QIs39. None of the patients were able to participate in the meeting, which took place online instead of face-to-face. Therefore, the decisions to include or exclude in this round may be biased in favor of those who attended the meeting. Nonetheless, this bias was minimized by giving the entire panel the opportunity to give their final appraisal by e-mail. A limitation of using the RAND-modified Delphi method is the loss of participant anonymity in the voting process during the consensus meeting39,40. However, a group consensus meeting allows panel experts to exchange important information, for example, clarification of recommendations or reasons for (dis)agreement. Additionally, seeking clarification helps to reach consensus or to generate an alternative recommendation.

In summary, this study developed a set of 21 evidence-based and EHR-extractable QIs for COPD in general practice using a RAND-modified Delphi method. These indicators cover different aspects of COPD primary care, including definition, screening, diagnosis, prevention, nonpharmacological treatment, pharmacological treatment, referral and end-of-life care. In addition, this set enables an automated audit and feedback intervention to assess and improve the quality of COPD primary care.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.