Audit and feedback had small but potentially important improvements in professional practice

Abstract

Data sources

Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Effective Practice and Organisation of Care (EPOC) Group Specialised Register, Medline, Ovid; Embase, CINAHL, EBSCO, Science Citation Index and Social Sciences Citation Index, ISIWeb of Science

Study selection

Randomised trials of audit and feedback that reported objectively measured health professional practice or patient outcomes.

Data extraction

Data were abstracted by two independent review authors using a data extraction form. The following factors were examined as possible explanations for the variation in the effectiveness of interventions across comparisons; format of feedback, source of feedback, frequency of feedback, instructions for improvement, direction of change required, baseline performance, profession of recipient and risk of bias within the trial itself. Risk of bias was assessed using EPOC criteria. The degree of confidence in the estimate of effect across studies was assessed using the GRADE approach. All outcomes were expressed as compliance with desired practice and both professional and patient outcomes were assessed, though separately.

Results

One hundred and forty studies were included. Using 70 of these 108 comparisons were made. After excluding studies at high risk of bias there were 82 comparisons from 49 studies with dichotomous outcomes, and the weighted median adjusted RD was 4.3% (interquartile range (IQR) 0.5% to 16%). For studies with continuous outcomes the weighted median adjusted percentage change relative to control was 1.3% (IQR = 1.3%to 28.9%). For patient outcomes the weighted median RD was 0.4% (IQR −1.3% to 1.6%) in studies reporting dichotomous outcomes, and the weighted median percentage change was 17% (IQR 1.5% to 17%) for studies with continuous outcomes. Multivariable meta-regression indicated that feedback may be more effective when baseline performance is low, the source is a supervisor or colleague, it is provided more than once, it is delivered in both verbal and written formats and when it includes both explicit targets and an action plan.

Conclusions

Audit and feedback generally lead to small but potentially important improvements in professional practice.

Commentary

The Cochrane Effective Practice and Organisation of Care (EPOC) Group focuses on interventions to improve the delivery, practice and organisation of healthcare services¹. A proportion of their reviews are concerned with how to encourage professional behaviour that is in accordance with best available evidence or a professional standard of practice. As well as this review on the effect of audit and feedback, there are also reviews on the effectiveness of educational meetings², educational outreach visits³ and local opinion leaders⁴ in changing professional behaviour.

The purpose of audit and feedback is to measure a clinician's performance, compare it to a professional standard and then to feed the results back to the clinician with the intention of improving practice. The feedback may be given verbally, in writing, or both. In the 140 studies included in this review some of the participants were simply given their results whilst in others they were also set targets or an action plan to help them improve.

The vast majority of the included studies involved doctors, mainly in a practice setting, however, there were two that involved dentists; one that involved recording periodontal care⁵ and another compliance with a guideline for impacted molars⁶. Most trials measured professional practice, such as prescribing or the use of laboratory tests, but in 112 of the 140 studies it was unclear how the feedback was given. Thirty-one percent of trials were judged to have been at low risk of bias, 51% had an unclear risk of bias and 18% had a high risk of bias.

The authors use the risk difference (RD – an absolute outcome) for dichotomous outcomes and the weighted median adjusted percentage change (a relative outcome) for continuous outcomes. For studies with dichotomous outcomes the overall RD was a 4.3% increase in compliance with desired practice. But using meta-regression they looked at the expected effect of a number of variables on the outcome.

There were relatively large differences in effect size when comparing these characteristics: feedback presented in both verbal and written format versus only verbal (expected difference in adjusted RD= 8%); delivered by a supervisor or senior colleague versus the investigators (expected difference in adjusted RD= 11%); frequency of monthly versus once only (expected difference in adjusted RD = 7%); containing both an explicit, measurable target and a specific action plan versus neither (expected difference in adjusted RD= 5%).

So whilst the effect of audit and feedback may be small overall this review suggests that it can be improved by ensuring the person responsible for the audit and feedback is a supervisor or colleague, that it is provided more than once, it is given both verbally and in writing and it includes clear targets and an action plan.

How does audit compare to other ways of changing clinician performance? The EPOC reviews suggest that there could be a positive risk difference with educational meetings of 6%, with the use of local opinion leaders of 9% and with educational outreach visits, 5.6%. Audit is a component of clinical governance in the National Health Service here in the UK as it is seen as part of the drive to improve care quality⁷. This review suggests that its effect in such a process may be small but given the large number of patients treated even small changes can result in large improvements in patient care.