Introduction

Evidence-based medicine is defined as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.1 The aim is to provide quality care, minimise medical error, and to ensure equitable provision of services. In specialties other than ophthalmology, there is a body of literature evaluating the proportion of clinical interventions that are evidence based. For example, in general medicine, 53% of interventions were based on randomised controlled trials and 29% on observational evidence.2 The range across specialties is between 71 and 97%.3, 4, 5, 6, 7, 8

Emergency ophthalmology requires not only clinical skill but also knowledge of management options for best quality care and suitable outcome of the patient. Lai et al9 examined the quality of interventions in an emergency eye clinic in Hong Kong and reported that 77% of interventions were evidence based. The question still exists, however, as to whether quality of interventions can be improved by such evaluations. We aimed to answer this question by judging the quality of interventions made in a busy eye casualty facility in a mixed inner city area at the Ophthalmology Department of North Middlesex University Hospital, London in one period in 2003, to issue guidelines and recommendations for treatment based on the findings, and to re-evaluate the effects of these 1 year later.

Methods

The first phase of the audit was carried out retrospectively in March 2003. To minimise the risk of bias from Hawthorne's phenomenon (changing prescribing habits due to prior knowledge that outcomes are being monitored), the senior house officers staffing the eye casualty were kept unaware of the timing or nature of the study. The first 300 attendees in the eye casualty were identified and every third case file requested, to examine the evidence base for outcomes for 100 patients. An electronic database was used to record information on patient demographics, symptoms, diagnosis, visit details (number of visit, level of doctor examining the patient, and senior opinion sought by discussion or consultation), and management of the condition (details of treatment and medical or surgical nature). For assistance with resource planning, the final outcome of the patient (discharge, casualty review, and referred to outpatient or other specialty) was also recorded. The evidence level for each intervention was checked in a similar method to Lai et al,9 using evidence levels for different types of study as follows:

Level 1 =Systematic review Level 2 =Meta-analysis Level 3 =Randomised controlled trial (RCT) Level 4 =Prospective study Level 5 =Retrospective study

Each intervention was searched for in the Cochrane Database of Systematic Reviews and in Medline from 1966 to March 2003. The highest level of evidence was recorded for each outcome and checked by the senior author (JR). Data were then analysed by each evidence level.

Departmental guidelines based on evidence in the literature were written specifically taking into account local circumstances, and an agreement was reached by all consultant ophthalmologists in the unit. These were issued to the incoming senior house officers and throughout the department in February 2004. For any conditions not covered in the guidelines, the recommendation was either to consult the Wills Eye Manual10 or to seek the opinion of a more senior member of staff. One calendar year after the initial phase, the audit was repeated using exactly the same methodology, in the first week of March 2004, 1 month after guidelines were introduced. This timing was decided because the incoming senior house officers would have a similar level of experience to those in the 2003 study, and so this would give a better idea of whether the guidelines were making a difference to the quality of interventions. Exactly, the same methodology as the 2003 phase of the study was used: case notes for every third patient attending the emergency clinic from 1 March 2004 for 300 cases were requested and analysed in the standardised way. Fisher's exact test was used for the statistical analysis, to evaluate the difference between proportions of interventions between the independent data sets, where some of the observed frequencies are small.

Results

The results of the initial audit are shown in Tables 1, 2, 3, 4, 5 6 and 7. In 2003, 88 case notes were obtained; 3 of these had been misrecorded and had not actually attended the eye casualty and so were excluded, and a further 5 had no diagnosis made in the ophthalmology emergency visit, so these were also excluded from the analysis of interventions. Therefore, there were 80 diagnosis–intervention pairs who were evaluated in the Cochrane Database of Systematic Reviews and Medline. The mean age of the patients was 38.8 years (median 38.5, range: 1 week–80 years), and 55% were men. Most consultations (99%) were with a senior house officer, and 10% of consultations were discussed with or seen by a more senior ophthalmologist. Table 1 shows the number and proportion of interventions for each evidence level in 2003 and those found by Lai et al9 in Hong Kong. The percentages for each evidence level correspond closely, and Fisher's exact test for the differences between proportions did not show a significant difference between our data and those of Lai et al.9 Tables 2, 3, 4, 5, 6 and 7 show a breakdown by level of evidence the number of patients for each diagnosis and intervention, and a citation from the literature search that supports that intervention. Medical interventions comprised 91% and surgical interventions 9%. The outcome of the patients was 35% discharged, 38% to be reviewed in casualty again, 25% to be seen in ophthalmic outpatient clinics, and 3% referred to another hospital specialty.

Table 1 Evidence-based medicine audit: summary of interventions by evidence level in 2003 and 2004
Full size table
Table 2 Evidence-based medicine audit: interventions based on evidence level l (systematic reviews) in 2003 and 2004
Full size table
Table 3 Evidence-based medicine audit: interventions based on evidence level 2 (meta-analysis) in 2003 and 2004
Full size table
Table 4 Evidence-based medicine audit: interventions based on evidence level 3 (randomised controlled trials) in 2003 and 2004
Full size table
Table 5 Evidence-based medicine audit: interventions based on evidence level 4 (prospective studies) in 2003 and 2004
Full size table
Table 6 Evidence-based medicine audit: interventions based on evidence level 5 (retrospective studies) in 2003 and 2004
Full size table
Table 7 Evidence-based medicine audit: interventions based on no evidence or against evidence in 2003 and 2004
Full size table

Guidelines agreed within the department were issued as detailed above. These were written with the knowledge of the results of the 2003 study, but included other common conditions. The conditions for which guidelines were written included ocular trauma, anterior and posterior segment pathology, glaucoma, oculoplastic, orbital, inflammatory, and neuro-ophthalmic diseases. The timing of the repeat study was not disclosed to prevent bias from Hawthorne's phenomenon.

In the 2004 audit assessing the impact of issued guidelines, 90 case notes were obtained from the 100 requested; 7 were miscoded and had not attended the ophthalmology department and 1 had no diagnosis made in the eye casualty. Therefore, 82 pairs of diagnosis–intervention were available for analysis and comparison to our previous study of 2003 to evaluate whether the quality of emergency ophthalmology could be raised. The mean age of patients seen was 47.6 years (median 48 years, range: 2–86 years), 54% were men. All patients were seen by a senior house officer, and 13% of cases were also discussed with or seen by a senior colleague, similar to the numbers for 2003.

Table 1 also shows the number of interventions in each evidence group for 2004. Overall, 82% of interventions in 2004 had some evidence, and 60% reached levels 1–3. Tables 2, 3, 4, 5, 6 and 7 also show a breakdown by level of evidence the number of patients for each diagnosis and intervention for 2004, and a citation from the literature search that supports that intervention. Medical interventions comprised 82% and surgical interventions 18%. The outcome of the patients was as follows: discharged in 55% (in 2003, 35%), review in the emergency clinic again in 22% (in 2003, 38%), ophthalmology outpatient clinic referral in 20% (in 2003, 25%), and referred to another specialty in 4% (in 2003, 3%).

Comparing the proportion of interventions by different evidence levels before and after departmental use of guidelines gave some interesting results. In 2003, 71% of interventions had some evidence compared to 82% in 2004 (P=0.04, Fisher's exact test), and 60% reached levels 1–3 compared with 45% previously (P=0.02). There was also a shift in the quality of interventions by evidence level. There were 8% with evidence level 1 in 2003 and 18% in 2004 (P=0.02, Fisher's exact test). Level 5 evidence was attributed to 15% in 2003 but was reduced to 4% in 2004 (P=0.009, Fisher's exact test), suggesting the improvement in quality. There was no significant difference for interventions reaching evidence levels 2, 3, or 4. Although the proportion of interventions against evidence stayed steady between our two studies, there was a significant reduction in those interventions with no evidence. When the proportion with no evidence or against evidence are grouped, as in the methodology of Lai et al,9 there is a significant difference between our results in 2003 (29%) compared to 2004 (18%).

Discussion

Evidence-based medicine is a concept that has been embraced over the last decade. We present the results of our audit in an effort to examine whether the use of evidence-based guidelines could improve the quality of care given to patients in an acute ophthalmic setting.

In the initial reports from general medicine, it was thought that the proportion of interventions based on evidence was low. Subsequent studies of interventions in a range of specialties showed that most medicine was in fact evidence based and included general medicine,2, 3 surgery,5, 8 anaesthesia,7 and dermatology.6 In a landmark report in our specialty, Lai et al9 showed that ophthalmic interventions reached a similar level. None of these reports, however, have attempted to use the knowledge gained from auditing their interventions to devise ways of improving outcomes for patients. We therefore examined our outcomes using the same methodology as the previously published report9 and used this information to improve the quality of care.

Lai et al9 collected data from 252 interventions in July 2002 for seven sessions in the eye casualty at their tertiary referral centre in Hong Kong and found that 77% were based on evidence, and 43% of their interventions were grade 3 or better. They concluded that interventions in ophthalmic emergency care were comparable to other specialties.9 The results of our initial-phase study in a primary eye care facility in London were comparable to all their measures in a tertiary referral centre in Hong Kong (Tables 1 and 2). Overall, 71% of our interventions were evidence based compared to 77% of theirs, and 45% of our interventions were attributed to levels 1–3 vs 43% of theirs. Interventions without or against evidence were also similar in this comparison.

We tested whether it would be possible to use guidelines to improve outcomes of clinical decisions beyond the published ‘gold standard’ for ophthalmology, by either increasing the total number of interventions that were evidence based, reducing the numbers that had no evidence or were against evidence, or increasing the quality of interventions. This report shows that evidence-based evaluation can be used to increase most of these facets. After introducing guidelines, the total number of evidence-based interventions increased significantly from 71 to 82% and their quality increased so that there were less with level 5 and more with level 1 interventions. Even measures of the total interventions with no evidence or against evidence showed a reduction. The only parameter that did not alter significantly was the proportion of interventions against evidence. This may imply that there are a certain number of difficult cases to which generalised studies cannot be applied or indeed that the evidence base from older studies may be out of date with current clinical practice.

The limitations of our study include the possibility of bias. Prescribing bias from prior knowledge of the study by the doctors involved (Hawthorne's phenomenon) was avoided by keeping the dates of both studies unknown to the participants. Bias from improvements in the clinical knowledge of the doctors involved was avoided by careful timing of the study when a new set of doctors would have similar clinical experience to the first set. There exists the possibility that after a year has elapsed, there is more evidence in the literature as a further year's worth of studies have been published. We avoided this potential bias by assessing the 2003 and 2004 interventions in Medline and Cochrane databases from 1966 to 2003 for both years.

However, bias may occur from the deliberately reductionist approach that this methodology takes, in that only the primary diagnosis and primary intervention were examined. Additionally, only two databases were searched, and other sources such as textbooks or case reports were ignored. The reason for this omission was for consistency with previously published studies to compare like for like. Similarly, there is some variation in the literature on which type of study should be assigned to a particular evidence level.2, 8 Our aim was to audit our outcomes against the ‘gold standard’ published by Lai et al,9 so we chose to replicate their method of attributing evidence levels to the various types of studies.

It may also be the case that our study is applicable only to a particular set of circumstances, either the location or staffing of our service. It is encouraging that a primary care facility has the similar outcomes to those of a tertiary referral centre, and that this can be improved upon by a simple method. Perhaps this technique is only useful for relatively junior medical staff, and it would be interesting to see the evidence level of interventions by more senior staff. It is quite possible that with experienced clinicians, the use of guidelines becomes obsolete. In one study examining the evidence base for interventions in eye casualty in Wolverhampton (UK), 90% of interventions were found to have been led by scientific evidence.60 In that study, 66% of interventions were carried out by nurse practitioners using departmental guidelines. This supports our view that locally agreed guidelines and protocols should be used for good quality ophthalmic care. However, it must be recognised that a proportion of clinical practice will exist for which there is little published or high-quality evidence. This includes interventions for rare conditions or surgical procedures, for both of which randomised controlled trials may be difficult to achieve.

The case mix between the study of Lai et al,9 and the two phases of our study is obviously different and is a potential confounding factor that is difficult to control for. This is related to the random nature of cases presented to an emergency service, and we recognise the limitation of this on the audit findings. This might be overcome by a larger clinical audit over a longer period, in an extension of the present study. Nevertheless, issuing guidelines on a wide range of conditions has had an impact on quality, in that there was a positive effect rather than no effect or a negative effect, and also has implications on patient outcomes with fewer return visits. Resource planning is an important issue in a nationally funded health service. A secondary aim of this study was to improve patient outcome and reduce the numbers of unnecessary casualty or outpatient reviews. Our use of guidelines was successful in achieving this with a discharge rate changing from 35 to 55% and a reduction in the number of review visits to the eye emergencies clinic. It is possible that this method gives relatively inexperienced medical staff the confidence to take such decisions, reducing the uncertainty that causes unnecessary re-attendance.

In summary, audit of evidence-based ophthalmology outcomes can be used to measure performance and to improve the quality of care by the use of locally agreed guidelines and has the added benefit of reducing the number of re-attendances required. It will be interesting to see whether other specialties in medicine can also refine their outcomes.