Introduction

Patients who are eligible to be registered as sight impaired (SI) or severely sight impaired (SSI) require a certificate of vision impairment to be completed by a consultant ophthalmologist, in the UK. This is undertaken with reference to the UK guidelines on certification [1].

Patients with a completed certificate of vision impairment can then choose to be registered with the local government social services department, which then allows access to services and support; although, such support can also be accessed without certification. A greater level of support is available to those registered as SSI compared with SI. In England, the certificate of vision impairment is used to inform government metrics of public health improvement and protection [2], and in Wales, it is used to indicate incident certifiable sight impairment [3].

An additional role of the certificate of vision impairment is the collection of epidemiological information about the incidence and causes of certifiable sight loss in the UK [4, 5]. Whilst the number of certificates issued in the UK accurately reflects those registered with social services as having vision impairment [6], it does not represent all individuals living with sight loss. Indeed, it is estimated that up to 51% of those eligible for certification are not certified [7,8,9] and the incidence of certification varies across geographical locations [10]. From studies involving medical record review [7, 8, 11] and patient interviews [9], it was found that those with a treatable condition or receiving ongoing treatment were less likely to be certified than those with untreatable conditions. In addition, those from ethnic minorities were less likely to be certified than Caucasian patients [7, 9] and those with visual field loss alone were less likely to be certified than those with reduced visual acuity [8].

There is a mismatch in demand and capacity for available secondary care ophthalmology appointments, and the long waiting times for appointments may put individuals at risk of irreversible sight loss [4]. Given the care capacity issues, the role of the primary care optometrist has expanded, with the introduction and development of enhanced eye care services [12].

In Wales, over 8500 individuals with low vision are examined by the primary care-based Low Vision Service Wales (LVSW) each year. Registration with vision impairment is not a prerequisite for access to this service. The LVSW is provided by 184 practitioners (171 optometrists and 13 dispensing opticians) who have completed and continuously undergo specialist training.

In order to assess the appropriateness of an expanded role of LVSW accredited optometrists in the certification of vision impairment, there is a need to evaluate their clinical ability in identifying the eligibility of a range of individuals for certification. The aims of this study were twofold. First, to assess the agreement between optometrists and a consensus panel, in identifying patient eligibility for certification, relative to the agreement between ophthalmologists and the consensus panel. The second aim was to explore whether the agreement between clinician groups and the consensus panel was influenced by the presence of bilateral atrophic age-related macular degeneration (AMD) as the cause of vision impairment. This is important given the potential to influence policy in Wales in the certification of patients with bilateral atrophic AMD by optometrists, as the clinical management of this group is predominantly based in primary care.

Materials and methods

Case records from 40 individuals were selected at random (www.random.org), from 8000 patients seen by the LVSW between April 2017 and April 2018, stratified by the three categories of severity of sight loss and anonymised. The case records for each individual conformed to the following inclusion criteria: consent had been given to use the data for research and individuals were at least 18 years old.

Details from each case record were transferred to a proforma and consisted of: age, gender, time since diagnosis, occupation, social and living situation, general health, the presence of a hearing impairment, problems reported, support received to date, diagnosis (right and left eye), refraction (distance and near), visual acuity (distance and near, monocular and binocular), binocular contrast sensitivity (measured using the Pelli–Robson chart, reported in terms of percentage loss and whether the loss was considered as: normal, noticeable loss, significant loss, severe loss [13]) and the visual field status. Visual field printouts were included where available.

The consensus panel consisted of four consultant ophthalmologists and three LVSW optometrists. Each consultant had been registered with the General Medical Council specialist register for ophthalmology for at least 2 years and had undergone the standard 7 years of speciality training prior to this. Each of the LVSW optometrists had been registered with the General Optical Council in addition to specialising in low vision for at least 15 years and had each completed a Masters-level qualification in low vision. The consensus panel met to determine the certification eligibility of each of the anonymised case records based on the information presented. In cases of disagreement on the certification outcome, the case was discussed until unanimous agreement was reached.

An anonymous online survey was then created (https://www.onlinesurveys.ac.uk), in order to present each of the case records in a random order for each new respondent. All National Health Service consultant ophthalmologists practicing in Wales (n = 58) and all LVSW optometrists (n = 162) were invited to take part as raters in the online survey to evaluate the 40 case records. Consultant ophthalmologists and LVSW optometrists who were in either the consensus panel and/or the research team were excluded from the survey. The survey was completed without time or other restrictions in an unsupervised environment.

Raters were asked to decide on the certification eligibility status (not eligible, SI, or SSI) of each of the 40 cases, with reference to the English guidelines (Department of Health 2013). These guidelines were provided at the start of the online survey, and were available to be viewed within each of the 40 cases.

The survey was available for completion from the 23rd April 2018 until 3rd July 2018. An incentive of 18 GBP was offered to optometrists to complete the survey. A pragmatic decision was taken not to offer the incentive to the ophthalmologists. It was advised by the ophthalmologists on the consensus panel that payment to the ophthalmologist participants would not have a significant effect on participation and the administrative process of claiming a payment could act as a disincentive to participation.

Ethical approval was gained from the School of Optometry and Vision Sciences research ethics and audit committee (approval number 1443) at Cardiff University. Consent to take part in the study was obtained at the beginning of the online survey. The research was conducted according to the tenets of the Declaration of Helsinki.

Analysis

The agreement between each rater group and the consensus panel was determined using eligibility as both a trichotomous variable (not eligible, SI, SSI) and a dichotomous variable (not eligible or eligible, i.e. encompassing both SI and SSI).

Modelling was then undertaken using the outcome: exact agreement with panel/disagree with panel. Given the 129 ratings for each case, i.e., one rating from each ophthalmologist and optometrist, we used a multilevel model in which inter-rater variability was quantified using a random effect. Initially, the variable, agreement (exact agreement with panel/disagree with panel), was modelled as an outcome in a logistic regression on rater group (ophthalmologist/optometrist), consensus panel eligibility rating, and an interaction term between the two. In addition, bilateral atrophic AMD was then included as another variable in the modelling.

Krippendorff’s alpha was used to calculate inter-rater agreement within each clinician group and is appropriate for use with the trichotomous rating outcomes [14].

All analyses were conducted in R Version 3.5.1 and mixed effects models were fitted using the lme4 package [15].

Results

The demographic and clinical details of each of the 40 cases are shown in Table S1 (online only supplementary table) and in Fig. 1a, b. Primary causes of vision loss in the 40 case records included: cataract, neovascular and atrophic AMD, Stargardt disease, cone dystrophy, diabetic eye disease, glaucoma, optic neuritis, nystagmus, retinal detachment, homonymous hemianopia resulting from stroke and retinitis pigmentosa.

Fig. 1: Case characteristics and agreement with consensus panel.
figure 1

a The percentage of cases by gender and consensus panel outcome, according to demographic and clinical characteristics. b The consensus panel outcome with respect to visual acuity and time since diagnosis. c The number of cases out of 40 in which ophthalmologists (left) and optometrists (right) agreed with the consensus panel in the assessment of eligibility for certification as a dichotomous variable (not eligible or eligible; top) and as a trichotomous variable (bottom). d The distributions of the number of cases out of 40 in which ophthalmologists and optometrists agreed with the consensus panel in the assessment of eligibility for certification as a dichotomous variable (not eligible or eligible; left panel) and a trichotomous variable (not eligible, sight impaired, or severely sight impaired; right panel). Boxplot limits in (b) and (d) indicate the lower sample quartile, sample median and upper sample quartile.

Full size image

Of the 40 cases, the consensus panel agreed that 12 were not eligible for certification, 15 were eligible to be certified as SI, and 13 certified as SSI (Table 1). There were no cases in which a group agreement by the consensus panel was not reached.

Table 1 Case characteristics by consensus panel outcome.
Full size table

Survey responses from 30 consultant ophthalmologists and 99 low vision optometrists were received, accounting for 52% and 61% of those eligible to take part (i.e. the total number of clinicians in Wales) from each rater group, respectively.

Each of the 40 cases therefore was rated by 129 clinicians, giving a total of 5160 ratings. Ophthalmologists produced 1200 ratings; and optometrists, 3960 (Table 2). These were not independent observations: each of the 129 raters classified the same 40 cases. The 28 cases rated as eligible for certification by the panel thus provided 28 × 30 = 840 observations by ophthalmologists and 28 × 99 = 2772 observations by optometrists. Seventy-five percent (n = 631) of the ophthalmologists’ ratings of those cases were eligible for certification. Eighty-eight percent (n = 2440) were eligible in the optometrists’ view (Table 2).

Table 2 Number of ratings by trichotomous classification of cases by the consensus panel and by each rater group.
Full size table

Compared with the consensus panel who considered 12 of the cases to be ineligible, a median of 19 (IQR 18.25, 20) cases were rated ineligible by ophthalmologists and 14 (12, 16) by optometrists. Fifteen cases were rated as SI by the consensus panel, whilst 11 (9.25, 11) and 17 (15, 19) cases were rated as SI by the ophthalmologists and optometrists, respectively. Thirteen cases were rated as SSI by the consensus panel, whilst 10 (9.25, 10) and 8 (7, 11) cases were rated as SSI by ophthalmologists and optometrists, respectively. This, however, does not indicate the level of agreement concerning individual cases. We then determined, for each case and rater, whether the rater agreed with the consensus panel’s outcome, either when considering the dichotomous (eligible/not eligible) rating or the trichotomous (not eligible/SI/SSI) rating.

The agreement between each rater group with the consensus panel is shown in Fig. 1c. Figure 1d is an alternative presentation of the same data. For the dichotomous rating, the optometrists’ distribution is clearly different to the ophthalmologists’ and in better accord with the consensus panel’s outcome. For ophthalmologists, the median (IQR) number of cases in which there was agreement with the consensus panel was 33.0 (31.0, 33.0); corresponding values for optometrists were 36.0 (34.0, 36.5). For ophthalmologists, the mode was 33, in which 13 ophthalmologists (43%) agreed with the consensus panel. Similarly, for optometrists, the mode was 36 cases, in which 26 optometrists (26%) agreed with the consensus panel. For the trichotomous rating, the median (IQR) number of cases in which there was full agreement with the consensus panel was 30.0 (28.3, 30.0); corresponding values for optometrists were 30.0 (27.0, 31.5).

Table 3 (charted in Fig. 2a) shows the probability of rating the eligibility of the cases in exact agreement with the consensus panel, for each rater group, together with 95% confidence intervals derived from the fitted model. The greatest differences between optometrists and ophthalmologists occurred for cases determined by the consensus panel as SI: optometrists considered 72% of those cases as SI while ophthalmologists rated only 52% as SI (95% CIs 0.70, 0.75 cf. 0.47, 0.56). For cases rated as SSI by the consensus panel, optometrists and ophthalmologists considered 62% and 76% as SSI, respectively (95% CIs 0.59, 0.65 cf. 0.71, 0.80). Agreement on cases that were, according to the consensus panel, not eligible, was closer between clinician groups.

Table 3 Modelling outcomes showing the estimated probability of rating the eligibility of the cases in exact agreement with the consensus panel, for each rater group, with 95% confidence intervals derived from the fitted model.
Full size table
Fig. 2: The probability of agreement with consensus panel and distribution of cases judged as eligible.
figure 2

a The probability of agreeing with the consensus panel outcome, for each rater group. The vertical grey bars represent the 95% confidence intervals. The top panel in (a) shows the overall agreement and the bottom panel shows the agreement for cases in which the primary cause of vision impairment was (right) and was not (left) bilateral atrophic AMD. b Distribution of cases by percentage of raters judging the cases to be eligible. The top panel shows the overall distribution. The bottom panel shows the distribution for cases determined by the consensus panel as sight impaired only.

Full size image

Bilateral atrophic AMD was then added in to the model as an explanatory variable (Fig. 2a, bottom panel), which was selected for inclusion given its clinical significance. Overall, both clinician groups were more likely to agree with the consensus panel outcomes for cases of bilateral atrophic AMD than for cases in which it was not present. As previously, the greatest differences between optometrists and ophthalmologists occurred for cases eligible for certification as SI; however, these differences were less marked in cases of bilateral atrophic AMD.

As Fig. 2b (top panel) suggests, both ophthalmologists and optometrists largely agreed that those cases considered not eligible by the consensus panel were truly ineligible and that the cases considered SSI by the panel were eligible. Figure 2b (top panel) shows that for both groups, the classification of most cases was unambiguous: 19 of the cases were judged as eligible by over 90% of the optometrists, while a further 6 of the cases were considered eligible by less than 10% of the optometrists, i.e., over 90% of the optometrists considered those 6 to be ineligible. Ophthalmologists demonstrated a similar pattern with near unanimity over the classification of 16 cases as eligible and 12 cases were considered eligible by less than 10% of the ophthalmologists. For the cases determined as SI by the consensus panel (Fig. 2b, bottom panel), there were seven cases in which less than 50% of ophthalmologists agreed with the consensus panel, but only two cases in which same was true for optometrists.

There were 11 cases of complete agreement amongst all the ophthalmologists and optometrists, 1 of which all considered ineligible, the others being eligible.

Moderate to substantial inter-rater agreement was demonstrated within each rater group. Based on all 40 cases, for ophthalmologists, Krippendorff’s alpha values were 0.72 (95% CI 0.62–0.81) and 0.8 (0.70–0.88), for the dichotomous and trichotomous classifications, respectively. Similarly, the corresponding values for optometrists were 0.67 (0.53–0.78) and 0.73 (0.63–0.81).

Discussion

This study evaluated the clinical decision-making abilities of low vision optometrists and consultant ophthalmologists in certifying patients as vision impaired. Unlike the ophthalmologists, the optometrists were inexperienced in the process of certification. However, low vision optometrists are experienced in managing patients with low vision and thus have a theoretical understanding of the certification of vision impairment, but not a current role in the formal certification process.

The key finding of this study was that optometrists demonstrated comparable agreement relative to ophthalmologists, with the consensus panel outcomes on the eligibility of cases for certification. The similarity in performance between groups is demonstrated by the number of cases in which there was agreement with the consensus panel and the overall probability of rating the eligibility of cases in exact agreement with the consensus panel. For cases rated as SI, the probability of agreement with the consensus panel was greater for optometrists than for ophthalmologists, whilst the opposite was true for cases rated as SSI. Both clinician groups rated fewer cases as eligible relative to the consensus panel. Ophthalmologists were least likely to agree with the consensus panel outcome for cases judged by the consensus panel as SI, whilst optometrists were least likely to agree with the consensus panel for SSI cases. Whilst the results for SSI cases may reflect the naivety of the optometrists in the certification process, and may be partly explained by a stronger adherence to the clinical guidelines, the overall similarity between groups supports their ability to provide this service to patients.

In one case (case 3), classified as SI by the consensus panel, interestingly, 90% of the ophthalmologists classified this case as not eligible (Fig. 2b). Although this individual had better visual acuities than the guideline criteria for certification, she had a severe loss of contrast sensitivity, recent diagnosis of AMD, and lived alone. This suggests the rating of the consensus panel may have allocated more weighting to the circumstantial factors than the visual acuity status, relative to that of the ophthalmologists. Whilst contrast sensitivity is not specifically mentioned in the certification guidelines, clinicians may differ in their consideration of this outcome when it is available. However, decisions are never made on this outcome alone.

Both clinician groups were more likely to agree with the consensus panel across all eligibility classifications in cases of bilateral atrophic AMD, relative to the other causes of vision impairment. In these cases, the differences between clinician groups were less, relative to those cases in which there was another cause of vision impairment, i.e., not atrophic AMD. This difference was most marked for the SI cases (Fig. 2a).

AMD is the leading cause of certifiable vision impairment in England and Wales accounting for 50% of all certifications of vision impairment [16] and of these cases, atrophic AMD is the leading cause of vision loss in ~50% [17]. Given the lack of clinical therapeutic options for atrophic AMD, patients with this condition would not be routinely monitored within the hospital eye service. Yet, the vision loss associated with severe atrophic AMD meets the threshold for eligibility for certification. Therefore, these patients would particularly benefit from access to certification through primary care optometry, should it become available.

This is the first study to measure the agreement between optometrists and consultant ophthalmologists in the consideration of eligibility for certification of patients with vision impairment. Previous studies have examined the agreement between optometrists and ophthalmologists, in other clinical tasks [18,19,20,21,22,23]. Some have shown moderate to substantial agreement between these groups in the grading of anterior chamber angles [19] and in the evaluation of glaucoma [20,21,22,23]. Others have demonstrated poor levels of agreement between and within consultant ophthalmologists, in classifying patients with glaucomatous visual field defects [18]. However, such comparisons to the present study are limited given the different nature of these clinical tasks.

The strengths of the study include the substantial proportion of clinicians relative to the overall workforce in Wales who took part in the study. A consensus panel was adopted to provide a reference standard for clinical decision-making.

The limitations of the study include the online delivery of the survey, which may have resulted in the self-selection of clinicians with a specific interest to act as participants. A moderate number of anonymised cases were reviewed, although they were representative of the variety of disease types and individual circumstances of such cases. The grading of anonymised cases does not fully simulate the interaction that occurs between a clinician and a patient. In addition, whilst the ophthalmologists were experienced in the real life process of certification, neither of the clinician groups were familiar with the task of classifying abridged cases. A possible risk of bias could be attributed to the incentivisation to optometrists but not ophthalmologists, however, the payment was offered independently of performance in the classification task, and therefore should be expected to be independent of the recorded outcomes for each participant. All optometrist participants claimed the incentive.

Although the possibility that an ophthalmologist participant completed the survey more than once cannot be excluded, it is unlikely, given the time taken to review each case, which was presented in a random order in each survey.

Overall, the performance of optometrists was comparable to that of ophthalmologists in the rating of eligibility of virtual patient cases for the certification of vision impairment. The findings support the clinical decision-making ability of low vision optometrists in the certification of patients with vision impairment, especially in cases of atrophic AMD. A prospective study comparing the assessment of patients with bilateral atrophic AMD by low vision optometrists against a reference standard is warranted.

Main findings

  • The findings support the clinical decision-making ability of low vision optometrists in the certification of patients with vision impairment.

  • Optometrists demonstrated a comparable agreement relative to ophthalmologists, with a consensus panel on the eligibility of randomly selected, abridged cases for certification.

  • Both clinician groups were more likely to agree with the consensus panel in cases of bilateral atrophic AMD, relative to the other causes of vision impairment.

  • The results provide evidence in support of policy change to allow low vision optometrists to certify individuals with atrophic AMD.

Summary

What was known before

  • The certification process to register patients as sight impaired or severely sight impaired is undertaken by consultant ophthalmologists, in the UK.

What this study adds

  • Ophthalmologists and optometrists demonstrated an equivalent rating of cases for the certification of vision impairment. The evidence suggests that it is feasible for low vision optometrists to certify patients with atrophic age-related macular degeneration.