Introduction

Healthcare technology advances have reduced mortality and morbidity, increased longevity and improved quality of life for people worldwide. However, this has also resulted in ever increasing healthcare costs both as an absolute value and as a percentage of GDP since the inception of the National Health Service in the UK [1]. Service evaluations are increasingly used to address key stakeholder requirements, informing funding provisions, and guide local and national service provision models.

Within ophthalmology the landscape of service provision has greatly changed due to technological advances as seen by the setting up of intravitreal injection clinics, glaucoma monitoring units, and diabetic retinopathy screening services. This rapidly changing world of clinical practice due to technological advances has resulted in service evaluations becoming a vital part of the armemantarium in the clinical governance toolkit.

Novel medical devices, imaging modalities and pharmaceuticals are rapidly expanding and accounting for ever increasing quality of healthcare delivery as well as percentage of healthcare budgets.

Procurement justification for novel medical devices is becoming increasingly focused on value rather than price alone, which has made Patient Reported Outcome Measures (PROMs) more widely used in their evaluation [2]. In the UK, 65% (121 out of 186) of novel medical device studies between 1998 and 2018 included PROMs [2]. While in the US, there has been over a 500% increase in the utilisation of PROMs in medical device trials between 2009 and 2015 [3].

The primary aim of this paper was to demonstrate the importance of service evaluations in the refinement of a new service. The example we provided is of novel artificial eye service, which can provide life-like artificial eyes within a short manufacturing timeframe. Our aim is to improve the initial rehabilitation pathway and long-term quality of life of artificial eye patients. Therefore, the outcome measures used are based on quality of life.

Service evaluations and clinical audits are both placed in the sphere of clinical governance. Clinical audits tend to evaluate specific outcomes of a service with a focus on clinical outcomes. Whilst service evaluations tend to have broader aims of addressing key stakeholder requirements, informing funding provisions, and guiding larger structural changes to service provision.

Artificial eyes

Patients may require surgery to remove a blind painful eye following trauma, cancers or congenital conditions. The operations carried out include eviscerations, enucleations and orbital exenterations. Following healing of the surgical area the subsequent rehabilitation involves fitting an artificial eye. Patients with ocular prostheses report a reduced quality of life and increased prevalence of depression and anxiety [4, 5]. In 2006, Song et al reported that patients who felt their artificial eye was imperceptible to others had higher patient satisfaction rates, irrespective of the surgery performed or implant provided [6]. Despite this knowledge, there has been limited research into the scope for improvement in appearance of ocular prostheses.

Following clinical observations and patient testimonials, there was clearly a high level of unmet patient need and demand for improvements in ocular prostheses within our local area. This stimulated the development of a novel technique for fabricating ocular prostheses by Maxillofacial prosthetists and medical illustrationists in Leeds Teaching Hospitals NHS Trust. This new manufacturing technique allows digital photography of the patient’s unaffected eye to produce high definition prostheses in a short time frame with increased patient personalisation [7].

Our service

The production of our new technique began development within LTHT over 10 years ago, and has anecdotally been met with great praise from patients. All patients were originally recruited into the service following the provision of a National Artificial Eye Service (NAES) prosthesis. NAES are currently the national providers of ocular prostheses to National Health Service (NHS) users. Although there are over 30 other artificial eye service providers nationally, mainly within maxillofacial prosthetics laboratories.

On referral to the LTHT service the maxillofacial prosthetist (PB) and medical photographers (TZ, TA) spend time with the patient designing their prosthesis and taking necessary digital photographs. Once fabricated, the prosthesis is fitted and minor adjustments made to suit patient’s preference if required [7]. There is future scope for this process to be undertaken within one clinical session, however at present this is spread over the course of a week due to non-clinical logistical pressures. Figure 1 demonstrates the current patient journey through this high definition ocular prosthetic service.

Fig. 1: Patient pathway.
figure 1

Flowchart describing patient flow through the new ocular prosthesis service [4].

Full size image

The scope of this paper is to demonstrate formalised quality of life responses using validated questionnaires from patients utilising this new service. This information will support service improvement by identifying areas for potential development.

Methods

Our service evaluation was undertaken at Leeds Teaching Hospitals NHS Trust in the Leeds Dental Institute. The Trust Research and Innovation Centre supported and approved the service evaluation. A sample size of ten adult patients (over 18 years old) was selected to capture a representative cross-section of services users on a convenience basis. This evaluation was performed over a nine month timescale (December 2018 to September 2019) and covered the complete patient pathway from initial visit, to manufacture of prosthesis, and finally patient questionnaires. A response rate of 80% (n = 8) was achieved.

Patients were evaluated via telephone conversations or letter responses, as per their personal preference. The majority (n = 7) opted for telephone conversations considering their visual impairment, often affecting their remaining biological eye and causing difficulty in reading a written questionnaire with ease. The option for either phone or letter responses was highlighted as very important by patients during a previous Patient and Public Involvement (PPI) initiative regarding this new service. This PPI allowed patients the freedom to express their thoughts about the service directly to those involved in its development.

Our choice of patient reported outcome measure (PROM) was based on literature review and online database searches; FACE-Q (FACE-Q © 2013), DAS24 (Derriford Appearance Scale 24) and HADS (Hospital Anxiety and Depression Scale) were selected [8,9,10]. Literature review of PROMs specifically in relation to artificial eyes is scarce. However, there are many well validated PROMs for anxiety, depression, social interaction and appearance which can be applied to those who have lost an eye and require prosthetic replacement [10, 11]. In itself this creates a challenge in balancing between questionnaire validity and applicability.

FACE-Q patient reported outcome instrument was assessed and eight individual questionnaires were selected based on clinical relevance: (1) Psychological function, (2) Satisfaction with eyes, (3) Adverse effects: Eyes, (4) Satisfaction with outcome, (5) Appearance-related psychological distress, (6) Social Function, (7) Satisfaction with facial appearance, (8) Expectations. Development of the FACE-Q instrument was well constructed resulting in reliable, valid and responsive measures [12,13,14,15]. Participants would be asked to state how much they agree or disagree with statements such as “I feel positive about myself” and “I feel confident” in the Psychological Function questionnaire. Scoring system is standardised with a range of 1–4 and higher scores reflecting a better outcome. A Conversion Table transforms the raw scale summed score into an equivalent Rasch transformed score from 0 (worst) to 100 (best).

DAS24 (Derriford Appearance Scale) is an instrument designed to measure adjustment to problems of visible difference and disfigurement [8]. The initial open scoping questions are followed by those with psychometric scales for measuring distress and dysfunction. Participants are asked questions such as “How distressed do you get by shopping in department stores/supermarkets?” and “How distressed do you get when other people make remarks about your feature?”. The total score ranges from 11-96. Higher scores represent greater levels of social anxiety and social avoidance.

HADS (Hospital Anxiety and Depression Scale) is a well-established validated scale with almost forty years of application to a broad variety of medical and surgical specialties [9]. It was developed to evaluate patients with physical health problems. Being a relatively straightforward and quick scale to complete helps ensure maximal respondent attention. The fourteen items are divided evenly between anxiety and depression subgroups. Participants are asked to answer questions such as “I still enjoy the things I used to enjoy” and “I feel restless as if I have to be on the move”. Cumulative scores for each subgroup range from 0 to 21 and higher scores indicate greater levels of anxiety or depression. Scores of 0–7 are in the “normal” range; 8–10 suggests borderline abnormal levels of depression or anxiety; 11–21 is strongly suggestive of clinical depression or clinical anxiety.

Although these measures were specifically designed for clinical trial evaluations with strong foundations in their development, the binary patient acceptability of the service was seen as a simple yet highly effective way to gauge patient preference and impact of the service.

Descriptive analysis of the Mean and 95% Confidence Interval (CI) was undertaken for all questionnaires. Statistical analysis was performed using SPSS v.21 (SPSS Inc, Chicago, Illinois). Principal Component Analysis considered significant correlation at factor loading α > 0.4. PCA was undertaken only on the comparable FACE-Q questionnaires due to the measures having good reliability, validity and responsiveness.

Results

Descriptive statistics

FACE-Q

The FACE-Q instruments’ equivalent Rasch transformed score Means and 95% CI are discussed below and detailed in Table 1.

Table 1 FACE-Q: Descriptive analysis of the mean and 95% confidence interval.
Full size table

The lowest scoring instrument suggesting highest levels of distress was for Adverse Effects: Eyes scoring at 36.6(CI:27.5,45.7) and asking whether participants were bothered by: “How your eyelid scars look?”, “Your eyes look hollowed out?”, and “Eye irritation?”. Appearance-Related Psychological Distress instrument also scored low at 37.4(CI:13.1,61.7) and assessed areas such as “I feel stressed with how I look” and “I tend to avoid being around people”.

The highest scoring instrument suggesting least distress was for Satisfaction with Outcome at 63.7(CI:47.5,80.0) including questions such as: “I am pleased with the result” and “I am surprised at how good I look in the mirror”. Expectations questionnaire also scored high at 61.8(CI:51.8,71.9) and covered areas such as “People will tell me how great I look” and “I will feel like I fit in.”

A broad comparison to Klassen et al. evaluation of FACE-Q questionnaires for patients undergoing cosmetic eye surgery is a good starting point, as there is currently no validated normative data for this qualitative tool with respect to artificial eye patients [15]. The following instruments had similar scores. Psychological Function 54.7(CI:32.6,76.9) and Social Function 54(CI:38.4,69.6) fell just below Klassen’s rounded up range of 60-90 for both questionnaires. While appearance related instruments of Satisfaction – Eyes 59(CI:38.7,79.3) and Satisfaction - Facial Appearance 53.6(CI:39.7,67.4) were within Klassen’s rounded up range of 50–80 and 50–70, respectively.

Table 1 demonstrates the FACE-Q instruments descriptive statistics.

DAS24

DAS24 Total Score Mean and 95% CI is 41.7(CI:24.6,58.7). There was an outlier with a total score of 91/96, which may have impacted the overall result. The 95% CI is broad indicating requirement for a larger data set in future studies. McBain and colleagues report a mean of 37.5 in a similar patient population of artificial eye users [5]. Questions contributing the most to the total score were “I am self-conscious of my feature”, “How confident do you feel”, and “How distressed do you get when other people make remarks about your feature?”.

Table 2a-b shows the DAS24 Total and Individual Question descriptive statistics.

Table 2 a-b. DAS24: Descriptive analysis of the mean and 95% confidence interval.
Full size table

HADS

HADS Total Anxiety Score Mean and 95% CI is 6.14(CI:2.5,9.7). According to the HADS classification of the total score the mean falls within the normal category (0–7). In a similar study population of artificial eye users, McBain et al. report a slightly higher mean score at 6.9 in their study [5]. In our study population the questions contributing the most to the anxiety total score were “I feel tense or wound up” and “I can sit at ease and feel relaxed”.

HADS Total Depression Score Mean and 95% CI is 6.28(CI:1.9,10.6). The HADS classification of the total score is the same as for anxiety scale. The mean falls within the normal category. McBain et al. reported a higher mean score of 7.6 [5]. We found the questions contributing the most to the total score were “I still enjoy the things I used to enjoy” and “I feel as if I am slowed down”.

Table 3a-d demonstrates the HADS Total Score and Individual Questions descriptive statistics.

Table 3 a–d. HADS: Descriptive analysis of the mean and 95% confidence interval.
Full size table

Binary acceptability

Binary patient acceptability of the new service was 100% (8/8) (Fig. 2).

Fig. 2: New ocular prostheses appearance.
figure 2

Three cases demonstrating the new ocular prostheses with different iris colours provided by the service. The artificial eyes are as follows: top (patient’s left eye with brown iris), middle (patient’s right eye with green iris), bottom (patient’s left eye with blue iris).

Full size image

Principal component analysis

PCA is an analytical statistical method of data reduction. As a dimension reduction tool it identifies the correlation between the variables (questionnaires) tested and assesses whether it is possible to reduce the number of variables while maintaining data variance. Correlations were significant when factor loading is at α > 0.4. The minimum number of variables is reached when their sum total accounts for ≥80% variance.

PCA analysis showed two main components extracted accounted for 84% of the total variance. Thereby two dimensions in the component space account for 84% of the variance.

The first seven questionnaires laid into one component (PC1) which accounted for 70% of the total data variance. The second component analysis (PC2) contained question number 8 (FACE-Q Expectations) which accounted to 14% of the data variance. Therefore, it was concluded that the number of variables can be reduced to 2 to include any of the variables in PC1 in addition to PC2. Table (1) shows the component matrix and the factor loading for each variable. PC1 contained questionnaires numbered (1–7) all of which were highly correlated with very good to excellent internal consistency with factor loading (α = 0.7-0.9). PC2 contained question number 8 with a factor loading of (α = 0.8).

Table 4 shows the FACE-Q Principal Component Analysis.

Table 4 FACE-Q: Principal component analysis.
Full size table

Discussion

The evolution of orbital prostheses from the traditional hand-painted design has been explored by previous research teams [16, 17], and usually published via single patient case reports. However, there is no evidence in the literature regarding the utilisation of digital photography and high-definition printing to identically replicate a patient’s unaffected eye. Our technique is currently used with patients who already have an ocular prosthesis, provided by a different service (NAES).

Comparative bias may have arisen within patient responses, despite our efforts to minimise this by encouraging patients to evaluate the new prosthesis solely on its own merits. Future research should look at PROM for patients who have never had a prosthesis before.

Reportable outcome measures from family, friends and clinicians of patients could prove useful. Family and friends have been shown to notice different aspects of the prostheses from our previous PPI experience. Some patients with an ocular prosthesis have poor visual function in their remaining eye, and therefore find it difficult to judge the appearance of a prosthesis. In these cases, those close to the patient are a valuable source of opinion.

The sample size for this evaluation was limited. This number may have reduced the range of responses that could have been received, and so there is scope for missing patient opinions and the identification of potential strengths and weaknesses of the service. However, at this time the evaluation team felt the sample size would provide an indicative representation of the current service. We plan a larger scale review of the service at a later date.

There is no specific PROM for use with patients who have an orbital prosthesis. We have therefore used a large number measures for a thorough preliminary service evaluation. However, patient fatigue and resultant degradation of data quality may occur when patients are required to complete excessive numbers of questionnaires. We have therefore used a statistical tool, PCA, in order to select a few high-yield questionnaires based on their data variance. Subsequent service evaluations that we will undertake will only use 2 out of the 8 FACE-Q questionnaires that we used for our initial evaluation. This should reduce patient fatigue, improve data quality and streamline the subsequent service evaluation. We would recommend PCA for refining service evaluations when there are no specific PROM.

Service evaluations are within the remit of clinical governance and not research. As a result they can provide broad although still valuable results to guide service provision decisions. However, they are not designed to evaluate potential confounding features such as depression, anxiety and reason for surgery with respect to PROM results. A feasibility study followed by a full clinical trial, within the sphere of research, would be able to compare and contrast the traditional artificial eye to the new high-definition artificial eye. This step-wise approach would allow for evaluation of the patient reported outcomes in the feasibility study, as there no specific outcomes for artificial eyes. And thereby allow a better understanding, evaluation and minimisation of potential sources of bias in the full study. We plan to further evaluate the novel high-definition artificial eye with this approach.

The results of this service evaluation will be disseminated to key stakeholders within the provision of orbital prosthesis within our region and also nationally, in the hope of making substantial changes and informing funding provisions [18]. This aligns with the primary goal of evaluations in engaging stakeholders as active participants in the process and that the findings will be meaningful and useful to those ultimately responsible for assessing and improving the service. Regarding this service the key stakeholders involved are the participants (patients), active respondents such as local Eye Clinic Liaison Officers and Royal National Institute of Blind People, consultants such as NAES and Specialist Commissioning Services Group representation, and the clinical team caring for the patients – Oculoplastic Surgery, Medical Imaging, Maxillofacial Prosthesis and Maxillofacial Surgery departments.

Conclusion

We hope to demonstrate the importance of service evaluations with respect to rapidly evolving technological advances in medical devices, pharmaceuticals and imaging modalities. Procurement justification for novel medical devices is becoming increasingly focused on value rather than price alone, which has made PROMs more widely used in their evaluation. This is becoming especially important with increasing technological advances that need to be balanced against rising healthcare costs. Further feasibility and full clinical studies are required to confirm the positive results of the novel artificial eye service we have evaluated with respect to the traditional approach.

Summary

What was known before

  • Healthcare technology advances have reduced mortality and morbidity, increased longevity and improved quality of life for people worldwide. However, this has also resulted in ever increasing healthcare costs. Ocular prostheses manufacture involving hand painting has changed little over the last century. There is a demand for improved aesthetics, personalisation and service provision from patients.

What this study adds

  • Service evaluations are important to efficiently incorporate rapidly evolving technological advances in medical devices, pharmaceuticals and imaging modalities. Patient reported outcome measures are increasingly used to demonstrate value and justify procurement of novel medical devices. Overall, patients had positive outcome measures with this new prosthesis. Further research will help to formally quantify the differences.