Introduction

Rapid advances in educational technology have led to significant changes in the delivery of medical education. Computer-assisted learning (CAL) programs are becoming increasingly sophisticated in their design to supplement medical curriculums and facilitate traditional modes of teaching. This leads to the creation of promising pedagogical strategies for medical educationalists and new virtual clinical environments for learning. Novel technology-enhanced learning modules have been developed in response to curriculum time constraints and shortage of educators and teaching resources.1 The advantages of ophthalmic CAL software include the absence of patient inconvenience in terms of stress or discomfort arising from students examining their eyes.2 The student benefits include accessibility to clinical cases at any time or any place with clinical presentations that cover the curriculum requirements.

The Virtual Ophthalmology Clinic (VOC) is an innovative CAL program designed to enhance teaching and learning by allowing medical students to sharpen their clinical reasoning skills by formulating a diagnosis and treatment plan on virtual patients with simulated eye conditions. The interaction occurs in a safe learning environment before commencing practice on real patients.

The VOC program offers flexibility and independence by enabling each student to learn in their own time and in off-site locations, creating better access for students in rural placements, while maintaining links with online tutors who provide regular and monitored feedback. To meet the needs of the learner, students are exposed to a broad range of conditions via virtual simulation, and any eye condition that students might not otherwise have the opportunity to experience in a clinical teaching environment can be integrated into the program. Recurrent access to clinical case information allows for easy revision.

The University of Sydney has conducted an extensive review of its medical programme, and the course satisfaction ranking of graduates has been found to be among the highest in Australia.3, 4 Subsequently, the purpose of this study was to determine the impact of VOC on student’s learning in ophthalmology within the medical program. A prospective randomised controlled trial (RCT) was conducted to determine the effectiveness of VOC in comparison with traditional modes of ophthalmology teaching. It was hypothesised that VOC would improve students’ knowledge and learning outcomes. A mixed methods research approach was employed to draw upon the strengths of both quantitative and qualitative dimensions in evaluating VOC. The evaluation was performed with a pre- and post-ophthalmic knowledge-based test, which covered history taking, interpretation of examination, management of eye disorders, and clinical reasoning. Secondary aims were to assess the long-term impact of VOC on retention rates and to determine students’ perception of the program.

Methods

Ethics approval was granted by The University of Sydney Human Research Ethics Committee (HREC, Protocol Number 11580). This was a randomised control trial that conformed to CONSORT (CONsolidated Standards of Reporting Trials) Statement guidelines and checklist items.5

To measure the students’ learning outcome, teaching experiences, and perceptions of VOC, two evaluation instruments were developed: an Ophthalmic Knowledge-Based Pre- and Post-Test (20 questions) and a VOC satisfaction questionnaire consisting of nine items. A qualitative evaluation of perceptions and experiences with VOC was made using a cross-sectional satisfaction questionnaire using a five-point Likert scale.6, 7, 8, 9, 10 Questionnaires were designed drawing from literature on questionnaire-wording skills.11, 12, 13, 14, 15, 16, 17, 18 The aim was to design questionnaires that were logically structured and worded to not influence the students’ answers.19, 20

A total of 188 medical students from the University of Sydney commencing their clinical ophthalmology rotation were randomised either to VOC incorporated into their ophthalmic rotation or to traditional ophthalmic rotation (Figure 1a). Sydney Medical School allocates students to clinical schools for the clinical component of the course. All clinical schools contain an ophthalmic unit that provides both secondary (general ophthalmic clinics) and tertiary ophthalmic care covering the following subspecialities: an anterior segment, cataract, glaucoma, medical and surgical retina, oculoplastics, and neuro-ophthalmology. Four clinical schools participated in the RCT (Table 1). The clinical schools have flexibility in allocating the clinical attachments. The length of time allocated to ophthalmology varies from 3 to 10 days of face-to-face clinical teaching. Students attend the ophthalmology clinical rotation in groups, and it was these groups that were randomised to either intervention (VOC) or control. The groups were randomised at the level of each clinical school, ensuring that the length of rotation was matched. The group allocation was made by the clinical school student coordinator from the supplied randomised list. Randomisation lists were generated by the statistician using the statistical software program Stats Direct (http://www.statsdirect.com). The analysis was performed with the intention-to-treat strategy. Students allocated to non-intervention were aware of the study at the outset, and participation was voluntary. The experimental group (n=95) received the VOC intervention as an additional educational resource, whereas students in the control group (n=93) took part in the traditional hospital-based teaching. Once the intervention was completed, the control group was given access to VOC. A 12-month follow-up test was administered. This was to allow equal access for all students to similar ophthalmology teaching experiences and resources. This analysis, as well as examination of retained ophthalmic knowledge, also addressed the potential bias of extra time spent on ophthalmic education activities within the VOC group.

Figure 1
figure 1

(a) Procedure for student randomisation and allocation. (b) Simulated patient consultation within the virtual ophthalmology clinic. The conversation navigator is shown with history-taking icons, and floats over the simulated consulting room where the common ophthalmic instruments including an ophthalmoscope, a slit lamp, a Snellen visual acuity chart, and eye movement charts are located. (c) The conversation navigator with systemic viewer panel displayed.

Full size image
Table 1 Student demographics and test scores
Full size table

All students underwent a baseline assessment before the teaching. Once randomised, each student was met face-to-face in the week before commencing their ophthalmology rotation to complete their pre-test. A post-test was administered at the end of each rotation to measure the within-subject change in the learning outcome. There was no time limit set for the test, which was supervised and closed book; however, all students completed the test within 20–30 min. To eliminate the possibility of crossover contamination by students sharing the resource with the control groups, each student was given a specific password to log onto VOC.

Statistical methods and data analysis

Data were analysed on an intention-to-treat basis using the package SPSS Version 17 (SPSS Inc., IBM, Chicago, IL, USA). The primary outcome variable was the within-subject change in knowledge score (maximum total 20) from pre- to post rotation. Independent sample t-tests were used to compare the within-subject change in score between the control and experimental groups and to test for differences between the groups at pre-test and at 12 months’ follow-up. Paired t-tests were used to test for within-subject improvements in score from pre- to post rotation separately in each group. Qualitative data from the student satisfaction questionnaire was coded, organised into themes, and interpreted in order to summarise, draw conclusions, and make recommendations for enhancing VOC.

Virtual Ophthalmology Clinic overview

The Virtual Ophthalmology Clinic (http://sydney.edu.au/medicine/eye/virto/web2-singlecase/) provides a computer-based setting of the doctor–patient relationship in a virtual consulting room. It comprises 10 clinical cases designed to cover the major causes of blindness and common ophthalmic presentations to primary care. These include the following: age-related macular degeneration (AMD), glaucoma, diabetic eye disease, foreign body, blepharitis, herpes simplex keratitis, myopia, scleritis, uveitis, and watery eye. All cases were recorded from real patient interviews, transcribed, and then simulated as real patient presentations performed by actors in a video interview. The patient responses were then mapped to the conversation navigator icons (Figures 1b and c).

Virtual Ophthalmology Clinic operation

After logging on, the student is randomly allocated a patient with a clinical ophthalmic problem. The whole clinical encounter is divided into four sections: medical history taking, clinical examination, laboratory investigations, and patient management formulation. These are performed sequentially, with the option to go back but not forward until the section is completed.

The Conversation Navigator (Figure 1b) is an interactive tool used for simulating history taking. This is made possible through specific button icons, which represent lines of enquiry or questions, including concepts of time, pain, and ocular symptoms and review of systemic health. The student determines the history-taking course by selecting icons to direct the virtual interview, thereby uncovering clues to the condition. The student directs the questioning in a nonlinear manner. Students must summarise their history findings and propose differential diagnoses before moving to the examination section of the virtual encounter. Similarly, the differential diagnoses are updated at the examination section before progressing to the investigation section. The findings are recorded in an electronic medical record incorporated into the program. These are saved and submitted to an online tutor for correction and feedback.

The systemic viewer (Figure 1c) allows the student to explore a systems review that may be related to the ophthalmic diagnosis. The systems are dermatology, ear nose throat, respiratory, cardiovascular, immunohaematology, endocrine, central nervous system, gastrointestinal, urogenital, and musculoskeletal.

Each icon represents a specific history question that allows the learner to interact with the virtual patient. In addition, some questions contain a submenu that allows students to probe deeper in their interview. The pain icon, for example, contains the following subquestions: location, radiation, intensity, type, and aggravation/relief factors.

Results

This was the first randomised controlled trial conducted in an Australian medical student ophthalmology curriculum, and it evaluated the effectiveness of an educational intervention.

The mean pre- and post-rotation scores in the intervention group were 12.5 (62%, SD 2.2) and 16.0 (80%, SD 1.8), and in the control group the results were 12.8 (65%, SD 2.5) and 14.8 (75%, SD 2.2). The pre-test scores were comparable (P=0.316).

There were statistically significant (P<0.001) within-subject improvements pre- to post rotation for both the control and intervention groups (mean improvement in the control group was 2.0 (95%CI 1.3–2.6) and that in the intervention group was 3.5 (95%CI 3.0–4.0)). The improvement was significantly greater in the intervention group (mean difference in improvement between groups was 1.5 (95%CI 0.8–2.3, P<0.001) (Figure 2).

Figure 2
figure 2

Mean pre- and post-rotation scores for the VOC intervention and control groups. There were statistically significant (P<0.001) within-subject improvements pre- to post rotation for both the control and intervention groups. The difference in within-subject change (Post–Pre) was highly significant (P<0.001, 95% CI 0.8–2.3), with the change observed in the intervention group being significantly larger than that observed in the control group.

Full size image

The post-test was re-administered 12 months after the original intervention to assess the retained knowledge. Those who were in the original intervention VOC group scored a mean of 15.1 (76%), whereas the original control group with subsequent voluntary access to VOC scored 13.5 (68%). Thus, the intervention group scored on average 1.6 (8%) (95% CI 0.4–2.7, P=0.007) more questions correctly out of 20 compared with controls. The learning benefit of the VOC was maintained after 12 months.

The students’ perceptions and attitudes towards the program were measured by the qualitative questionnaire (Table 2). Ninety-three students completed the questionnaire. The VOC module was highly regarded and enthusiastically received as seen in the students’ overall satisfaction and positive responses. Of the students, 88% found using VOC a highly valuable activity that aided their learning in ophthalmology, and 93% agreed that the VOC would be a useful addition to teaching in the ophthalmology curriculum. In all, 92% of students agreed that the feedback and answers that they received for each case aided their learning. An overall 75% of students agreed that they learnt history-taking skills from the VOC that they could apply to their clinical work, and 73% agreed that they learnt eye examination skills that they could apply to their clinical work. Among the students, 88% agreed that their understanding of the diagnosis and management of eye diseases was aided by VOC, and 75% agreed that VOC aided understanding in how an ocular history provides clues to the site of pathology within the eye, facilitating which part of the eye to examine. Overall, the VOC program was perceived as easy to navigate (57% agreed). The students’ overall assessment of the VOC program compared with traditional methods found that 56% of students rated it more effective than traditional methods.

Table 2 VOC appraisal responses
Full size table

Discussion

The application of advanced educational technology has been of growing importance for teaching in ophthalmology.21 CAL addresses some of the issues associated with limited time, patient availability, and exposure to ophthalmic cases, by enabling increasing amounts of knowledge to be presented in the ophthalmic sciences. Devitt et al22 have shown that CAL used in ophthalmology can be an effective learning resource in teaching medical students clinical skills competence,23 as well as problem solving skills,24 including application to surgical traineeships.25 Furthermore, assessment methods used in ophthalmoscopy have shown better outcomes with similarities to real-life clinical activities compared with methods that use traditional model eyes.26 Improved student learning outcomes in ophthalmology have also been found with the implementation of CAL programs.24, 27 Application of virtual patients particularly for teaching and for assessing medical students’ clinical skills and reasoning has proven to be valid and cost-effective.28 The use of the virtual ophthalmology clinic in our study reinforces these findings.

We implemented a simulated ophthalmology encounter using a virtual ophthalmology clinic setting and a conversation navigator as a tool to enable history and examination assessment. The conversation navigator is unique within ophthalmic education and in most medical education CAL programs in that it allows nonlinear student (practitioner)-directed control over the interview process.

In this study our VOC trial found an improvement in the number of correct answers for both the control (mean improvement 2.0, 95%CI=1.3–2.6, P<0.001) and intervention groups (mean improvement 3.5, 95% CI=3.0–4.0, P<0.001). The improvement was significantly higher in the intervention group (mean difference between within-subject improvement 1.5, 95% CI=0.8–2.3, P<0.001). The learning benefit of the VOC was maintained after 12 months when follow-up testing of graduates resulted in the intervention group scoring on average 1.6 (95%CI 0.4–2.7, P=0.007) more questions correctly out of 20 compared with controls.

Ophthalmology e-learning modules have been reported to be engaging and fun by students,29 and frequent use, which may have been enhanced by the engaging presentation mode, led to higher marks.29, 30 Computer animation technology30 and digital PBL sessions31 can result in improved learning outcomes, and the virtual cases stimulated interest and motivated students. Similarly, in our evaluation we found that 94% of students reported that VOC would be a useful addition to the ophthalmology curriculum. Eighty-six percent of students regarded VOC as a valuable activity assisting their learning in ophthalmology. The results of our evaluation demonstrated that 56% of students rated VOC as more effective compared with traditional methods of ophthalmology teaching (Table 2).

There has been only one previous randomised study conducted in ophthalmology medical student teaching. Prinz et al32 evaluated ‘Ophthalmic Operation Vienna’, in which surgical videos were accompanied by 3D animated sequences of all surgical steps for five ophthalmic operations. In the evaluation of students’ understanding of cataract and glaucoma surgery, the 3D group outperformed the control group. These results using an improved methodology highlight the advantages of high-quality multimedia learning modules within a curriculum.

The VOC intervention examined a computer-based program to enhance ophthalmic history taking as well as selection and interpretation of ocular examination findings and ocular investigation results within a traditional problem-based learning curriculum. We found that the gain in ophthalmic knowledge was greater in the VOC intervention group.

Limitations in our study included that although the overall mean result was higher in the intervention than in the control group, there was variability in individual scores and not every student within the intervention group showed an improvement in scores. Reasons for this may include student characteristics such as age, gender, IT skills, and educational background (initial undergraduate degree). Further, we did not include the perceptions of academics in this study, which will be addressed in a future evaluation.

The virtual ophthalmology clinic addresses a number of issues in medical education, including time constraints, variability in clinical experience, and the integration of history-taking skills with medical science using ophthalmology as the model. These features form the basis for guiding integration of clinical information into a relevant management plan for the patient. The program, by providing the student with the opportunity to make decisions and by assisting in self-directed learning, allows the teacher to be more available for the important task of sharing in the group discussions and assisting in the decisions about patient needs.

On the basis of a statistically significant gain in knowledge, higher knowledge retention rates, and supported by qualitative data indicating positive student learning experiences we can conclude that VOC is an effective resource for teaching ophthalmology to new generations of tech-savvy medical students.