Introduction

Ophthalmology has a long tradition of relying on objective, psychophysical measurements of vision to define a person's functional ability or impairment. Measures of visual acuity and visual fields, in particular, have been used clinically to judge the effect of disease and the efficacy of treatment on visual function. Although many clinical treatment trials and epidemiological studies depend upon vision impairment measures as the primary study variables,1, 2 they are no longer considered sufficient. There is a growing demand to include patient-based, visual function assessments in the measurement of treatment outcomes and demographic distributions.3 Such patient-based, visual function assessments are usually accomplished with quality-of-life or disability questionnaires,4 and several such questionnaires have been developed and evaluated among a variety of patient populations with a spectrum of ocular diseases.5, 6, 7, 8, 9, 10, 11, 12

For the National Eye Institute Visual Function Questionnaire-25 (NEI-VFQ-25)13, 14 questions were distilled from a series of focus groups with patients who had age-related cataracts, glaucoma, age-related macular degeneration, diabetic retinopathy, or cytomegalovirus retinitis,13 and from questions in the earlier 51-item NEI-VFQ.15 The VFQ-25 consists of a base set of 25 vision-targeted questions representing 11 vision-related tasks, plus an additional single-item general health-rating question. The reliability and validity of the VFQ-25 seems to be similar to those of the 51-item version.15

The NEI-VFQ-25 has been administered primarily through an interview conducted by a sufficiently qualified and trained professional (interviewer-NEI-VFQ-25). Performance time is extremely dependant upon the patient's age and his or her hearing difficulty. The elderly may require up to 20 min to complete the questionnaire. In addition, there seem to be several problems associated with the interviewer-NEI-VFQ-25 questionnaire that limit its routine use in the ophthalmologist's or optometrist's office. First, an interviewer who can question knowledgeably must be trained and must be salaried for the time spent. The patient's answers may be filtered or manipulated by the interviewer or altered by the patient in order to please the interviewer. The patients' answers may be adjusted or coloured by what they believe the interviewer would like to hear, or by their fear of consequences such as losing their driver's licence or independence. In addition, the information that has been returned from such questionnaires, while indicating task difficulty and accompanying depression, is often viewed by the physician as being without practical solutions for assistance (such as refraction, task lighting, etc.). For these reasons, the interviewer-NEI-VFQ-25 has not been used routinely in clinical practice, often limiting, unfortunately, the physician's ability to understand the true disability of their patients and to offer assistance.

The self-administered-NEI-VFQ-25 (self-NEI-VFQ-25) is conducted through a paper survey and requires a considerably longer time to complete than the interviewer-NEI-VFQ-25,15 especially if the patient's vision is reduced. Furthermore, the paper format does not easily allow response-based bypass of inappropriate sections.

We have developed an interactive, computer-based, visual function questionnaire designed to be delivered through point-and-click responses carried out on a PalmPilot-PDA. In this form, a self-administered VFQ that is interviewer independent and that can be used in an office setting for a wide range of visual acuities, ages, and ocular pathologies can be offered to a practitioner. The questionnaire represents a modification of the NEI-VQ-25 questions and answer scales for persons with limited education in order to enhance the clarity and understanding for self-administration.

The purpose of this study was (1) to evaluate the Palm Pilot-Visual Function Questionnaire (PalmPilot-VFQ) testing performance in comparison with both the interviewer-NEI-VFQ-25 and self-NEI-VFQ-25, and (2) to determine Rasch-scaled measures of visual disability for the self-administered PalmPilot-VFQ.

Materials and methods

Subject recruitment, eligibility and procedures

One hundred and thirty-five consecutive subjects were recruited to this study from among the patients visiting an out-patient retina clinic. Subjects were recruited without regard to age, gender, racial or ethnic restrictions. The tenets of the Declaration of Helsinki were followed and the study was approved by the Jefferson Health Care Main Line Hospital Association Ethical Committee. Informed consent was obtained after the nature of study had been explained. The interviewer-NEI-VFQ-25 was administered followed by the PalmPilot-VFQ on two different visits spaced at least 1 week but <3 months apart and without intervening treatment. At a third visit, the self-NEI-VFQ-25 was administered to 50 patients, or a retest of the PalmPilot-VFQ questionnaire to 45 patients. Performance times were compared between the three questionnaires. On the same visit in which the patient completed the PalmPilot-VFQ, the acuity was measured in both eyes, with their habitual correction using the Early Treatment Diabetic Retinopathy Study Chart16 recording the vision in the better-seeing eye (HCVA).

Exclusion criteria

Excluded were those individuals (1) who did not understand written English fluently as a first or second language; (2) who had had ocular surgery carried out within 3 months; (3) who were scheduled to have return ocular visits more than 3 months apart; (4) who were under the age of 18; (5) who manifested a neurological problem such as tremor or hand disability that was thought to prohibit their taking the survey questionnaires.

PalmPilot-VFQ

The intended population for use of the instrument was one that was undergoing retinal practice and was heavily populated with persons suffering a multiplicity of macular diseases. The development of the PalmPilot-VFQ represents a modification of the NEI-VQ-25 questions and answer scales for a person with limited, 6th grade education in order to enhance the clarity and understanding for self-administration. In addition, questions that were not task-specific (ie, those directed toward depression evaluation) were omitted in order to enhance reporting acceptance by clinicians. The PalmPilot-VFQ instructions, questions, and answer scales are presented in Table 1. The print size on the screen for instructions, questions, and answers for the PalmPilot-VFQ (Ariel bold, size 11) was the same as that for the self-NEI-VFQ-25. For those with sufficient vision loss that they could not read the print on the Palm Pilot screen or on the paper, self-NEI-VFQ-25, 6D magnification spectacles with 10D base-in prism were provided.

Table 1 PalmPilot-VFQ: Palm Pilot-Visual Functional Questionnaire
Full size table

Two different answering methods were required in the PalmPilot-VFQ. A 5-point scale (0–4) was used for the patient to answer 22 of the 24 questions. The first two questions were aimd at assessing both overall general health and general eyesight (0=very good, 1=good, 2=fair, 3=poor, and 4=very poor). Twenty questions regarding specific tasks that may have been affected by their vision were answered with an endpoint specific scale from 0 to 4: (0) No, my vision does not limit me, (1) A little—My vision limits me some, (2) Moderately—I struggle doing this alone; (3) A lot—I need extra help; (4) I can't do this due tobecause of my vision. All items were scored in the same direction and in the same units. Two driving questions were answered with ‘Yes’ or ‘No’. These questions provided the opportunity for truncated questioning about driving performance and were not evaluated in the Rasch analysis. The item numbers of vision-targeted subscales for both the interviewer-NEI-VFQ-25 and the PalmPilot-VFQ are presented in Table 2.

Table 2 Comparison of the PalmPilot-VFQ and NEI-VFQ-25 in the division into subscales according to the questioned tasks
Full size table

Statistics

The responses of the patients to the PalmPilot-VFQ were analysed for (1) testing performance, (2) Rasch analysis, (3) reliability, and (4) validity.

Rasch analysis

More recently Rasch analysis has been applied to disability measurement using questionnaires for both the validation and modification of existing scales or the development of new scales in many areas of medicine,17, 18 including measures of visual disability in the low-vision population.9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24

Rasch analysis was carried out (using Winsteps: ver 3.65.0, April 2008, Chicago, Illinois, USA25) for 19 of the 24 questions of the PalmPilot-VFQ (excluding the driving-related questions) for the purposes of optimization of category function, calculation of item-fit statistics, assessment of item targeting, and calculation of the separation indices.26

Outfit and infit mean-square (MNSQ) values were calculated. Values substantially <1.0 indicate dependency in the data; values substantially more than 1.0 indicate the presence of unexpected outliers (noise). Items with outfit or infit (MNSQ) values >1.3 are usually labelled as potential misfits to the Rasch model conditions and are considered for deletion from the assessed sequence.27 For both the outfit and infit (MNSQ) statistics, the ZSTD (the Z standardized mean of MNSQ±1SD) was calculated. The separation indices28 were then calculated to determine real person and model separations. Finally, a Rasch person/item map was plotted to determine whether the difficulty level of each of the questioned items was targeted appropriately to match the level of difficulty experienced by the population examined.29

Testability

Testability was assessed by examining descriptive statistics to assess the frequency with which this retinal clinic population (with a high prevalence of maculopathy) could answer the PalmPilot-VFQ, and the time required in comparison with both the interviewer-NEI-VFQ-25 and self-NEI-VFQ-25 (using one-way ANOVA).

Reliability

To assess the reliability of the PalmPilot-VFQ, the internal consistency, measured using between-item correlations to evaluate similarity, was assessed by calculating Cronbach's α coefficient.30, 31, 32 Test–retest reliability was assessed by calculating a two-way ANOVA random-effects model intraclass correlation coefficient (ICC)33, 34, 35 for repeated testing of the PalmPilot-VFQ in comparison with that of the interviewer-NEI-VFQ-25.

Validity

For the PalmPilot-VFQ, validity was determined using a number of methods. Firstly, content validity36, 37, 38 was achieved at the design stage by using a multidisciplinary approach to item generation. This was also in part confirmed by Cronbach's α coefficient. Concurrent validity assesses a person's current criterion status.37 If our questionnaire is to be a measure of perceived visual difficulty, then the person measure should be able to differentiate subjects on the basis of their global rating of vision. To evaluate this, the means of PalmPilot-VFQ scores were compared with the general vision question (the second question in the PalmPilot-VFQ) using a one-way ANOVA. Finally, to assess convergent validity, at the end of the PalmPilot-VFQ, two additional questions, termed visual analogue scale (VAS, Table 1) questions, were included, which were similar to the first two questions, but with the answers scored from 0 to 10, in which 0 was the worse possibility and 10 was the best possibility (reverse of the other scales). For convergent validity,37 the Pearson correlation coefficient (PCC) was calculated by comparing the mean of the PalmPilot-VFQ scores, including the VAS question scores (VAS-1 and VAS-2), with the mean of interviewer-NEI-VFQ-25 scores and with HCVA in the better-seeing eye.

Results

Descriptive statistics

The mean age of the 135 patients enrolled was 70.8 years (range 18–100 years), with 42% being female. The average time delay between the interviewer-NEI-VFQ-25 and the PalmPilot-VFQ was 28 days (range 7–90 days). Mean HCVA in the better-seeing eye ranged from 20/20 to 20/200, with 20% of cases being poorer than 20/64.

Comparisons between the PalmPilot-VFQ and the two NEI-VFQ-25's of the performance time and HCVA in the better-seeing eye are presented in Table 3. One hundred twenty-two (90%) patients were able to complete the PalmPilot-VFQ by themselves, of whom 108 (80% of the total) were able to do so without difficulty using their own reading spectacles. In this group, the HCVA in the better-seeing eye varied between 20/20 and 20/50 and the time required for the PalmPilot-VFQ averaged 2.55 min (range 1.45–4.00 min), compared with an average of 6.8 min (range 4–16 min) for the interviewer-NEI-VFQ-25 and 11.2 min (range 6–25 min) for the self-NEI-VFQ-25 (P<0.0001). An additional 10% (13 of the patients) could complete the PalmPilot-VFQ with the aid of the high-plus, base-in prism reading spectacles (HCVA range 20/64–20/200), and did so in 7.30 min on average (range 4–9 min). Among the 50 patients who carried out the self-NEI-VFQ-25, 5 (10%) were unable to read the questionnaire with their regular reading glasses but could complete the questionnaire with high-plus, base-in prism spectacles in an average time of 17 min (range 11–28 min) (HCVA range 20/64 to 20/200) (P<0.000). Thirteen (10%) of the 135 patients were unable to complete the PalmPilot-VFQ by themselves, and 5 out of 50 (10%) were unable to complete the self-NEI-VFQ-25 with the high-plus spectacles provided. In these cases, the HCVA was worse than 20/200 in the better-seeing eye.

Table 3 Comparisons of the performance time and the HCVA between PalmPilot-VFQ, interviewer-NEI-VFQ-25 and self-NEI-VFQ-25
Full size table

Rasch analysis

Rasch analysis was carried out for the 122 patients who were able to complete the PalmPilot-VFQ. Table 4 shows the item-fit statistics listed in order of decreasing visual ability required to perform each task. The values in the table are item logits that indicate the difference between the mean item measure for all 19 items and the item measure for each questioned task. The item measure corresponds to the visual ability required for that task and has the same sign as that of the item logit. If the task logit is positive, the required visual ability for that task is higher than the mean required visual ability for all the questioned tasks, and if the task logit is negative, the required visual ability for that item is less than the mean for all items. The most difficult task was menu reading and the easiest item was the ability to recognize facial expressions. The mean of MNSQ±1SD infit (ZSTD infit) for the PalmPilot-VFQ was 0.99±0.40, whereas the ZSTD outfit was 0.98±0.44, both close to the desired 1.00. The separation indexes, for both the real-person and model-person separation, were 3.79 and 4.29, respectively, indicating that the PalmPilot-VFQ was able to separate between two ability levels (ie, to separate those with distance-vision difficulties from those without). Figure 1 shows a Rasch mapping of patient ability versus item difficulty for the 19 task items of the PalmPilot-VFQ (excluding driving-related questions). The subscale values are listed from the top to the bottom in descending order of ability, with the patient frequency marked in Xs on the left and with item names on the right in ascending order of difficulty. Both persons and items appear along the same logit scale. In this data set, the items are, on the whole, too easy for the abilities of the patients, represented by the X's predominantly congregated lower on the graph, whereas the task items are located higher. Two items in the PalmPilot-VFQ, ‘general health’ and ‘kitchen activities’, had outfit mean squares of 2.60 and 1.53, respectively, which were outside of the optimum upper limit for the recommended 1.30 cut-off for outfit scores,27 indicating these two items as misfits. After removal of these items, Rasch analysis was repeated on the 17 remaining item tasks (Figure 2). All remaining items had outfit mean squares within the optimal range. The mean square±SD outfit scores were all found to be close to 1.00 and the separation indices, for both the real-person and model-person separations, were 3.23 and 4.01, respectively, indicating a good fit.

Table 4 19 Items included in the Palm Pilot-Visual Function Questionnaire with Rasch-fit statistics and item calibration
Full size table
Figure 1
figure 1

Patient ability/item difficulty map for the 19-item Palm-VFQ. To the left of the dashed line are the patients, represented by X, and on the right are the items denoted by their content. More able patients and more difficult items are near the bottom of the diagram, with less able patients and easier items near the top. M, mean; S, 1SD from the mean; T, 2SD from the mean.

Full size image
Figure 2
figure 2

Patient ability/item difficulty map for the 17-item Palm-VFQ after removing two items. To the left of the dashed line are the patients, represented by X, and on the right are the items denoted by their content. M, mean; S, 1SD from the mean; T, 2SD from the mean.

Full size image

Reliability

Internal consistency assessment of the 19 questioned items in the PalmPilot-VFQ yielded a Cronbach's α coefficient of 0.97, whereas the test–retest reliability, as assessed by the ICC, was 0.74. For the shortened 17-item version of the data set, Cronbach's α was 0.89, whereas the ICC was 0.79.

Validity

For assessment of the concurrent validity, the mean of each of the PalmPilot-VFQ scores was compared with that of the question about general vision (question 2), and a statistically significant association was observed between groups (P<0.0001). On examining the convergent validity, the PalmPilot-VFQ's mean score showed a strongly positive correlation with the corresponding interviewer-NEI-VFQ-25 mean score (r=0.79, P=0.0001) and with the HCVA in the better-seeing eye (r=0.614, P=0.0001), whereas a strong negative correlation was observed with the VAS-1 and VAS-2 scores (r=−0.509, P=0.0001, r=−0.822, P=0.0001, respectively).

Discussion

The purpose of this study is to report the testing results of a visual function questionnaire that was developed for self-administration on an interactive, computer-based, hand-held device with a high-contrast, illuminated screen and that could be easily carried out with easy point-and-click responses. The questions that were selected from the NEI-VFQ-25 for use in this instrument, we believe, were appropriately modified for use in a retinal practice office. This study showed that the questionnaire could be successfully administered to 90% of the patients undergoing an average retina practice with a wide range of ocular pathology and affected acuities (with 20% below 20/64 in the better-seeing eye), but in a fraction of the time required for administration of the NEI-VFQ-25's and without the difficulty of hiring, training and paying a professional to conduct the interview. Although we believe the answers are more straightforward, avoiding many of the filters and subconscious influences that such an interview process invokes, these differences could not be evaluated using the study design and the number of patients evaluated in the current study.

In comparison with the NEI-VFQ-25, the mean PalmPilot-VFQ scores showed a strongly positive correlation with the mean NEI-VFQ-25 scores. The PalmPilot-VFQ also showed a very strong, positive correlation with the HCVA in the better-seeing eye and a negative correlation with the VAS questions.

Rasch analysis showed that in the population tested, the most common and the most visually demanding was extended reading, similar to that reported in earlier studies. The least common and demanding was facial recognition, which was surprising as this is a frequent complaint in our clinical experience among patients with maculopathic vision loss. Rasch analysis also showed that both the long and reduced versions of the PalmPilot-VFQ seem to be reliable, producing consistent test–retest results (ICC=0.79), as well as being internally consistent (Cronbach's α=0.89) in the shortened version. Nearly all of the questions showed appropriate targeting between the item difficulty and the patient's ability, with the only exception being the question about kitchen activities, suggesting that this physical activity was less dependent on vision than the other questioned tasks. After administering the interviewer-NEI-VFQ-25, we realized that the answer to this question was strongly influenced by gender. The male patients most often reported that they hesitated to answer the question regarding their ability to perform kitchen-related activity because they did not perform the activity frequently during their daily life, whereas female patients showed no hesitation to answer the question. In response to this, the question may be removed and placed in the truncated questioning category, similar to the questions regarding driving. Alternatively, the format of the question could be changed to make it more easily answered, even if the patient does not know how to perform the task (eg, Would your vision limit your ability for cooking, and for looking under kitchen counters or in cabinets or closets in the kitchen?). As expected, the question regarding general health showed an excessive outfit score to the Rasch analysis, indicating a mismatch between general health and visual health or disability.

The Rasch analysis showed overall a misfit of 2 logits between the mean of the patient scores and the mean item score, indicating that the NEI-VFQ, from which the PalmPilot-VFQ was derived, overall is in general not a good instrument for evaluating populations similar to this study population, which was heavily populated with people having a spectrum of macular disease rather than those with severe vision impairment. This is surprising given that the NEI-VFQ instrument is commonly reported in pharmaceutical trials on such patients, but is expected given the number of more recent studies of Rasch analysis on the NEI-VFQ.24, 39 In order to pursue implementation of an instrument such as the PalmPilot-VFQ, we must necessarily include additional questions of more difficult vision tasks in order to reduce the misfit. However, providing such a device to practitioners who are witnessing a tremendous rise in macular disease as a cause of vision loss, we believe, is mandatory to empower those practitioners to better understand the vision problems of their patients and to provide more accurate assistance for their difficulties.