Introduction

Most assessments of mobility for people with a spinal cord injury (SCI) rely on standardised outcome measures that are devised and used by health-care professionals. These include outcome measures such as the Functional Independence Measure,1 Walking Index of Spinal Cord Injury2 and Clinical Outcome Variables Scale.3 These measures typically capture aspects of movement believed to be important by health-care professionals—such as speed, kinematics, need for assistance or use of assistive devices. However, traditional measures of mobility may not reflect the priorities of people with an SCI. For example, people with an SCI who live each day with limited mobility may place a high priority on the appearance of movement and little priority on speed or need for assistance, or vice versa.

One way to capture the perspectives of people with an SCI is through the Global Impression of Change Scale (GICS).4 The GICS is an anchored scale whereby the assessor (in this case, person with an SCI) makes a judgment about observed change in clinical status (in this case, mobility). There are variations of the GICS but the most widely used version is a 7-point scale where +1 depicts ‘almost the same—not much better’ and +7 depicts ‘a very great deal better’. The GICS is appealing because a rater can take into account many different aspects of change and can intuitively weight the respective importance of each within one score.5 The GICS has previously been used during clinical trials in gerontology,4, 5, 6 psychiatry,7 chronic pain8 and multiple sclerosis.9 It has also been utilised in an SCI research on neuropathic pain and pharmacological therapies.10, 11, 12 In addition, we have asked patients to rate their own performance of motor tasks using the GICS.13 However, it is not ideal to ask patients to rate themselves in clinical trials because they are not blinded and they may be influenced by expectations of treatment effectiveness. One way to overcome this problem is to ask independent people with an SCI to rate the change in performance of their peers from videos using the GICS. Those undertaking the ratings are not involved in the trial and can thus be kept blinded to group allocation, thereby minimising the systematic bias. Others have used a similar concept but have always relied on health-care professionals to provide the ratings.14, 15 We wanted to use people with an SCI to capture their perspectives. Therefore, the purpose of this study was to explore the potential of using people with an SCI to rate change in mobility from videos using the GICS. Specifically, we sought to determine whether people with a recent and established SCI can reliably rate change in motor performance of their peers and to determine what is their level of ratings compared with those done by the physiotherapists. We used two discrete groups of people with an SCI because we reasoned that the priorities of a person with a recent SCI may be different to the priorities of a person with an established SCI.

Materials and methods

The videos

Data used for this study were 51 pairs of videos taken on two different occasions. The data collection methodology is described in a previous study13 in which we asked patients to rate their own change in mobility without viewing videos. Their self ratings were compared with the ratings provided by two physiotherapists who did view videos. Both sets of ratings were also compared with the results of standardised outcome measures. In the current study, all ratings were performed by independent people with an SCI and physiotherapists after watching videos of performance. The people with an SCI who rated the videos were not the patients depicted in the videos.

In brief, short videos were recorded between 2008 and 2009 of 30 patients admitted to two Sydney Spinal Units for rehabilitation following a recent SCI. The videos of each patient were recorded on two occasions. The time lapse between the two occasions ranged from 1 h to 5 months. The pairs of videos depicted varying amounts of change in performance of three motor tasks: sitting unsupported (n=22), transferring (n=19) and walking (n=10). On three occasions, pairs of videos were taken on the same day. This was done to ensure that at least one pair of videos for each mobility task depicted no (or very little) change in motor performance. Care was taken to ensure that there were no cues provided to the raters that some pairs of videos were taken on the same day, and thus the following aspects of each video taken on the same day were changed: patients’ clothing, camera angle, the physical environment and the therapist providing assistance or supervision.

The pairs of videos were assembled using a software package that allowed the two videos to be presented side by side and allowed raters to see and play each pair of videos together or separately on the one screen. The position of the videos on the screen was randomised so that sometimes the video depicting the earlier performance was on the left-hand side of the screen and sometimes on the right-hand side of the screen. The raters were not told which video was which, or how much time had elapsed between each recording. Instead, they were simply asked to state whether the performance between the two videos was the same or different. If different, they were then asked to identify which video (right or left) depicted superior performance and to rate the difference in performance of the two videos on a 7-point scale ranging from 1 (‘almost the same—not much better’) to 7 (‘a very great deal better’).

The rating of the videos

Raters were given no information about the people in the videos other than that they had recently sustained an SCI and were receiving inpatient rehabilitation. They were not told the extent or type of therapy the patients received. The instructions to the raters were as follows: ‘use your judgment to score. Take into account all relevant factors evident to you (for example, weight, age, spasticity, neurological status etc.) including any neurological change.’ A research assistant supervised the ratings provided by people with an SCI, primarily to assist those with poor hand function and to ensure that the raters were diligent and could navigate the software.

The 51 pairs of videos for each of the three mobility tasks were rated by 20 people with an SCI and 10 physiotherapists (that is, a total of 30 ratings of 51 videos=1530 ratings). The raters were 10 people with a recent SCI, 10 people with an established SCI and 10 physiotherapists. The people with an SCI only rated videos of motor tasks relevant to them. For example, a person with extensive paralysis and unable to walk was not asked to rate the walking videos, and a person walking who could readily sit unsupported was not asked to rate the sitting videos. This was done to ensure that those rating the videos had some understanding and first-hand experience of the difficulties required to perform the motor tasks they were rating. Therefore, pools of 18 people with a recent SCI and 16 people with an established SCI were used. Of these, only 10 people with recent and 10 people with an established SCI rated each of the three mobility skills. The same 10 people with an established SCI rated the sitting unsupported and transferring videos. Care was taken when recruiting raters to ensure that they were representative of their respective populations as outlined in the following paragraphs.

The people with a recent SCI who rated the videos

In all,18 people with a recent SCI were prospectively recruited as admitted to a Sydney SCI Unit between April 2011 and April 2012. The mean (s.d.) time since injury was 4.8 months (3.1). The inclusion criteria were a recent SCI and currently participating in therapy to improve ability to sit unsupported, transfer or walk. People were excluded if they were unable to speak English or were suffering from any psychiatric condition that prevented them from cooperating. People were also excluded if they did not live in the Sydney metropolitan area in order to match this sample with the sample of people with an established SCI (only people with an established SCI who lived in the Sydney metropolitan area were eligible for inclusion because of travel constraints). Therefore, 66 people were screened for inclusion, 2 people declined to be involved, 40 were outside the Sydney metropolitan area and 6 people were excluded because they had not sustained a recent injury (n=3), had a psychiatric illness (n=1) or were not receiving therapy directed at any of the three mobility tasks (n=2).

The people with an established SCI who rated the videos

In all, 16 people with an established SCI were prospectively recruited from a database of consecutive admissions to a Sydney SCI unit between July 2000 and June 2005. The mean (s.d.) time since injury was 9.2 years (1.2). The inclusion and exclusion criterion were the same as for people with a recent SCI; however, instead of currently participating in therapy to improve ability to sit unsupported, transfer or walk, they needed to have participated in these therapies at the time of initial rehabilitation. In all, 224 people were screened for inclusion. Of these, three people declined to be involved and 205 people were excluded for the following reasons: living outside Sydney metropolitan region (109); relocated (4), deceased (5); unable to be contacted (62); not eligible (4); no reason provided (10); previously participated in similar research project (10); and recruited but then did not rate (1).

The physiotherapists who rated the videos

In all, 10 physiotherapists were recruited from a database of 130 Australian and New Zealand SCI physiotherapists. This database contains almost all physiotherapists working in the area of SCI across Australia and New Zealand. Physiotherapists were eligible for inclusion if they were currently working in SCI and had more than 5 years post graduate experience. Physiotherapists working in one of Sydney SCI units where the data were collected were excluded. Ultimately, 72 physiotherapists were randomly selected from the database to attain the required 10. The reasons for excluding the remaining 62 physiotherapists were as follows: contact details were incorrect (24); did not respond to invitation to participate (24); currently abroad (6); on maternity leave (3); and declined to be involved (5).

Statistical Analyses

All data were analysed using Stata V11.1 (StataCorp, College Station, TX, USA). The analyses were performed in two ways, depending on the purpose of the analysis. For example, to describe the ratings, the data were expressed with respect to the video recorded on the second occasion. Thus, if the video depicting the second occasion was presented to raters on the left of the screen but the rater identified the video on the right of the screen as superior, the rater’s score was flipped from positive to negative. In contrast, for the analyses of reliability, the data were expressed with respect to the video presented on the right of the screen regardless of whether this was the video recorded on the first or second occasion. Nine intra-class correlation coefficients (95% confidence interval (CI)) were calculated for the three groups of raters and the three mobility tasks.

All institutional and governmental regulations concerning the ethical use of human volunteers were followed during the course of this research.

Results

The median (interquartile range and s.e.m.) ratings for the pairs of videos and the nine intra-class correlation coefficients (95% CI) for the three groups of raters and the three mobility tasks are shown in Table 1. The median ratings of change ranged from 2 to 5, with raters generally scoring the transferring and walking videos higher than the sitting unsupported videos. The corresponding interquartile ranges spanned from −1 to 6. The negative values indicated that some raters believed that the video recorded on the first occasion reflected superior performance to the video recorded on the second occasion. That is, unbeknown to them, they were scoring deterioration in performance over time. The s.e.m. ranged from 0.2 to 0.3 reflecting the minimum score required at an individual level to be confident that the change was real and not error.

Table 1 Median (IQR range; s.e.m.) ratings of change and intra-class correlation coefficient (95% CI) for the ratings of pairs of videos by people with a recent SCI (n=10), people with an established SCI (n=10) and physiotherapists (n=10) with 0 being ‘no change’ and +7 being ‘a very great deal better’
Full size table

The ICC for the three sets of raters and the three mobility tasks ranged from 0.24 to 0.96. Generally, rater reliability was better for scoring transferring (ICCs were 0.76, 0.66 and 0.89) and walking (ICCs were 0.79, 0.81 and 0.96) than for sitting unsupported (ICCs were 0.45, 0.24 and 0.65) where the ICCs represent people with a recent SCI, people with an established SCI and physiotherapists, respectively. There was little difference between people with an established SCI and people with a recent SCI for all three mobility tasks. Physiotherapists were consistently but only marginally more reliable at rating than people with an SCI.

Discussion

The aim of this study was to devise a way of capturing the unbiased perspectives of people living with an SCI for assessments of mobility, and the novel aspect was that we asked people with an SCI to rate impressions of change in the mobility of their peers from videos. This study is an extension of previous work we have done in this area, but, instead of relying on patients to rate themselves from recall,13 we asked independent people with an SCI to rate their peers from videos. The benefit of relying on videos is that this methodology can be used in clinical trials whereby independent raters who are blinded to group allocation can be used. This has advantages over the traditional ways of using the GICS and how we have used the GICS before, where participants or investigators of trials are asked to rate change. There are problems with asking participants in trials to rate change in mobility because they tend to forget how much they or their patients have changed over time and they can be biased by knowledge and expectations of the intervention.16

The results of this study indicate that people with an SCI are acceptably reliable at rating change in mobility from videos and that this methodology is worthy of further investigation. The reliability of the ratings provided by people with an SCI was not as consistent as the reliability of physiotherapists, but was in the range generally considered ‘good’,17 especially when rating transferring and walking. The differences in reliability between the people with an SCI and physiotherapists may reflect the training of physiotherapists, trained to observe subtle components of movement patterns, or may reflect the varying focus of the different groups of raters. People with an SCI may differ among themselves as to what they prioritise as important when looking at videos of performance. In contrast, physiotherapists may be more likely to consistently look at one or two aspects of movement. This factor alone, regardless of differences in education and training, could explain the better reliability of the physiotherapists.

Both people with an SCI and the physiotherapists were less reliable at rating videos of sitting unsupported than rating videos of transferring and walking. There are two possible explanations for this. First, the reduced reliability for sitting unsupported may be because the patients in the videos were reasonably good at sitting unsupported and demonstrated little change over time thus reducing score variability. This may have made it more difficult for the raters to detect change and reduced the range of scores used by raters (reliability is reduced when scores are restricted). Second, the reduced reliability may be because people with an SCI and physiotherapists are more familiar with transferring and walking typically devoting more time and attention to these two motor tasks. The limited reliability of rating sitting unsupported does not undermine the potential value of using this outcome within a clinical trial, although it does indicate the need for an increased sample size to attain precise estimates of treatment effectiveness.

Some may query our decision not to train the people with an SCI to rate. Clearly, people with an SCI could be trained to rate like any health-care professional. The reliability of their ratings could be increased if they were trained, especially if given a criterion to rate against. However, training people with an SCI to rate undermines the construct of the measure5 and hence defeats the purpose of using people with an SCI as naive raters. The strength of the measure lies in its inherent ability to capture the perspectives of people with an SCI without prompts from health-care professionals about what is important. Our results indicate that using people with an SCI is only associated with a small compromise in reliability compared with using physiotherapists, although people with an SCI may be observing and rating different aspects of movement than physiotherapists.

Another strength of the measure is its inherent ability to capture clinically important change. The GICS ‘is not intended as a sensitive measure of small changes that may not be clinically significant (page 2784).’ Rather it is intended to only capture clear change that people with an SCI would consider clinically meaningful. For this reason, even a one-point change is ‘considered clinically significant by definition (page 2784).’ This methodology may therefore bypass the need to define minimally clinically important differences18 for the more traditional mobility outcomes used in trials. Instead, people living with an SCI and limited mobility can be the judges of whether the observed changes in mobility are meaningful after taking into account the time, cost and inconvenience of the intervention.19 Some may dismiss the concept of relying on untrained and naive observers using a crude measure to determine meaningful change for their peers. However, we argue that naive observers who live with an SCI and limited mobility are good judges of meaningful change in mobility of their peers despite their lack of training and despite the simplicity and crudity of the measure.

The results of this study do not provide sufficient data to justify the widespread use of the GICS by people with SCI in clinical trials. There are still many issues which need to be resolved and investigated. The study does, however, for the first time provide details about a promising methodology for better capturing the perspectives of people with an SCI for mobility-related clinical trials.

DATA ARCHIVING

There were no data to deposit.