Introduction

The International Spinal Cord Injury (SCI) Upper Extremity Basic Data Set (ISCI-UE) [1] is one of several international SCI data sets developed to promote standardized data collection for SCI patients, which is aimed at improving SCI examination, treatment, and rehabilitation and comparison among SCI centers for research. Upper-extremity function is critical for individuals with tetraplegia. Based on previous studies [2,3,4], evidence suggests that regaining even partial function can lead to greater independence and enhancement in the quality of life (QoL). A survey of the needs of individuals with SCI indicates that improvement of hand function was the highest priority for people with tetraplegia compared to other needs in improving their QoL [3]. In this respect, adequate therapy of the upper extremities is vital for individuals with tetraplegia, and relevant for the improvement of upper-extremity function.

Upper-extremity function can be assessed through two main methods: basic manual muscle testing in accordance with the International Standards for Neurological Classification of SCI (ISNCSCI) [5] and using novel assessment tools developed to evaluate the efficacy of therapies and surgery for restoring hand function. These novel assessments focus on primary hand functions (i.e., lateral pinch, pulp pinch, and palmar grasp) in individuals with tetraplegia and other patient populations. These instruments can be classified as assessment area according to International Classification of Functioning, Disability, and Health (ICF) domain. The tests that assess body function and structure domain are six-minute arm test (6-MAT), graded refined assessment of strength, sensibility, and prehension (GRASSP), and handheld myometer. The activity domain should be assessed by box and block test (BBT), capabilities of upper-extremity instrument (CUE), Jebsen hand function test (JHFT), modified functional reach test (mFRT), Sollerman hand function test (SHFT), tetraplegia hand activity questionnaire (THAQ), and Van Lieshout test-short version (VLT-SV). The information regarding how to select these measurements can be assessed on the following websites: Academy of Neurologic Physical Therapy [6], National Institute of Neurological Disorders and Stroke (NINDS) common data elements [7], and Spinal Cord Injury Research Evidence (SCIRE) [8].

The ISCI-UE was developed for standardized data collection and reporting of a minimal amount of information about upper-extremity status in individuals with SCI. However, it is important to emphasize that the basic data sets are not intended to be used as measures [9]. The data sets represent the result of important methods used to systematically record highly relevant clinical information at the point of care, and the collection and recording must be consistent across practitioners in order to pool data or compare outcomes of care across facilities or programs.

Fin Biering-Sørensen and an international working group developed the ISCI-UE and adjusted for appropriateness according to the comment from the members of the Executive Committee of the International SCI Standards and Data Sets and the members of the International Spinal Cord Society (ISCoS) Executive and Scientific Committees and American Spinal Injury Association (ASIA) board members. Following this, the international organizations and societies (around 40) and individuals were invited to review the data set. The data set was posted on the ISCoS and ASIA websites to allow for further comments and suggestions. The data set was then reviewed again and approved by the members of the ISCoS Executive and Scientific Committees and ASIA board members. The data set was then used to collect information on the training cases to confirm its appropriateness for clinical use [1, 10, 11]. The objective of the data set is to establish terminology that is commonly understood among health-care providers for evaluating the hand and upper-extremity function of individuals with SCI. The following minimal data elements were required for the evaluation of upper-extremity function of individuals with SCI: date performed, the ability to reach and grasp, shoulder function classification, and use of assistive devices. In addition, SCI-related complications affecting upper-extremity function like pain, spasms, contractures and edema, and upper extremity/hand reconstructive surgery were evaluated.

Using the ISCI-UE in a clinical setting requires additional work for data set evaluation, according to the recommendations of the Executive Committee of the International SCI Standards and Data Sets [9]. For all international data sets, reliability is a crucial element that allows for a meaningful comparison of data from different resources. The aim of this study was to determine the inter-rater reliability of the ISCI-UE data set.

Methods

Participants

Individuals with tetraplegia who received treatment at Maharaj Nakorn Chiang Mai Hospital, Chiang Mai, Thailand, from July 2015 to July 2016 were enrolled in the study. The inclusion criteria were as follows: (1) age—18–65 years and (2) neurological level—T1 or above. The exclusion criteria were as follows: (1) concomitant upper-extremity injury at the time of injury (e.g., brachial plexus injury, fracture, and amputation), (2) diseases or injuries of upper extremities prior to the onset of SCI, and (3) history of severe traumatic brain injury with inconsistent orientation to person, time, and place (Rancho Los Amigos scale <7). Individuals who agreed to participate in the study provided informed consent.

Measures

The International SCI Core Data Set was used to record descriptive characteristics, which included the following: age, gender, time post injury, cause of injury, and neurologic status [12]. Neurological examinations were performed by a physician trained in the use of International Standards for Neurological Classification (ISNCSCI). The core data set and ISNCSCI were collected by the same physician. The ISCI-UE for each patient was completed by two additional raters.

The ISCI-UE was used to assess the hand and upper-extremity function in the SCI population. The data set was composed of five variables: ability to reach and grasp, shoulder function classification, use of assistive devices, SCI-related complications affecting upper-extremity function, and upper extremity/hand reconstructive surgery. The ability to reach and grasp was classified into five levels (1–5). Shoulder function was classified into four levels (A–D). Scores from the ability to reach and grasp (numerical values from 1 to 5) were coupled with scores from shoulder function classification (letter values from A to D), and this defined the entire upper-extremity function [1, 10]. There were 20 patterns of the score (1–5 plus A–D). For example, a patient with central cord syndrome had right/left ability to reach and grasp scores 3/3 and right/left shoulder function scores A/B, then this patient had basic right hand upper-extremity function 3A and basic left hand upper-extremity function 3B. The third section (use of assistive devices) includes devices used to enhance upper-extremity function. Complications affecting upper-extremity function are classified as minimal, moderate, or extensive. Finally, upper extremity/hand reconstructive surgery is included if it was specifically performed to restore arm and/or hand function. These last three domains have 4, 3, and 3 response items [1, 10]. The data sheet and the syllabus are available on the ISCoS website (www.iscos.org.uk).

Raters

The raters included a physiatrist experienced in SCI and a physician training in rehabilitation medicine. After reviewing the ISCI-UE data sheet and syllabus, the raters were standardized by scoring the training cases of the ISCI-UE data set [13]. Then, a discussion was conducted between the four investigators (two experienced SCI physiatrists and two physicians). Any difference of scoring and comments in each variable were discussed. Both raters’ scores were nearly the same; therefore, this result ensures the consistency of raters in data collection. Raters performed separate assessments of the same SCI participant. The time interval between repeat ratings is less than 1 day in order to prevent changes to the attributes under examination. Also, raters were blinded against each other’s ratings.

Statistical analysis

The descriptive statistics, including frequency, percentage, mean, standard deviation, median, and interquartile range, were used to present the following demographic data: age, gender, time post injury, severity of SCI, causes of injury, level of ability to reach and grasp, and level of shoulder function.

The analytical methods utilized in this study were modeled from reliability studies on the musculoskeletal and bowel function data sets, and included percentage agreement and Cohen’s kappa for assessing reliability [9, 14,15,16,17]. Quadratic weighted kappa (κ) was used to determine reliability because it provided an estimation of agreement corrected for chance [17]. The κ values were interpreted based on Landis and Koch, where a score ≤0 = poor, 0.01–0.20 = slight, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial, and 0.81–1.0 = almost perfect agreement [16, 17]. The authors gave a positive rating for reliability when κ ≥ 0.70 in a sample size of at least 50 patients [9, 18]. Percentage agreements with an a priori minimum of 90% for variables with two categories and 80% for variables with three or more categories were considered satisfactory [9]. Percentage agreement values were used if a high prevalence existed in some variables and none in the extreme variables. For instance, kappa may inaccurately reflect agreement. It may be more appropriate to consider the percentage of agreement to supplement the kappa coefficients such as sub-analysis of the ability to reach and grasp, shoulder function classification, and whole upper-extremity function. Data collection and data management were conducted by another researcher. Statistical analysis was conducted using SPSS 22.0 for Windows.

Results

Demographics

The two raters separately classified 60 individuals with SCI from traumatic and non-traumatic causes. The characteristics of participants are listed in Table 1. A total of 52 (87%) individuals were male, with a mean (SD) age of 42.9 (14.3) years, and median (interquartile range) time since injury of 9.5 (1–53) months. The two most common causes of injury were transport-related activities (55%) and falls (33%). The distribution on the ASIA impairment scale (AIS) was as follows: A, 22%; B, 12%; C, 18%; D, 45%; and E, 3%. One-third of the 60 injuries (34%) had a neurological level of C4.

Table 1 Characteristics of the study group
Full size table

Of the 120 upper extremities assessed, three samples were excluded due to concomitant upper-extremity injury at the time of injury as follows: elbow dislocation, anterior interosseous nerve injury, and amputation. Thus, a total of 117 upper extremities were analyzed. No sampled patients underwent upper-extremity surgery for the improvement of arm and/or hand function.

Inter-rater agreement

Ability to reach and grasp

The frequency of the level of hand–arm function (ability to reach and grasp) rate, as assessed by the two raters, is presented in Table 2. The percentage of agreement in the ability to reach and grasp was 91.5, and the weighted kappa coefficient was almost in perfect agreement at 0.98 (95% CI, 0.96–0.99).

Table 2 The frequency of the ability to reach and grasp scores from rater one and two
Full size table

The ability to reach and grasp was also analyzed by category by the AIS grade and neurological level. The percentage agreement was acceptable in all AIS subgroups as follows: A, 85%; B, 93%; C, 82%; D, 90%; and E, 100%. Likewise, a positive percentage agreement existed when categories with neurological level were high (C1–C4), 89%; mid (C5–C6), 92%; and low (C7–T1), 94% cervical levels.

Shoulder function classification

The frequency of the level of shoulder function rate, as assessed by the two raters, is presented in Table 3. The percentage of agreement in the level of shoulder function was 93.2, and the weighted kappa was almost in perfect agreement at 0.97 (95% CI, 0.94–0.99). After performing sub-analysis, the level of agreement between raters was not different. The percentage agreements of AIS subgroups were A, 81%; B, 86%; C, 91%; D, 96%; and E, 100%. The percentage agreements, as classified by neurological level, were as follows: high (C1–C4), 94%; mid (C5–C6), 94%; and low (C7–T1), 89% cervical levels.

Table 3 The frequency of shoulder function scores from rater one and two
Full size table

For whole upper-extremity function, a percentage agreement within one point was acceptable (84%).

Use of assistive devices

The percentage agreement in the use of assistive devices was acceptable (93%) and the weighted kappa was very good 0.89 (95% CI, 0.77–1.00).

Complications affecting upper-extremity function (e.g., pain, spasm, contractures, and edema)

SCI-related complications also exhibited an acceptable percentage agreement (89%). Weighted kappa was substantial at 0.74 (95% CI, 0.52–0.96). Also, there was a high prevalence of no complications or complications having only minimal impact on function.

Upper extremity/hand reconstructive surgery

Since no patients underwent upper-limb reconstructive surgery, the percentage agreement was 100%.

Discussion

To the authors’ knowledge, the present study represents the first study evaluating inter-rater reliability of the ISCI-UE. The results provide strong evidence to support the ISCI-UE having satisfactory inter-rater reliability for the ability to reach and grasp as well as shoulder function. The percentage agreement for whole upper-extremity function was also acceptable (>80%). Inter-rater reliability for the use of assistive devices and complications affecting upper-extremity function exhibited almost perfect agreement and substantial agreement, respectively.

A particular strength of this study was the large number of cases included, which was due to the tertiary care center serving a large proportion of Northern Thailand. A comparison of the raters, each having different degrees of SCI experience (resident and SCI attending), was also carried out. Interpretation of the results of this study did not yield any significant difference in reliability between the raters, indicating that the data set was easy to understand and evaluate. When introducing a new measurement in clinical practice, the minimum comprehensive validation for inter-rater reliability should be investigated. In order to test the validity of this data set, content validity testing was conducted during data set development. However, the construct validity of the ability to reach and grasp, as well as shoulder function classification in ISCI-UE has not been studied. Although the ability to reach and grasp, which was adopted from RULER SCI (RITZ upper-limb evidence-based rehabilitation in SCI), had been benchmarked with a spinal cord independence measure (SCIM), the correlation was not defined [19, 20].

The inter-rater reliability of this study was acceptable and correlated with the inter-rater reliability tests of other SCI basic data sets. For example, 9 out of 12 items on the International SCI Musculoskeletal Basic Data Set scored satisfactory for inter-rater reliability (κ = 0.62–1.0 and percentage agreement = 75–100%) [14]. Regarding the agreement of the International SCI Bowel Function Basic and Extended Data Sets, 47 of 52 items were confirmed between versions from both Italy and Denmark (κ > 0.2) [15], while the International SCI Spinal Column Injury Basic Data Set (κ = 0.79–0.89) [21] and the Korea version of the International SCI Basic Pain Data Set (κ = 0.693–0.780 and intraclass correlation = 0.731–0.858) were also in agreement with the version from these nations [22].

The benefits of the ISCI-UE include no requirement of equipment, no copyright, and no fee for its use; therefore, it is suitable for clinical use. However, this data set may not be sensitive enough for minimal clinical changes due to the limited data requirement for hand function evaluation.

Some issues were experienced during the analysis of this data set. First, the pathology was not in the order of the classification; for example, in the case of a tetraplegic patient who had finger flexor function without wrist extensor function. Such an individual should be scored for the ability to reach and grasp group 4, as the definition of this group is voluntary control of wrist and some extrinsic hand muscles. Another issue was encountered while evaluating chronic SCI patients with severe spasticity or joint contracture, one rater scored less than the true level. However, this data set also involved complications affecting upper-extremity function, therefore, assessors should address during this section.

One limitation of this study is that it did not involve evaluation of intra-rater reliability. However, if the inter-rater reliability is strong, intra-rater reliability is also likely to be strong [9]. The construct validity of the ability to reach and grasp, as well as shoulder function classification has not been tested; therefore, it should be studied in the future. Future studies could also involve an investigation of the inter-rater reliability between other SCI health-care providers, such as physical therapists and occupational therapists, to assess and determine their reliability.

The present study concludes that the inter-rater reliability of the International Spinal Cord Injury Upper Extremity Basic Data Set is very high. However, further studies must be conducted to assess the construct validity of the data set before it is applied in research and clinical use.