To determine the accuracy of a previously described Dutch clinical prediction rule for ambulation outcome in routine clinical practice.
Adult (⩾18 years) patients who were admitted to the Austin Hospital with a traumatic spinal cord injury between January 2006 and August 2014.
Data from medical records were extracted to determine the score of the Dutch clinical ambulation prediction rule proposed by van Middendorp et al. in 2011. A receiver-operating characteristics (ROC) curve was generated to investigate the performance of the prediction rule. Univariate analyses were performed to investigate which factors significantly influence ambulation after a traumatic spinal cord injury.
The area under the ROC curve (AUC) obtained during the current study (0.939, 95% confidence interval (CI) (0.892, 0.986)) was not significantly different from the AUC from the original Dutch clinical prediction model (0.956, 95% CI (0.936, 0.976)). Factors that were found to have a significant influence on ambulation outcome were time spent in the ICU, number of days hospitalised and injury severity. Age at injury initially showed a significant influence on ambulation however, this effect was not apparent after inclusion of the 24 patients who died due to the trauma (and therefore did not walk after their injuries).
The Dutch ambulation prediction rule performed similarly in routine clinical practice as in the original, controlled study environment in which it was developed. The potential effect of survival bias in the original model requires further investigation.
The initial primary goal for many patients after a traumatic spinal cord injury (SCI) is to regain the ability to walk1 to allow maximal independence.2 Despite the devastating effects of a traumatic SCI, no treatment is available that consistently results in major neurological or functional recovery.3, 4 Considering the importance placed on walking by patients, reliable prognosis of the chances of regaining walking are important. Moreover, this prognosis is important in counselling, discharge and rehabilitation planning5 and would greatly assist decisions about medical equipment and home renovations for mobility and accessibility.6 It is also very important to be able to assess prognosis accurately to avoid unrealistic expectations of recovery. Further, rehabilitation professionals require the most accurate prognostic information when deciding on interventions to facilitate the recovery of limb function.7 Therefore, understanding the factors that influence the outcome of walking after a traumatic SCI is needed.
Recovery after a traumatic SCI varies in rate and degree, but there are some clear discernible patterns across impairment groups.8 Compared with patients with an American Spinal Injury Association (ASIA) Impairment Scale (AIS) A or B SCI, patients with an AIS C or D injury have a greater potential to recover to a higher level of function, including the ability to walk.9 Also, patients with an AIS D SCI are more likely to be able to walk regardless of the injury level.10 In addition, younger patients (<65 years) have been reported to have a better potential to functionally recover from neurological improvements compared with older patients.11
The largest amount of recovery from an SCI occurs within the first 6 months post injury, with the rate of recovery typically plateauing after 12–18 months.12 As such, 1-year follow-up data are typically considered representative of long-term outcome in SCI research.13 A clinical prediction rule for ambulation outcomes after a traumatic SCI has been published by van Middendorp et al.3 This Dutch prediction rule is reported to provide a highly accurate and early prognosis of a patient’s ability to walk at 1 year post injury, using age and four neurological tests: motor scores of the quadriceps femoris (L3), gastrocsoleus (S1) muscles, and light touch sensation of dermatomes L3 and S1.3 However, the accuracy of this prediction rule has only been established within the original Dutch study that involved specifically trained neurologists and rehabilitation physicians. For the prediction tool to be implemented into widespread clinical practice, it is important to establish how well the tool works ‘in the real world’. As such, the purpose of this study was to investigate the accuracy of the Dutch clinical prediction rule for ambulation outcomes, using routinely collected clinical data.
Patients and methods
Study design and patient population
Data from neurological admission status and functional outcomes of patients with a traumatic SCI were collected prospectively and collated retrospectively. The data were extracted from the medical records of the patients who were admitted to the Austin Hospital with a traumatic SCI between January 2006 and August 2014.
All adult (⩾18 years) patients with a traumatic SCI, including conus medullaris and cauda equina injuries, were included in this study. Patients were excluded if they had a non-traumatic SCI, if they did not have a neurological assessment within 15 days post injury or a follow-up mobility status measure at 6 months or later post injury. These criteria were set to match the inclusion and exclusion criteria used by van Middendorp et al.3 as closely as possible. Approval for this audit was provided by the Austin Health Human Ethics Committee.
The neurological admission status was defined as the standard AIS (Table 1), which was assessed during physical examination of the patient by a doctor or physiotherapist.
Neurological examinations were conducted according to the International Standards for Neurological Classification of Spinal Cord Injuries (ISNCSCI).14 Examination included motor score testing (graded on a six-point scale adapted from the Medical Research Council scale), light touch sensory and pinprick sensory testing (0=absent, 1=impaired and 2=normal) and sacral sparing scores, including voluntary anal contraction and anal sensation (0=absent and 1=present).14
To be able to get an accurate estimation of the neurological status at admission to the hospital, only complete ISNCSCI assessments within 15 days after injury were included, as in the van Middendorp study.3 To predict ambulation 1 year after injury, the Dutch clinical prediction rule requires the AIS grade, the neurological level of injury, and the highest motor and sensory scores (left or right) for neurological levels L3 and S1.3
In addition, the categorisation of injury severity in this study was performed according to the guidelines of DeVivo et al.15 However, the category of injury levels T1–S5 with an AIS A, B or C was split into two as distinction in the thoracic and abdominal muscles could influence the ambulation outcome.
The primary functional outcome was defined as the ability to walk independently at 1 year post injury. Independently was defined as able to walk without supervision, but potentially with the usage of aids like a walking cane. Mobility was determined by the status at discharge, or later during outpatient appointments, by a doctor or physiotherapist working at the hospital. As per the study protocol of van Middendorp et al.3, 6-month follow-up measurements were used for patients without 1-year follow-up measurements.16
Descriptive statistics were calculated with absolute and relative frequencies for categorical variables, and mean, s.d., median and range were calculated for continuous variables.
After determining the outcome for every patient based on the information available in their discharge summary and medical record, the prediction rule score according to van Middendorp et al.3 was calculated.
Data were analysed using Stata12. A receiver-operating characteristics (ROC) curve was plotted to assess the area under the ROC curve (AUC). The performance of the Dutch prediction rule in discriminating between patients who can and cannot walk 1 year after injury was defined as the AUC. Two-sample t-tests and chi-squared tests were used to determine which factors influenced the outcome of ambulation 1 year after traumatic SCI.
Statistical significance was set at P<0.05.
Between January 2006 and August 2014, 373 adults were admitted to the Austin Hospital with a traumatic SCI. Of them, 291 had complete neurological examinations within the first 15 days post injury (78%). Ambulation outcome measurements were available for 184 patients (63%). The patient characteristics for these 184 included patients are described in Table 2.
A more extensive comparison of the characteristics of the patients who were included and those who were excluded is available online (Supplementary Table A1) and similarly a comparison of the patient characteristics between those with 1-year follow-up data, 6-month follow-up data and no available follow-up data (Supplementary Table A2). Briefly, excluded patients spent 78 days less in hospital compared with the included patients (110 days and 188 days, respectively). Also, 26% of the excluded patients had a missing initial neurological examination and for 22% it was unknown whether they had undergone spinal surgery, differences attributable to the study inclusion and exclusion criteria.
Analyses comparing the patients with a 1-year follow-up, a 6-month follow-up and a missing follow-up showed significant differences (P<0.05) in time hospitalised. Patients with a 6-month follow-up were in the hospital for the longest time, 199 days, compared with 182 days for patients with a 1-year follow-up and 102 days for no follow-up. A C5–C8, AIS A, B, C injury occurred in 31% of the patients with 1-year follow-up and in 32% of those with a 6-month follow-up, but in the patients without follow-up 30% had a T7–S5, AIS A, B, C injury. As above, these differences were considered a direct consequence of the study inclusion and exclusion criteria.
Scores were calculated using the Dutch prediction rule developed by van Middendorp et al.3 for all of the patients with available follow-up data. Figure 1 shows the frequencies of prediction rule scores for patients with SCI who did or did not walk independently. Higher scores are associated with a greater percentage chance of walking at 12 months post injury.
Figure 2 shows the ROC curve of the calculated prediction rule scores. The AUC was 0.939, 95% CI (0.892, 0.986).
Initial analyses suggested that patients who walked were significantly older, and spent less time in the ICU and hospital than those who did not (Table 3). However, 24 patients were not included in the analysis because they died (as per the van Middendorp paper). A sensitivity analysis was performed by including them in the group that does not walk and the difference in age was no longer significant (P=0.1889). There was no significant difference between those who did and those who did not walk in terms of gender or whether the patients had spinal surgery (Table 3). As expected, patients with more severe injuries were less likely to walk.
In this study the accuracy of the Dutch prediction rule developed by van Middendorp et al.3 was investigated when employed in usual clinical care. An ROC curve was generated to compare the performance of the Dutch ambulation prediction rule in clinical setting with that reported by van Middendorp et al. The AUC for the clinical setting was 0.939, 95% CI (0.892, 0.986), which is not significantly different from the AUC in the paper by van Middendorp et al. (0.956, 95% CI (0.936, 0.976)).
In the second part of this study, the factors that influenced the outcome of walking after a traumatic SCI were investigated. Gender and whether the patients had undergone spinal surgery did not significantly influence walking outcome. Patients who were able to walk had a less-severe injury on lower spinal levels (levels T7–S5, AIS D), had a shorter stay in the ICU and a shorter stay in the hospital (P<0.05).
These findings were expected given the relationships between the severity of the SCI and hospital stay: more severe injuries commonly result in longer ICU and hospital stays. Moreover, Kay et al.6 showed that patients with an AIS C injury were more likely to walk at the time of discharge compared with patients with an AIS A or B injury; patients with an AIS D injury were more likely to walk at discharge than those with an AIS C injury, and patients with an AIS D injury were all likely to walk at discharge regardless of their injury level.6 In the current study, all of the patients with an AIS D injury were able to walk independently 1 year after their injury.
Age at injury initially showed a significant influence, but this effect was in the opposite direction to that which might be expected and as was observed in the original paper. Patients who were able to walk again were older than those who did not walk, but age was no longer significantly different when the 24 patients who died before discharge to the community were included in the analysis. This difference in results between our findings and the van Middendorp model may reflect a selection bias in both the original article and the current sample. As discussed by van Middendorp, those who died in the derivation data set before discharge to the community were not included in the model development.
Prediction of walking outcomes are made clinically throughout the hospitalisation period as well as following discharge. A robust walking prediction tool must therefore not be limited in its application to those people who have survived to discharge to the community. In the current study, a more conservative approach to the data where we included all those who died in the ‘did not walk at 1 year’ group removed age as a significant discriminator of ambulation. It is thus possible that the factors ‘age’ and ‘died before 1 year’ are interacting in the model in some manner and perhaps limiting the clinical applicability of the model. As such, we recommend further research into the influence of age on ambulation outcomes.
Other factors that might influence the outcome of walking that were not investigated in this study are trauma mechanism and pre-existing comorbidity. Higher velocity trauma would likely be associated with more severe, higher and complete injuries that would require spinal surgery, a longer stay in the ICU and a longer stay in the hospital. Patients with an injury sustained in a higher velocity trauma could therefore be less likely to be able to walk again. Ambulation outcome may also be affected by comorbidity, including pre-existing myelomalacia and premorbid poor mobility.
In this study the external validation of the prediction rule of van Middendorp et al.3 was tested when used by doctors and/or physiotherapists in a normal clinical setting. In most cases the neurological examination was carried out by registrars with varying amounts of experience and training in performing the examination. More recently, these examinations were performed by the senior physiotherapist and the registrar together at the bedside. Performance of the Dutch prediction rule in clinical practice was comparable to the original study, despite it being completed by different health professionals with varying types and amounts of training. Importantly, the accuracy of the Dutch ambulation prediction rule was tested in only one hospital in Australia. Although all assessments are conducted in accordance with ISNCSCI guidelines, a limitation of this study is that we cannot exclude a site-specific cultural bias in the assessments.
The accuracy of the Dutch ambulation prediction rule developed by van Middendorp et al. was tested for use in clinical practice, and comparable performance of the prediction rule was found when used in a hospital spinal unit as part of routine clinical care. The potential effects of both survival bias and the construct validity of the original model require further investigation, although overall the prediction rule appears to function as designed in routine clinical practice.
There were no data to deposit.
Ditunno PL, Patrick M, Stineman M, Ditunno JF . Who wants to walk? Preferences for recovery after SCI: a longitudinal and cross-sectional study. Spinal Cord 2008; 46: 500–506.
Davies H . Hope as a coping strategy for the spinal cord injured individual. Axone 1993; 15: 40–46.
van Middendorp JJ, Hosman AJ, Donders AR, Pouw MH, Ditunno JF Jr, Curt A et al. A clinical prediction rule for ambulation outcomes after traumatic spinal cord injury: a longitudinal cohort study. Lancet 2011; 377: 1004–1010.
Tator CH . Review of treatment trials in human spinal cord injury: issues, difficulties, and recommendations. Neurosurgery 2006; 59: 957–982 discussion 82-7.
Kirshblum SC, Priebe MM, Ho CH, Scelza WM, Chiodo AE, Wuermser LA . Spinal cord injury medicine. 3. Rehabilitation phase after acute spinal cord injury. Arch Phys Med Rehabil 2007; 88: S62–S70.
Kay ED, Deutsch A, Wuermser LA . Predicting walking at discharge from inpatient rehabilitation after a traumatic spinal cord injury. Arch Phys Med Rehabil 2007; 88: 745–750.
Field-Fote EC, Tepavac D . Improved intralimb coordination in people with incomplete spinal cord injury following training with body weight support and electrical stimulation. Phys Ther 2002; 82: 707–715.
Kapadia N, Masani K, Catharine CB, Giangregorio LM, Hitzig SL, Richards K et al. A randomized trial of functional electrical stimulation for walking in incomplete spinal cord injury: ffects on walking competency. J Spinal Cord Med 2014; 37: 511–524.
Fawcett JW, Curt A, Steeves JD, Coleman WP, Tuszynski MH, Lammertse D et al. Guidelines for the conduct of clinical trials for spinal cord injury as developed by the ICCP panel: spontaneous recovery after spinal cord injury and statistical power needed for therapeutic clinical trials. Spinal Cord 2007; 45: 190–205.
Burns SP, Golding DG, Rolle WA Jr, Graziani V, Ditunno JF Jr . Recovery of ambulation in motor-incomplete tetraplegia. Arch Phys Med Rehabil 1997; 78: 1169–1172.
Furlan JC, Fehlings MG . The impact of age on mortality, impairment, and disability among adults with acute traumatic spinal cord injury. J Neurotrauma 2009; 26: 1707–1717.
Curt A, Van Hedel HJ, Klaus D, Dietz V, Group E-SS. Recovery from a spinal cord injury: significance of compensation, neural plasticity, and repair. J Neurotrauma 2008; 25: 677–685.
Kirshblum S, Millis S, McKinley W, Tulsky D . Late neurologic recovery after traumatic spinal cord injury. Arch Phys Med Rehabil 2004; 85: 1811–1817.
Kirshblum SC, Burns SP, Biering-Sorensen F, Donovan W, Graves DE, Jha A . International standards for neurological classification of spinal cord injury (revised 2011). J Spinal Cord Med 2011; 34: 535–546.
DeVivo MJ, Biering-Sorensen F, New P, Chen Y, International Spinal Cord Injury Data S. Standardization of data analysis and reporting of results from the International Spinal Cord Injury Core Data Set. Spinal Cord 2011; 49: 596–599.
Van Middendorp JJ, Hosman AJ, Pouw MH, Van de Meent H . Is determination between complete and incomplete traumatic spinal cord injury clinically relevant? Validation of the ASIA sacral sparing criteria in a prospective cohort of 432 patients. Spinal Cord 2009; 47: 809–816.
We thank the staff of the Victorian Spinal Cord Service for assisting in the data collection process.
The authors declare no conflict of interest.
Supplementary Information accompanies this paper on the Spinal Cord website
About this article
Cite this article
van Silfhout, L., Peters, A., Graco, M. et al. Validation of the Dutch clinical prediction rule for ambulation outcomes in an inpatient setting following traumatic spinal cord injury. Spinal Cord 54, 614–618 (2016). https://doi.org/10.1038/sc.2015.201
This article is cited by
Performance of the Dutch clinical prediction rule for the ambulation outcome after spinal cord injury in a middle-income country clinical setting: an external validation study in the Thai retrospective cohort
Spinal Cord (2023)
Neurocritical Care (2023)
Clinical judgment is a cornerstone for validating and using clinical prediction rules: a head-to-head study on ambulation outcomes for spinal cord injured patients
Spinal Cord (2021)
Prediction of independence in bowel function after spinal cord injury: validation of a logistic regression model
Spinal Cord (2021)
An arrow that missed the mark: a pediatric case report of remarkable neurologic improvement following penetrating spinal cord injury
Child's Nervous System (2021)