Introduction

Traumatic spinal cord injury (t-SCI) is an impairing neurological condition with a devastating impact for patients and their families [1]. T-SCI is defined as damage to the spinal cord, caused by an external physical impact resulting in a temporary or permanent neurological deficit [2, 3]. The estimated incidence of t-SCI in the Netherlands is 11.7 per million residents per year, which are about 200 new cases of t-SCI each year [4, 5].

The adagium ‘time is spine’ has been a central theme in the management of t-SCI, assuming there is a critical time window after the primary injury during which the severe effects of the secondary injury mechanisms should be diverted [6]. The management of patients with t-SCI is complex and involves multiple stages of care, which often continues for years after the initial injury. As the incidence of t-SCI is low and the time factor is very important in the acute care, specialized Acute Spinal Cord Injury (ASCI) Units (in Dutch: Acute Ruggenmergletsel Unit, ARU) have been implemented. ASCI units were first developed in 1944 in England and have been implemented in North America [7]. The ARU of Nijmegen was founded in 2015 with the aim to improve the quality of care in the acute and the primary rehabilitation phase of the patient journey of t-SCI, which is in accordance with the aim of ASCI units. These ASCI units enable people with t-SCI to receive the necessary expertise and care immediately after the injury [8]. The ARU of Nijmegen is the first in the Netherlands to provide the full cycle of care in a regional network. Although many studies have been conducted to evaluate new interventions for t-SCI, it appears that little is known about standardization of care in t-SCI. Several studies have described the status of SCI care in other countries and a systematic review by Maharaj mentioned a lack of standardization within ASCI units on a global scale, with significantly different outcomes reported across published studies [9,10,11]. In a recent study by Fransen et al. the authors concluded that a large practice variance exists in pre-hospital and acute t-SCI management among ASCI-units and level 1 trauma centers in the Netherlands [12]. In addition, a study by Nijendijk et al. showed that a significant proportion of young survivors with t-SCI are not referred to specialized rehabilitation centers [4]. On the other hand, the population of elderly patients with SCI has increased and the majority is being referred to nursing homes that are not specialized in SCI.

To gain insight and improve the quality of care, several (inter)national databases exist in which data about t-SCI are collected. For example, the European Multicenter Study about Spinal Cord Injury (EMSCI) database supports a network consisting of 19 European SCI-centers [13]. In this network, systematic and standardized data are collected from patients with acute t-SCI. The core data set of The International Spinal Cord Society (ISCoS Core Data Set) contains recommendations for uniform reporting of basic data regarding SCI to facilitate accurate descriptions of patient populations and for globally meaningful comparisons of results between published studies [14,15,16]. The EMSCI database and the Dutch SCI Database (Nederlandse Dataset Dwarslaesie; NDD) [17] – which is derived from the ISCoS Core Data – are the data sources currently being collected in the Netherlands, covering several separate domains of SCI. However, these data sets do not collect interdisciplinary and collective measurements, derived from different phases in t-SCI treatment. Therefore, there is no uniformity to monitor and evaluate the quality in the full cycle of care of the “patient’s journey”, based on the three different domains (Body Functions and Body Structure, Activity and Participation) of the International Classification of Functioning, Disability and Health [18].

Systematic and continuous outcome monitoring by harmonization and standardization of treatment outcomes is important for assessing the value of provided care [19]. Case-mix adjustment, a statistical process that aims to account for differences in the mix of patient’s attributes across definitive patient cohorts, is required to make comparisons between hospitals with different patient populations [20]. Routine use of patient-reported outcome measures (PROMs) are valuable tools for patients and clinicians for (shared) decision-making, and enable comparisons of providers’ performance facilitating quality improvement [21, 22]. A patient outcomes registry supports outcome monitoring and is expected to contribute to the improvement of the quality of care and contribute to a better understanding of practice variation and treatment outcomes [23]. The data can be used to describe care patterns, including suitability of care and inequalities in care provision. A standard set of outcomes that follows the patient journey in the full cycle of care, from the acute phase of the injury until the discharge after rehabilitation phase, does not exist in the Netherlands. Therefore, the purpose of this study is to develop a standardized (network) outcomes set for t-SCI based on existing data sets, which includes patient-relevant outcome domains, adequate measurement instruments, and case-mix variables and risk factors based on the three domains of the ICF [18]. This standardized outcomes set is to be implemented in our ASCI Unit (ARU Nijmegen) and to be implemented for use nationwide.

Methods

Design

A modified Delphi study was performed, which consisted of a preparatory stage in which a preliminary standard network outcome set, including modifying case-mix and risk factors was developed, and three formal consensus rounds (modified Delphi study). These Delphi rounds consisted of two online surveys and one (final) online face-to-face meeting. This study was registered in the Core Outcome Measures in Effectiveness Trials (COMET) database (ID 1703) [24] and guidelines for the development of a core set of outcomes were applied [25]. The recommendations from the Guidance on Conducting and Reporting Delphi Studies (CREDES) [26] and the Core Outcome Set standards for reporting (COS-STAR Statement) were used to report the study findings [27]. The study was performed between March 2019 (study preparations) and November 2020 (data synthesis and reporting).

Project team

The study was performed by an independent project team, not participating in the Delphi study, consisting of a spine surgeon (MS), a rehabilitation physician (MVvdH), a methodologist (MvH) and a research assistant (TvS). Members of the project team work at the Radboud University Hospital and the Sint Maartenskliniek (rehabilitation center). The project team was responsible for the design and conduct of the full study. They performed the day-to-day activities, design, and key aspects of the study. An independent moderator was added to the project team to lead the third (online) face-to-face meeting.

Expert panel

The expert panel consisted of 19 members from ARU t-SCI network in Nijmegen, The Netherlands, consisting of an academic hospital and a rehabilitation center. Health care professionals involved in the full cycle of care in t-SCI with at least three years of experience were included. The multidisciplinary panel consisted of orthopedic surgeons (n = 3), rehabilitation physicians (n = 3), neurosurgeons (n = 2), physiotherapists (n = 4), occupational therapists (n = 2), specialized nurses (n = 3), a psychologist (n = 1), and a physician assistant (n = 1), with an average of 14 years (range 3–35) of experience in the treatment of traumatic spinal cord injury.

Preparatory stage - preliminary outcome set

The project team drafted a preliminary set of outcomes, casemix and risk factors, and adequate measurement instruments, based on peer-reviewed literature and various (inter)national studies and databases such as the EMSCI database, the ISCoS Core Data Set and the NDD [14,15,16,17, 28]. The standard set included demographics and patient characteristics, admission and discharge date of initial acute and rehabilitation care, cause of trauma, presence of vertebral fractures, spinal surgery, and neurological and respiratory status [14, 15].

The preliminary set was developed based on the three-tiered outcome measures hierarchy as suggested by M. Porter [29]. To measure outcomes for any medical condition, Porter proposed a hierarchy based on three tiers: the achieved or retained health status, the process of recovery, and the sustainability of health. The top tier is most important and the lower tiers are dependent on the higher ones. Each medical condition has its own outcome measures and Porter states that a data set should consist of at least one outcome dimension at each tier [29]. The three-tiered outcome hierarchy was used as a framework to capture full cycle of care, which is endorsed by the International Consortium for Health Outcomes Measurement (ICHOM) [30] and the Organization for Economic Cooperation and Development (OECD) Health Ministers [31]. The initial preliminary set consists of 77 patient-reported and clinician-reported outcome domains, casemix and risk factors and subsequent measurement instruments, covering the acute surgical, subacute and chronic rehabilitation phase, and served as input for the Delphi rounds.

Procedure modified Delphi rounds

The first two rounds were performed through an online survey, and the third and final round was being held during an online face-to-face meeting. A final online round, including all panelists was held to confirm the pre-final set. The panelists were asked to answer “Yes or No” to questions related to outcome domains, measurement instruments (patient-reported and clinician-reported), and potentially relevant case-mix and risk factors. For each question the panelists could provide feedback in open text boxes. They were asked to consider their own professional opinion as well as the evidence provided from the literature and were encouraged to provide free text feedback. After each round the panelists received an anonymized feedback report with an overview of votes and a summary of the panelist’s feedback. This report was used as input for subsequent rounds.

The threshold for consensus was reached if ≥70% of panelists agreed on inclusion of a proposed outcome domain, measurement instrument, and contributing risk factor. Consensus of <30% lead to direct exclusion of the specific item. An agreement between 31% and 70% for the proposed outcome, factor or instrument lead to inclusion for consensus in the second Delphi round [25].

During the first online Delphi round (Delphi 1), formal consensus was sought for proposed outcome domains and measurements, as well as for casemix and risk factors. In the second online Delphi round (Delphi 2) the panelists were asked to vote and re-evaluate items for which no consensus was reached in the previous round and to confirm those items that already reached formal consensus.

The third Delphi round yielded an online face-to-face meeting using a videoconferencing platform (Zoom Video Communications Inc., 2016) with a multidisciplinary subgroup of the expert panel, consisting of 10 panelists, covering all relevant medical domains (orthopedic surgeons [n = 2], rehabilitation specialists [n = 2], physiotherapists [n = 2], occupational therapist [n = 1], psychologist [n = 1], physician assistant [n = 1] and a specialized nurse [n = 1]). The meeting was guided by an experienced, independent moderator, a methodologist who was not familiar with the management of t-SCI. Consensus was reached after discussion, for outstanding inconclusive outcome domains, measurement instruments and case-mix and risk factors, or the timing of data collection. After the online meeting, pre-final draft ‘factsheets’ were compiled of the standard outcomes set to be confirmed by all panelists. These factsheets included: the outcome domains, the appropriate combination of PROM’s, clinician-reported outcomes measures, the case-mix and risk factors, and the recommended timing of the data collection.

Results

Response

18 panelists of the 19 invited panelists responded to the first round of the online survey, achieving a response rate of 18/19 (94.7%). 16 panelists of the 19 invited responded to the second round of the online survey, achieving a response rate of 16/19 (84.2%). 10 panelists participated in the face-to-face meeting forming an expert panel. In Fig. 1 the Delphi process is shown, which includes the flow of agreed outcome domains, casemix and risk factors, and corresponding measurement instruments.

Fig. 1: Flow of results throughout the modified Delphi procedure for outcome domains, casemix and risk factors and measurement instruments.
figure 1

The threshold for consensus was set at >70% agreement. No items reached the minimum threshold for inclusion (<30% agreement). Items with 31–70% consensus were made available again for voting in the subsequent round. *Dimensions as proposed by M. Porter; †Timing of data collection included.

Full size image
Fig. 2: Voting results Delphi round 1 on outcome domains and measurement instruments.
figure 2

In favor (black), not in favor (striped) and not applicable (gray). *For measurement instruments, see supplementary fact sheet. **Threshold for consensus (≥70%).

Full size image
Fig. 3: Voting results Delphi round 1 on casemix and risk factors and measurement instruments.
figure 3

In favor (black), not in favor (striped) and not applicable (gray). *For measurement instruments, see supplementary fact sheet. **Threshold for consensus (≥70%). †No measurement.

Full size image

Delphi Round 1 (Figs. 2 and 3)

Consensus was reached on 24/26 outcome domains and on 13/24 measurement instruments. For the casemix and risk factors, consensus was reached on 17/20 items and on 13/19 proposed measurement instruments. A complete overview of the comments per outcome domain, measurement instrument, and casemix and risk factors in Round 1 is provided in the Supplementary Material.

Fig. 4: Voting results of Delphi round 2.
figure 4

In favor (black), not in favor (striped) and not applicable (gray). A Accepted questions from Delphi round 1, B items with no consensus from round Delphi round 1. **Threshold for consensus (≥70%).

Full size image

Delphi Round 2 (Fig. 4)

Consensus was reached on 9/10 previously accepted dimensions and on 10/14 outcome domains and measurement instruments that did not reach consensus in Delphi Round 1. The project team reconsidered four items (“Hoffer test and 2 MWT” [5/12; 42%]; “CHART-SF” [8/18; 44%]; “USER-P” [10/18; 56%] “Dural tear/CSF leakage” [10/18; 56%]) from Delphi round 1 that did not reach consensus and suggested to exclude these items based on the feedback of panelists. In Delphi 2 the expert panel unanimously agreed to remove these items from the standard set of outcomes.

The previously accepted dimension “Time to recovery and time to return to normal activities” was discussed. Panelists mentioned that these are not outcome measurements, but rather process indicators. The project team re-formulated these items and proposed a new timing schedule as well (see Supplementary Material). No consensus was reached on dural tear and/or CSF leakage (11/16; 69%), rate of re-operations (11/16; 69%), and the duration of surgery (11/16; 69%). These items returned in Delphi 3.

A complete overview of the comments and discussion in Round 2 is provided in Supplementary Material.

Fig. 5: Voting results of Delphi round 3.
figure 5

In favor (black), not in favor (striped) and not applicable (gray). A Items (n = 4) to be included in core data set, B items to be excluded from core data set. **Threshold for consensus (≥70%).

Full size image

Delphi Round 3 (Fig. 5)

During the final online face-to-face meeting the expert panel discussed the items that did not reach consensus in previous rounds, being the dimension of “Time to recovery and time to return to normal activities” and the outcome domains CSF leakage, rate of re-operations and duration of surgery.

Also, the project team enclosed four additional items based on received feedback on missing items (redefining timing of treatment and recovery, categorization of the rehabilitation program, WISCI II for walking ability, timing of data collection following EMSCI). The expert panel reached consensus on all the items (n = 8). A complete overview of the discussion in Round 3 is provided in Supplementary Material.

Outcome domains

Timepoint start rehabilitation phase

The variability of the timepoint when rehabilitation starts, during or after hospital admission, was discussed. The panel agreed that it is of relevance to identify the exact starting point of the rehabilitation (e.g., rehabilitation physician gave indication rehabilitation and active multidisciplinary rehabilitation had started [physiotherapist and occupational therapist are involved in treatment]). Suggested was to rename the second timepoint (arrival at hospital), and to define it as ‘admission to intervention center’.

The expert panel agreed (9/10 [90%]) on the timing of treatment and recovery as suggested by the project team (see Supplementary Material).

Duration of surgery, rate of re-operations and dural tear and/or CSF leakage

Duration of surgery was not deemed relevant for a standard outcomes set because of the amount of variation caused by other external factors, e.g., surgeon’s skills and difficulty of the procedure. Therefore, ‘duration of surgery’ was removed from the standard set (10/10 [100%]). Re-operations are rare and after discussion the ‘rate of re-operations of the spine’ was removed from the standard set (10/10 [100%]). Unanimous agreement was reached (10/10 [100%]) to exclude dural tear and/or CSF leakage from the standard set.

Casemix and risk factors

Rehabilitation program

The expert panel proposed that the earlier categories (Individual Physiotherapy, Technology Assisted, Group Walking Training, Fitness) should be revised and expanded. New categories should be defined in the future (e.g., based on the treatment frame SCI rehabilitation [in Dutch: ‘Behandelkader Dwarsleasie’]) [32]. The expert panel unanimously agreed (10/10 [100%] that the rehabilitation program should be categorized and included in the standard set.

Measurement instruments

Walking Index for Spinal Cord Injury II (WISCI II)

Issues were raised on including the WISCI II to measure walking ability, which was considered essential to include next to 10MWT and TuG. The 10MWT, TuG, WISCI II are required instruments in the EMSCI database and as such needed for a complete impression of walking ability. Two panelists withheld from voting, and consensus was reached (7/8 [87.5%]) resulting in the inclusion of the WISCI II in the standard set.

Arm and hand function

From a research perspective the relevance of the Graded Redefined Assessment of Strength, Sensibility and Prehension (GRASSP) test is acknowledged [33]. The use of the GRASSP test in daily practice is questioned and as such, it was removed from the standard set (10/10 [100%]).

Time schedule for data collection

The panelists agreed (9/9 [100%]) to include the time schedule for assessments as proposed following the EMSCI guidelines, providing a fixed time schedule in which patients can be tested and documented after SCI consisting of an acute care phase (0-15 days) and a follow-up phase (4 weeks, 12 weeks, 24 weeks and 12 months) [34].

Final standard network outcomes set

The pre-final standard set was confirmed by all panelists (response rate 18/19 [95%]) and is provided in Table 1. The confirmed casemix and risk factors are provided in Table 2.

Table 1 Standard network outcomes set for patients with traumatic spinal cord injury based on the three-tiered outcome hierarchy as developed by Porter [29].
Full size table
Table 2 Casemix and risk factors for patients with traumatic spinal cord injury.
Full size table

Discussion

Standardization and systematically monitoring outcome domains is essential for the improvement of quality in clinical practice and multicenter research in t-SCI. This study provides a standard network outcomes set for patients with traumatic spinal cord injury (t-SCI). This set is based on peer-reviewed literature, existing databases as the European Multicenter Study about Spinal Cord Injury (EMSCI) and the Dutch SCI Database (Nederlandse Dataset Dwarslaesie; NDD), and includes patient-relevant outcome domains covering the full cycle of care for patients with t-SCI. The current network core outcomes set slightly differs from other proposed sets (e.g., ICF Core Sets for individuals with spinal cord injury in the early post-acute context) because this set starts from the acute phase of SCI, in contrast to other data or outcomes sets that start measuring outcome domains from the post-acute phase [35, 36]. A recent study by Khosravi et al. mentioned the importance of the Delphi method to establish standards that can be used by all hospitals and rehabilitation centers involved in the care of patients with t-SCI [37]. In this study, consensus was reached among a panel of multidisciplinary experts of a regional Acute Spinal Cord Injury (ASCI) unit, on a network outcomes set consisting of outcome domains (n = 23), casemix and risk factors (n = 15), corresponding measurement instruments, and a schedule for timing of assessments.

To our knowledge, this standard network outcomes set is the first that provides patient-relevant outcome domains for the full cycle of care in form acute to chronic SCI care. This is based on the three domains of the ICF, assessed with patient-reported outcome measures (PROMs) to be used in daily clinical practice in the Netherlands. PROMs are widely used in orthopedic surgery and have the potential to provide information on value-based care, to improve shared decision making and endpoints that patients are interested in [22, 38]. Furthermore, it has been recognized that health care professionals and patients differently appraise certain outcomes. For example, physicians treating patients with t-SCI often assume that walking ability is the most important goal to achieve. However, most patients value improved sexual- or bladder function as most important [39, 40]. It is necessary that these mismatches are identified and resolved to improve the treatment of SCI.

During the Delphi study several items led to discussion among panelists. For example, the timing of the start of the rehabilitation phase was questioned. To resolve this issue guidelines are proposed for the management of SCI [41]. A study by Fehlings et al. suggests that rehabilitation should be offered to patients with SCI when they are medically stable and can tolerate required rehabilitation intensity. The evidence for this suggestion is weak and future studies are needed to explore the timing of rehabilitation after SCI [42]. This could lead to standardization of t-SCI care and in future this might contribute to improvement of the quality of care. Another example is the removal of ‘duration of surgery’ as a risk factor from the standard outcomes set. To our knowledge, no studies exist on the effect of duration of surgery on outcome in patients with t-SCI undergoing surgery. In other medical domains prolonged duration of surgery resulted in an increased risk for infection in total knee arthroplasty (TKA) and total hip arthroplasty (THA) [43,44,45,46,47,48,49]. As yet, duration of surgery was deemed irrelevant by the panelists to include in the standard outcomes set. In future, its relevance could be reconsidered when evidence is available for t-SCI.

Several rehabilitation items elicited discussion as well. Firstly, the item “rehabilitation program in a specialized SCI center” which was obtained in the standard set, needs further elucidation. Although rehabilitation after t-SCI follows (inter)national guidelines, multidisciplinary rehabilitation programs, do not follow a uniform and standardized protocol [41]. Protocolized care is not able to address the large interindividual variation of a patient with SCI, as the consequences of provided care and the patient-related factors determine which needs are most important for the patient. In future, to optimize rehabilitation programs, it is recommended to study specific treatment elements, such as individual physiotherapy, technology assisted training (e.g., exoskeleton) and walking training in group. Optimizing rehabilitation programs based on patient characteristics will ultimately result in achieving improved outcome for these patients.

This study has some limitations. First, we did not include a patient representative in the expert panel, so only health care professionals were involved in the modified Delphi process. Recent guidelines on Delphi procedures have proposed involving patient representatives in the process [50, 51]. In retrospect, it would have been relevant to involve a patient representative, since the patient can certainly contribute to improving the treatment process. In order to include the patients’ perspective, in a following study the current developed network core outcomes set will be validated using a patient sample.

Another limitation is that we did not engage program or data administrators to ensure feasibility of the collection of data. When implementing the data set, we recommend to include data administrators to ensure the feasibility of the data collection.

Finally, selection bias might have occurred. The expert panel was recruited from one ASCI network (ARU Nijmegen, The Netherlands). The ARU network in Nijmegen is a patient-centered network, containing the full cycle of care in the management of t-SCI. The response rates in the Delphi rounds were around 100%, extensive reviews of the literature were performed, and efforts were taken to strictly follow existing methodological guidelines. A next step would be to collaborate with and implement the standard outcomes set in comparable networks.

In this study a standard network outcomes set was developed, which is unique in its form. The standard outcomes set follows the patient journey for patients with t-SCI in the full cycle of care, from the acute phase of the injury until the discharge after rehabilitation phase. This network outcome set encompasses the recommendations of the EMSCI and NDD and can be used to compare casemix and risk-adjusted outcomes across regions, studies, and registries in order to gain insight and improve the quality of care for t-SCI in clinical practice. This network outcomes set will be implemented and evaluated in the ASCI Unit (ARU Nijmegen) and when feasible nationwide implemented. Implementation and use of these standard outcomes serve different purposes: individual evaluation of patient care, continuous evaluation of the quality of the care provided in a defined subgroup of patients, case mix and risk-adjusted benchmark between professionals and institutions, value-related healthcare, research (e.g., comparative effectiveness of procedures). By sharing the acquired knowledge with the patient, this ultimately supports patient-specific decision-making, and ultimately improves outcomes and value of care for t-SCI patients.