Introduction

Traumatic spinal cord injury (TSCI) is a devastating injury that impacts the individual, their support network and society as a whole. Although the incidence of TSCI is relatively low, the economic burden is high.1 Similar to other chronic conditions, individuals who suffer a TSCI require continued medical care to manage complications and consequences of their injuries. Suffering an adverse event (AE) in acute care can impact patient outcomes2 and recurrent hospital admission;3 therefore, if symptoms are identified promptly and treated appropriately, this economic burden could be reduced with improved patient outcomes.

The International Classification of Diseases, Tenth Revision codes (ICD-10) are assigned for inpatient hospital discharges, and represent a patient’s diagnostic and procedural events.4 The codes are entered into an administrative database and primarily used for management, planning and evaluation purposes. Common AEs are included, with associated ICD-10 codes, and provide an opportunity to study their incidence and prevalence. However, studies utilizing ICD codes in administrative databases should be approached with caution, as their original purpose was not intended for research. Previous studies have questioned the accuracy of administrative data for capturing clinical diagnoses, particularly AEs.5, 6, 7

The content validity and inter- and intra-observer reliability of a prospective AE identification system, the Spine Adverse Events Severity system (SAVES), was previously demonstrated when applied to all spine patients at a quaternary referral center.8 Discrepancies between ICD-9 coding and this prospective system were noted, further supporting the initial work by Rampersaud et al.9 in developing the SAVES system. In a study examining the effect of motor score on the occurrence of AEs in patients with TSCI using the SAVES system, it was found that 77.2% of patients had at least one AE during the acute care phase, with advancing age and lower initial motor score significantly impacting occurrence.2 The most commonly observed types of postoperative AEs were urinary tract infections, pneumonias, neuropathic pain, pressure sores and delirium.2

Accurate and reliable AE data identification and collection is critical to the development of clinical care guidelines and standards, to resource and funding allocation and to allow meaningful multicenter and multidisciplinary collaboration. It is vital that the process is rigorous, valid and reliable, and the data accurate and representative of patient complexity. The purpose of this study was to compare the previously validated prospective SAVES system with the conventionally used institutional ICD-10 coding process for identifying acute care AEs in patients with TSCI.

Materials and methods

Patient population

The study sample consisted of all adults with acute TSCI that were discharged from the Acute Spine Unit of a quaternary referral center and enrolled in the Rick Hansen SCI Registry between 2006 and 2010.

Data collection

AE data for this study was collected by two different systems, (1) from ICD-10 codes4 extracted from the Discharge Abstract Database ,10 and (2) from the prospective SAVES system, introduced in 2008, where all patients are discussed at weekly clinical care meetings. Full details of the SAVES system have been described elsewhere.8

The study sample was divided into two cohorts for analysis. The ICD-10 cohort consisted of Rick Hansen SCI Registry patients discharged from VGH between 1 January 2006 and 31 December 2007 with AE data collected from ICD-10 codes, while the SAVES cohort consisted of Rick Hansen SCI Registry patients discharged between 13 February 2008 and 14 May 2010 with AE data collected from SAVES. Two consecutive periods were used for the comparison to prevent potential bias against the ICD-10 coding process. Data population of the Discharge Abstract Database is dependent on documentation of the events by physicians in the patients’ hospital record. We were concerned that, after introduction of the SAVES process, physicians may be less likely to document events separately in the chart, thus undermining the coding system or that conversely, the data collected through the SAVES process may indirectly influence the ICD-10 documentation.

Data analysis

To compare the ability of SAVES to identify AEs relative to ICD-10 coding, a map was created to match AEs in SAVES with the corresponding ICD-10 code(s). This mapping was performed with the expert input of a number of spine surgeons and health information management professionals.

Incidence

Incidence of AEs was determined as a percentage of the total number of patients in each cohort. Chi-square and Kolmogorov−Smirnov tests were used to determine the difference in the total number of AEs and the number of events per person, respectively. Similarly, Chi-square and Fisher exact tests were used to determine differences in incidence of each type of AE between the ICD-10 and SAVES cohorts. Intra-operative AEs were tabulated together and reported as a single category, which included the following: allergic reaction, anesthesia-related, bone implant interface failure requiring revision, cardiac, cord injury, dural tear, hardware malposition requiring revision, hypotension, massive blood loss, nerve root injury, pressure sores identified during surgery, vascular injury and airway/ventilation.

Bivariate analysis

For the SAVES cohort, the relationship between age at injury and initial motor score with acute length of stay (LOS) was determined using linear regression and Pearson correlation coefficients. One-way analysis of variance analyses were done to test for the differences in the means of log-transformed LOS using the SAVES between patients who had AEs and those who did not. To make the analysis of variance reasonable, LOS was log-transformed in each analysis to obtain a normal distribution to take into account that LOS was right-skewed.

To determine the risk factors associated with AEs, Pearson correlation coefficients and Chi-square tests were used. The strength of association between the risk factors significantly associated with number of AEs was determined using linear regression, and those associated with types of AEs using logistic regression. The slopes from these regressions were compared for ICD-10 and SAVES cohorts. Data was analyzed using SAS 9.2 Software (SAS Canada, Toronto, ON, Canada).

Statement of ethics

This study was approved by the Clinical Research Ethics Board at the University of British Columbia and the Vancouver Coastal Health Research Institute. We certify that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed during the course of this research.

Results

Patient population

There were 212 patients in the ICD-10 cohort and 173 in the SAVES cohort. The cohorts had similar distribution with respect to age, gender and initial motor score in acute care. The cohorts differed only with respect to the mechanism of injury, with patients in the SAVES cohort experiencing more sports-related injuries. The percentage of injuries caused by falls and motor vehicle accidents were similar for both cohorts (Table 1).

Table 1 Distribution of patient characteristics in each cohort
Full size table

Incidence

The number of AEs identified using the SAVES system was 1.7 times greater than with the ICD-10 system, and the mean number of AEs per person was 2.1 times greater using SAVES (Table 2). In total, 45 different AEs were examined in this study (Supplementary Table S1), 15 of which were identified with increased frequency using SAVES (Table 3). Anemia was the only event captured more frequently using the ICD-10 system (Table 3). Of the five most common AEs recorded, three (neuropathic pain, pressure sores and UTIs) were captured significantly more frequently using the SAVES system (Figure 1 and Table 3). Additionally, SAVES captured significantly more intra-operative events, electrolyte abnormalities and pulmonary events as compared with ICD-10 coding.

Table 2 ICD-10 vs SAVES methods
Full size table
Table 3 ICD-10 vs SAVES methods
Full size table
Figure 1
figure 1

ICD-10 vs SAVES methods. The differences between the ICD-10 and SAVES methods for detecting the most common AEs. Abbreviations: ICD, International Classification of Disease; SAVES, Spine Adverse Events Severity System. *P<0.05, **P<0.01, ***P<0.001.

Full size image

Acute length of stay

Among the AEs that were identified more frequently using the SAVES system, eight were found to significantly impact the acute LOS (Table 4), including pressure sores, UTIs, electrolyte abnormalities and pulmonary events.

Table 4 AEs more frequently identified using the SAVES method that have a significant effect on acute LOS
Full size table

Risk factors and AEs

When the patient characteristics (demographics and injury variables) were analyzed as risk factors for AEs, the age at injury and initial motor score were found to be risk factors associated with the number and type of AEs using the SAVES system (Table 5). Age at injury and initial motor score were both risk factors for the development of urinary tract infection, pneumonia and delirium; additionally, initial motor score was a risk factor for pressure sores. The mechanism of injury, specifically a fall from a standing height, was a risk factor for delirium only. In contrast, when using data collected with the ICD-10 system, only initial motor score was identified to be a risk factor associated with number and type of AEs, specifically for urinary tract infection and pneumonia (Table 5).

Table 5 ICD-10 vs SAVES methods
Full size table

The strength of association between risk factors and AEs was determined as the degree of slope in the regressions, such that the change in slope between the ICD-10 and SAVES analyses represents the difference between the two systems (Table 6). There is a significant difference between the ICD-10 and SAVES systems with respect to the strength of association of age as a risk factor for number of AEs (Supplementary Figure S1a), pneumonia (Supplementary Figure S1b) and delirium (Supplementary Figure S1c), such that the association is significantly stronger using the SAVES system.

Table 6 ICD-10 vs SAVES methods. Comparing strength of association between patient characteristics and AEsa
Full size table

Discussion

In this observational cohort comparison, we demonstrate that the identification of AEs using the institutional ICD-10 coding system was significantly less reliable than the prospective SAVES system for an acute TSCI population. We have shown that important AEs were significantly underreported or missed with ICD-10 coding. Additionally, several of the underrepresented AEs had a significant impact on LOS and correlated with patient characteristics as risk factors. The discrepancies in AEs found by the two systems reflect their ability to detect for AEs, and not the differences between the two cohorts studied. The cohorts were comparable except for the distribution of mechanisms of injury; however, as a previous study showed that mechanism of injury was not a predictor of AEs,11 this difference was not considered to impact our results.

To use the ICD-10 coding data alone from our institution would mean that the incidence of acute care AEs would be significantly underreported and that clinically relevant correlations between risk factors and number/type of AEs would go unrecognized. As a result, the true complexity of this patient population and the enormous medical and economic burden of their acute care would be underestimated. Similarly, opportunities for early complication detection and treatment would be missed, thus compromising patient care.

As long as ICD-10 coding is used for research purposes, it is valuable to assess the validity of the administrative data and identify sources of error and means for improvement. When a patient is discharged from a hospital, the patient chart is reviewed by a health information management professional and coded using the ICD standards. Therefore, the code is an interpretation of the patient chart and not a clinical diagnosis. Sources of error for ICD coding include miscoding, misinterpretation of the chart, insufficient or illegible clinical notes, inaccurate data and incomplete charting. For AE reporting, inaccuracy could also be the result of inconsistencies with establishing a diagnosis, particularly for an AE that lacks an objective confirmatory laboratory test.

The stringent rules used for coding patient charts also limit the use of ICD-10 coding for AE identification. For example, intra-operative AEs can only be coded if there is a documented intra-operative consult by another surgeon, a return to the operating room, or a repair of the damaged organ. Infections can only be coded if they are clearly documented by a physician; coders cannot provide their own interpretation, even if there is evidence of a positive laboratory test result in the patient charts. Other AEs are only coded if they are considered to have a significant impact on patient outcome, which is typically an increase in LOS. Improving the accuracy of coding for AEs using the ICD coding system would entail the improvement of the coding system itself, standardized methods for AE diagnosis and education regarding the importance of complete patient chart recording.

We were interested to observe that anemia was recorded more frequently using the ICD-10 coding (3.3%) than by SAVES (0%). There does not appear to be a clear explanation for this, as the documentation of a low postoperative hemoglobin, and the need for transfusion are essential requirements for both systems of event recording. As part of our ongoing quality assurance process, we are currently examining each of the individual patient cases of all AEs, to determine if the observed differences are true (given that the data was generated from different time periods) or due to recording/coding differences. We anticipate that this process will further our understanding of the fundamental differences between these systems, and thus strengthen our argument for the widespread use of a universal prospective AE collection system in patients with TSCI.

Additionally, variation in definitions, methods of diagnosis and length of follow-up can impact the reporting of AEs. These variations are discussed in two recent systematic reviews.12, 13 Both reviews found that there is currently no consistent definition for an AE in the spine literature12, 13 and further there is no consensus regarding the appropriate assessment of AE incidence12 and very low strength of evidence with regard to the impact of complications on patient-centered outcomes.13 Clearly, the ‘science’ of AE reporting in spine surgery has a long way to go, and our systematic comparison with ICD-10 coding supports the contention for the adoption of a validated and reliable rigorous prospective system.

Previous literature assessing the validity of ICD coding have also found that prospective systems,6, 7 as well as retrospective chart review,5, 9, 14 capture more AEs compared with ICD coding from administrative databases. Campbell et al.7 found that their prospective approach to recording complications in spine surgery yielded a significantly greater number of complications, as well as clinically relevant complications. In a study that compared postcardiac surgery delirium rates between prospective data collection and a hospital administrative database, it was determined that ICD-10 codes significantly underestimated delirium rates.6 A systematic review that evaluated the impact of study design on the incidence of complications associated with spinal surgery showed that retrospective reviews significantly underestimate the incidence.12

Although there have been several studies that assess AEs relating to spinal surgery,7, 8, 9, 12, 15 few have focussed on TSCI.2 When all spinal surgery patients with varied pathologies are grouped, it creates a heterogeneous sample where it is difficult to apply the results to one specific SCI population, such as TSCI. In comparison to previously published spine literature,8, 14 our AE incidence rates are generally higher. This difference could be attributed to our rigorous process involving a prospective reporting method, with increased awareness among surgeons by incorporating dedicated rounds and a research assistant to ensure quality reporting, such that we may simply be reporting more AEs. However, even when the SAVES process is used in a spine trauma population,8 our reporting rates with TSCI are the highest, suggesting that the severity of injury and deficit associated with the TSCI population contributes to the incidence of AEs.

Limitations to this study include the resource intensive nature of the prospective collection process, which requires dedicated staff to regularly record AEs on all patients. Also, it was conducted at a single institution using consecutive cohorts, whereas the use of concurrent cohorts would have been optimal. Future directions for this work include an inter- and intra-rater reliability study on the SAVES system at our institute, in addition to a larger multicenter study. This would involve the standardization of the SAVES system across centers, as well as specific in-house validation and reliability studies.

The current findings support previous work on AEs following spinal surgery and are the first to compare a prospective method with ICD coding in the TSCI population. Interpretation of results from administrative databases without appropriate validation should be viewed with caution. For TSCI, the AE incidence calculated based on ICD coding may be inaccurate and not representative for this complex patient population. Prospective methods of data collection for AEs are far superior for determining incidence and associations with patient characteristics that contribute to risk assessment in the acute care setting.

Data archiving

There were no data to deposit.