Introduction

Disability has a tremendous impact on both the individual and the society. It is expected that disability rates will continue to rise as the population ages, as survival from injuries and diseases increases with medical advances and as the surveillance of disability improves.1 With the realization that disability is not just a medical issue there has been considerable interest in understanding and measuring the behavioral and social impacts of disability.

Several conceptual models have been developed to assess disability dating back to Nagi's model of the disability process in 19652 and more recently the International Classification of Functioning, Disability and Health (ICF)3 published by the World Health Organization in 2001. Although the terminology in these models may differ, the idea of measuring a person's participation in his or her life activities is common to all models and helps to understand the impact of disability.

Over time there have been some significant changes in how the concept of participation is operationalized. In the International Classification of Impairments, Disabilities and Handicaps (ICIDH),4 the predecessor to the ICF, the term handicap was used instead of participation. Handicap was defined as ‘the disadvantage of a given individual resulting from an impairment or a disability that limits or prevents the fulfillment of a role that is normal (depending on age, sex and social and cultural factors) for that individual’ (page 182).4 Handicap was measured by determining how much a person deviates from roles fulfilled by an able-bodied member of the society.5

In the ICF, the term handicap was replaced with the term participation due to its negative connotations.6 Participation is defined as ‘the involvement in a life situation’ (page 10).3 In this definition, participation refers more to the personal fulfillment of roles rather than fulfilling roles deemed important by the society, which is an important shift.7 The ICF also recognizes the importance of contextual factors, specifically personal factors and environmental factors, which can have an impact on the individual and the individual's health state.3 In addition to measuring objective or quantifiable information about participation (either with or without comparisons to societal norms), it has been recognized that it is important to also capture the person's perspective or subjective information regarding participation.6, 8

Participation is cited as being important for a person's quality of life and well-being and is an important outcome in rehabilitation.9, 10 It is therefore critical that instruments are available to measure participation. As the measurement properties of an instrument are not an intrinsic property of an instrument but rather provide evidence as to the meaning of scores for a particular health condition,11, 12 it is necessary that participation instruments are tested with persons with SCI. The purpose of this paper is to provide an overview of participation instruments assessed in persons with SCI and to critically evaluate their measurement properties.

Materials and methods

Search strategy

A review of instruments assessing participation in individuals with SCI was conducted as a part of the Spinal Cord Injury Rehabilitation Evidence project. Instruments were identified using a key word search of multiple electronic databases (MEDLINE/PubMed, CINAHL, EMBASE and PsychInfo), reviewing the references in the papers identified from the search, and from hand searches of SCI journals from 1980 to March 2008. The search phrase ‘spinal cord injury’ was used in each database, whereas the following terms varied depending on the database searched: validation studies, instrument validation, external validity, internal validity, criterion-related validity, concurrent validity, discriminant validity, content validity, face validity, predictive validity, reliability, inter-rater reliability, intra-rater reliability, test–retest reliability, reproducibility, responsiveness, sensitivity to change, evidence-based medicine, outcome measures, clinical assessment tools, scales and measures.

Inclusion criteria

A team consisting of researchers and clinicians with expertise in SCI reviewed all the papers. Instruments were included in this review if there was information published on its measurement properties (reliability, validity and/or responsiveness) in adults with SCI and published in at least one peer-reviewed journal. Papers were only included if the focus of the paper was to assess the measurement properties of an instrument and if the papers were published in English. Instruments were considered to assess participation if the domains covered two or more of the following ICF chapter headings: domestic life (chapter 6); interpersonal interactions and relationships (chapter 7); major life areas (chapter 8); and community, social and civic life (chapter 9).3, 13, 14 Instruments such as the Spinal Cord Independence Measure III (SCIM III),15 which primarily assess mobility (chapter 4) and self-care (chapter 5) were considered to assess the ICF concept activity.3, 13, 14 Instruments such as the Sickness Impact Profile,16 which include not only content related to participation but also other ICF concepts (for example, body functions and activity), were classified as health-status instruments.

Data abstraction

Data were abstracted based on the 11 criteria proposed by Andresen17 for assessing instruments used in people with disabilities. The definitions for five criteria (reliability, validity, responsiveness, item/instrument bias and measurement model) were modified slightly for this review. Reliability and validity each has three types and so a total of nine measurement properties were assessed and then assigned a grade (see Table 1) based on the evidence available for persons with SCI. Data pertaining to reliability, validity, responsiveness and the measurement model of the instrument were abstracted by two team members. Item/instrument bias was assessed by reviewing the original papers describing the instrument as well as papers included in this review to determine if individuals with SCI examined the instrument content. When the grades varied either within or across studies (for example, grade A in one study and grade B in another study), the results are presented as a range. Information on the remaining six criteria [conceptual model; norms available; respondent burden (that is, time to complete); administrative burden (that is, scoring/interpretation); alternate forms (that is, modes of administration); and language adaptations] were obtained but not graded as the information is generic. Study-specific sample characteristics were also included.

Table 1 Criteria for evaluating the instruments17 (page S16)

Results

The systematic search identified eight instruments: Athletic Identity Measurement Scale;18 Craig Handicap Assessment and Reporting Technique (CHART);5 Employment Questionnaire;19 Impact on Participation and Autonomy Questionnaire (IPA);7 Assessment of Life Habits Scale (Life-H);20 Occupational Performance History Interview (OPHI);21 Physical Activity Recall Assessment for People with Spinal Cord Injury (PARA-SCI);22 and Reintegration to Normal Living Index (RNL).23 The Athletic Identity Measurement Scale and Employment Questionnaire were excluded based on the definition of participation used in this review as these instruments primarily assess athletic/recreation activities (related to ICF chapter 9) and employment (related to ICF chapter 8), respectively.

Instrument description

An overview of the six instruments included in the systematic search is described in Table 2. Each of the instruments was developed using different conceptual models. The CHART was designed to measure the level of handicap as defined in the ICIDH5 and the IPA was based on beta version of the ICF called the ICIDH-2.24 A conceptual model very similar to the ICF, called the Disability Creation Process Model, was used to develop the Life-H.20 The OPHI was developed to be used with the Model of Human Occupation21 and the RNL was designed to measure the concept of re-integration to normal living.23 The PARA-SCI measures physical activity in persons with SCI,22 making it the only condition-specific instrument. In terms of the content, Table 2 provides an overview of the domains covered. All of the instruments assess aspects of mobility (ICF chapter 4), daily activities (related to ICF chapters 5/6), recreation/leisure (related to ICF chapter 9) and work/education (related to ICF chapter 8). Interpersonal relationships (related to ICF chapter 7) are not specifically mentioned in the PARA-SCI or the OPHI.

Table 2 Description of participation instruments

There are differences in the types of information asked in six instruments. The CHART and the PARA-SCI include questions measuring objective or quantifiable data, for example ‘on a typical day, how many hours are you out of bed’ (CHART) or number of minutes engaged in moderate-intensity leisure-time physical activity (PARA-SCI). In the CHART, quantifiable information is used to determine the amount the respondent deviates from roles generally fulfilled by able-bodied members of the society. The PARA-SCI quantifies activity intensity within leisure, lifestyle and cumulative activity to measure physical fitness. As these two instruments use quantifiable information they are considered to assess objective aspects of participation.

The remaining four instruments primarily assess subjective aspects of participation. Questions in these instruments require judgement from the respondent, for example asking about difficulty in ‘maintaining relationships’ (Life-H), ‘my chances of fulfilling my role at home as I would like are’ (IPA) and ‘I am able to take trips out of town as I feel are necessary’ (RNL). The OPHI rating scale is the only instrument in which the evaluation is provided by a clinician following an interview with the individual that assesses his or her behaviour. All four instruments include the environment as either part of the question (IPA and RNL) or as separate question (Life-H and OPHI). However, in all four instruments, the environment questions are included in the participation score. The Life-H also includes additional questions that ask about the level of satisfaction for each aspect of participation.

There is considerable variation in the number of questions, ranging from 11 in the RNL to 240 in the Life-H (Long Form), requiring anywhere from 10 to 120 min to complete. Two instruments must be administered by interview (PARA-SCI and OPHI) and the remaining four instruments can be self-administered. Four of the instruments have been translated into other languages (CHART, Life-H, IPA and RNL). All the instruments include domain scores with the exception of the RNL, which only reports an overall score. It has been recommended that the CHART is reported using the domain scores as the total score can provide a misleading assessment.27 Normative data on various types of neurological injury and other health conditions are available for the CHART to compare scores.25, 27

Measurement properties

An overview of the subjects included in the studies assessing the instruments is included in Table 3. Evidence pertaining to the reliability of the instruments is provided in Table 4. Two instruments had information on the internal consistency of the domains/overall score (IPA and Life-H). Internal consistency for all the IPA participation questions was 0.94 and 0.82 for all the problem questions and these estimates were derived from the person separation index in a Rasch analysis that is similar to internal consistency.28 Test–retest reliability was assessed in CHART, Life-H and PARA-SCI. The CHART had the most evidence with three studies assessing test–retest reliability.5, 29, 30 The test–retest reliability was higher in the Life-H (Short Form) (ICC=0.83) compared with the Life-H (Long Form) (ICC=0.74).20 In the PARA-SCI, the leisure-time physical activity domain (ICC=0.72) had a lower test–retest reliability compared with the lifestyle activity (ICC=0.78) and cumulative activity (ICC=0.79).22 Only the CHART assessed proxy inter-rater reliability and it was consistently lower compared with the test–retest reliability as would be expected. Within the CHART, mobility had the highest proxy inter-rater reliability (r=0.84, ICC=0.86) and the domains with the lowest proxy inter-rater reliability included social integration (r=0.28) and cognitive independence (ICC=0.34).5, 31

Table 3 Description of study subjects
Table 4 Reliability of the participation instruments

Validity was assessed in all of the instruments (see Table 5). Rasch analysis was used in testing the CHART, IPA and Life-H, providing evidence for the factorial structure.5, 32, 34 Convergent validity was assessed in all of the instruments with the exception of the IPA. Two instruments included in this review (CHART and Life-H) were compared by mapping all of the Life-H (Short Form) questions into the corresponding domains of the CHART.35 The Life-H and CHART physical independence domains had the highest correlation (ρ=0.76) and social integration domains had the lowest (ρ=0.14).35 Discriminant validity was assessed in four of the six instruments. There were differences among the four instruments. For example, the CHART27 and PARA-SCI leisure-time physical activity22 were able to discriminate among subjects of various ages, but the Life-H (Short Form)36 was generally not able to. Only one instrument (PARA-SCI)22 involved persons with SCI in the development of the content; however, persons with SCI were part of the initial testing for the CHART,5 IPA,24 OPHI21 and Life-H.20 Rehabilitation clinicians were asked to review the content in all six instruments. Floor effects were noted in PARA-SCI for high-intensity lifestyle activity22 and ceiling effects were observed in the domains of the CHART.27 In the IPA when the participation questions were considered as one scale, 4% achieved the best possible score32 and so ceiling effects may be an issue with the domain scores. None of the instruments assessed responsiveness.

Table 5 Validity and Related Measurement Properties of the Participation Instruments

A summary of the grades assigned for each of the eight measurement properties is included in Table 6. Responsiveness is not included as no information was available. The CHART had the most evidence supporting its measurement properties and grades were assigned for seven of the eight measurement properties achieving a good-to-moderate rating in all categories. The OPHI and RNL had the least amount of evidence, with only three measurement properties assessed in each instrument.

Table 6 Summary of the measurement properties

Discussion

Even though participation is an important goal of the rehabilitation process, it is not often measured.27 This paper included six participation instruments that have been assessed in persons with SCI, and each instrument was reviewed considering critical measurement criteria. Information from this review may assist clinicians in selecting an appropriate measure, identifying areas of future research and hopefully will increase awareness about the importance of this rehabilitation outcome.

Instruments were considered to be a measure of participation if their domains cover a minimum of two ICF chapters from 6 to 9. This operational definition of participation is just one way to differentiate activity from participation. In the ICF, the user is provided with a list of activity and participation domains (ICF chapter headings) and can allow for total, partial or no overlap within either of the domains or categories included within each domain.3 On the basis of this definition of participation, instruments such as the SCIM III,15 which include questions covering ICF chapters related to mobility (ICF chapter 4) and self-care (ICF chapter 5), were not included in this review. However, if a broader definition of participation was used, then some may consider instruments such as the SCIM III to be measuring aspects of participation. It has been suggested that activity and participation are distinct concepts that must be differentiated conceptually and operationally.40 Whiteneck40 stated that activity is at an individual level and performed alone, where as participation is at a societal level and performed with others. However, Nordenfelt41 recommended not distinguishing between activity and participation and instead suggested combining them and refer to them as ‘actions’ based on philosophical action theory.42 Future study must continue to clarify if activity does differ from participation and then if these are two distinct concepts, how they differ.

Although the ICF was used to classify the participation instruments, only the IPA was based on a draft version of the ICF, referred to as the ICIDH-2. As the conceptual models used in developing the instruments differ it is difficult to compare the content of the instruments using the ICF chapter headings. On the basis of the domains within the instruments, it appears that they all assess aspects of mobility, daily activities, recreation/leisure and work/education. As the PARA-SCI was developed to assess physical activity and the OPHI was developed to be used with the Model of Human Occupation, it is not too surprising that interpersonal relationships are not specifically covered in these instruments. To obtain a better understanding of the content it would be useful to link the individual questions to the ICF to determine what ICF categories each question covers. The CHART and the IPA have been linked to the ICF by Perenboom and Chorus,9 and these authors commented on the differences in how activity was distinguished from participation. More recently, Cieza et al.43, 44 published methodology on how to link content within instruments to the ICF, and future studies should use these methods to link the content of the participation instruments included in this review. In addition, this methodology could also be used to examine if environmental factors are included in the participation instruments, as environmental factors significantly influence a person's participation.6

In terms of how participation was operationalized, two instruments assess objective aspects of participation and four primarily assess subjective aspects of participation. Quantifiable information obtained from participation instruments is helpful to describe participation restrictions in epidemiological studies and enables comparisons to be made between groups or even societies. As the PARA-SCI was developed specifically for persons with SCI, the comparisons using scores derived from this instrument will provide meaningful information that can be used in intervention studies as well as to create public health guidelines for persons with SCI. When using a participation instrument, such as the CHART, that uses normative data in the scoring procedure, it is important to consider the source of this data and ensure that it is relevant. For example, the CHART scoring uses normative from able-bodied individuals in the US population, and this must be reviewed to ensure that it is relevant in other countries. It may be helpful to report the individual question responses as well as the scores to address the ceiling effects as well as cultural differences in other countries. For example, Dijkers et al.45 excluded the domain economic self-sufficiency in the CHART and reported data for some questions when comparing samples of persons with SCI in the USA and Turkey. To address the skewed score distributions resulting from ceiling effects in the CHART, domain and total scores have also not been ’capped’ at the maximum score (that is, allowing a domain score to be greater than 100).45

The four instruments that assess subjective aspects of participation differ in the wording used to measure areas of participation. The Life-H includes questions asking about difficulty and type of assistance as well as satisfaction for each life habit. In the RNL, the respondent rates how he or she has re-integrated back to the various roles in their life. The IPA assesses perceived participation and autonomy as well as problems experienced when participating in life situations. The questions asking about perceived problems with participation in the IPA allow rehabilitation professionals to set goals as not all restrictions are deemed equivalent by the respondent.7 The OPHI rating scale is different from the three other instruments as the interviewer rates how well the person has adapted and is able to function following an interview. Although this instrument uses an external person to rate adaptive status it was considered to be measuring subjective aspects of participation as the interviewer elicits subjective information from the interviewee and uses this information to determine the rating. Future research should compare these four instruments with each other using qualitative and quantitative studies to determine if the subjective information obtained from these instruments differs.

The amount of evidence available on the instruments’ measurement properties varied, with the CHART having the most evidence and the OPHI and RNL having the least evidence. In terms of reliability, internal consistency has only been assessed in the IPA and Life-H, and the results suggest that these are very good. Test–retest reliability in the CHART, Life-H and PARA-SCI varied between the studies; however, overall, the results suggest that these instruments provide stable information. Proxy reliability has only been assessed in the CHART and was higher at 1 week compared with 2 weeks following the initial assessments.5, 31 The domains social integration and cognitive independence appear to be the least reliable assessment when provided by proxy ratings, and this is because these domains may vary with time or may not be as easy to observe behaviour.31 It is therefore questionable whether instruments assessing subjective aspects of participation could be completed by proxy as subjective information is not considered to be as reliable as objective information and to be valid must be reported by the person.46

Advances in measurement, including Rasch and other item response theory models, may enhance how participation is operationalized and measured. Of the instruments that have been tested using Rasch analysis, the IPA and Life-H both contained questions that did not fit the Rasch model. By removing misfitting items in the IPA, all of the questions in the perceived participation domains and all of the perceived problem questions each formed a unidimensional scale (participation scale and problem scale).32 Future study must determine if modern measurement methods are suitable for assessing participation as there are a number of assumptions that must be met47 and by removing misfitting items important content may be lost. The criteria used to assess participation instruments may need to be modified once more research has determined the most appropriate evaluation methods.

Convergent validity has been assessed in five of the six instruments. The RNL was compared with the Quality of Life Index and a correlation of r=−0.65 was reported that is not surprising, given the similarities between the questions used to measure participation and quality of life.39 Only the CHART and Life-H have been compared with each other, and the low correlation (ρ=0.14) in domains such as social integration demonstrates the differences in objective versus subjective aspects of participation.35 Various types of known groups have been assessed. The CHART and the PARA-SCI were able to differentiate individuals based on the age, but this was not evident in the Life-H. Objective aspects of participation such as combined income are included in the CHART, may be affected by age and these questions are not included in the Life-H.

The criterion, item/instrument bias, assesses how the instrument content was developed. The PARA-SCI was the only instrument that involved persons with SCI in developing the content. However, the CHART, IPA, OPHI and Life-H included persons with SCI in the initial testing, which supports their suitability for persons with SCI. The involvement of persons with SCI in the development of new instruments is critical. A study by Gray et al.48 that included persons with SCI described how the findings from qualitative focus groups regarding what participation means as well as what are the barriers and supports to participation formed the basis for developing the Participation Survey/Mobility (Parts/M). Developers of new instruments will gain tremendous insight and produce more meaningful measures by engaging individuals who live with conditions such as SCI.

Finally, the measurement model of the participation instruments was assessed considering floor and ceiling effects. If information was not reported on the percentage of the sample with extreme scores, then the mean and standard deviation was considered. Floor effects were noted for high-intensity activities in lifestyle activities in the PARA-SCI, which may not be a problem as moderate-intensity activities are also assessed for these activities. Ceiling effects were noted in the CHART and may be a problem with the IPA domain scores. If participation instruments are going to be used to assess rehabilitation interventions or changes over time due to ageing, then floor and ceiling effects may prevent these changes from being captured. To date, none of the instruments have reported evidence assessing responsiveness.

Recommendations

A comprehensive instrument that contains objective and subjective information and has minimal floor and ceiling effects would be ideal. Moreover, participation is considered as one of the most coveted rehabilitation outcomes existing, and new instruments require evidence on responsiveness so that clinicians and researchers can assess if their interventions are making a difference.

Currently, the CHART has been tested the most in persons with SCI and includes questions covering content from chapters 6 to 9 of the ICF. However, the CHART only assesses objective aspects of participation and uses normative data on able-bodied persons to quantify the participation restriction. Instruments such as Life-H, IPA and RNL also include questions covering content from chapters 6 to 9 of the ICF and assess subjective aspects of participation. To date, there is more evidence supporting the measurement properties of the Life-H compared with the IPA or RNL. No information on the measurement properties of the Life-H satisfaction questions was reported, and this should be tested in future studies. In addition, there has only been one study that compared the Life-H and the CHART, and future research should continue to concurrently assess instruments included in this review to determine how they compare. Since the ICF was published in 2001 a number of new participation instruments have been developed using the ICF.48, 49, 50, 51, 52, 53, 54, 55 Some of these instruments have been assessed in persons with SCI,48, 50, 51, 52, 54 but no data on the measurement properties were specified for this group and so they were not included in this review. Future studies reporting the measurement properties of these instruments in persons with SCI will provide the evidence necessary to use these instruments in research and clinical practice.

Conclusions

The instruments included in this review vary in content as well as in how participation is operationalized. It is therefore important to first determine what information is required about the concept participation before selecting an instrument. Future study determining the role of modern measurement methods such as item response theory (including Rasch) as well as more research assessing the measurement properties of participation instruments in persons with SCI is needed. Projects such as Spinal Cord Injury Rehabilitation Evidence (SCIRE) can assist clinicians and researchers in selecting appropriate instruments. These reviews should be regularly updated and include the six instruments described in this paper as well as new participation instruments as information becomes available on their use in persons with SCI.