Abstract
Study design
Mixed methods.
Objectives
The aim of our study was to investigate the content validity of the Work Rehabilitation Questionnaire (WORQ) for use in persons with post-acute and chronic spinal cord injury (SCI).
Setting
A university-based Rehabilitation Center in The Netherlands.
Methods
Contents of the WORQ, brief ICF core sets for SCI for post-acute care and for chronic situation were compared with semi-guided interviews with persons with SCI and controlled for relevance by SCI rehabilitation professionals in two group meetings.
Results
Fourteen interviews with persons with SCI were performed. Two group meetings with 8 and 9 SCI rehabilitation professionals were held. Thirty seven of the 46 ICF categories (80%) of the WORQ were confirmed by both sources: mentioned in interviews with persons with SCI and considered important by the SCI professionals. The remaining 9 categories (20%) were confirmed by either the persons with SCI or the SCI professionals. Fourteen ICF categories that are part of the brief ICF core set for SCI for acute care and/or chronic situation, however are not part of the WORQ, have revealed importance by persons with SCI and SCI professionals.
Conclusion
Our study confirms that most categories of the WORQ are important to consider for VR in persons with SCI, however, there are ICF categories that are absent in the WORQ and deemed relevant for use in VR in persons with SCI. Consequently, the content validity of the WORQ without additional items is insufficient for persons with SCI.
Similar content being viewed by others
Introduction
The benefits of participation in paid work for the individual with a spinal cord injury (SCI) are multifaceted and go beyond monetary compensation [1]. Society also benefits from high return to work rates, for example by avoiding expenses for unemployment benefits [2]. It is known however that persons with SCI encounter difficulties in returning to paid work after the onset of their condition. Hence, return to work is an important goal in SCI rehabilitation medicine and vocational rehabilitation (VR) gains attention worldwide.
The Work Rehabilitation Questionnaire (WORQ) was developed to assess work-related functioning and consists of a selection of categories from the International Classification of Functioning, Disability, and Health (ICF) [3,4,5]. The WORQ can be used to describe a person’s levels of functioning on work-related domains at the start of VR; to set VR goals and monitor functioning over time; to facilitate interdisciplinary and client communication in VR and to stimulate the client to actively participate in his/her VR process [6]. Moreover, the WORQ is free and easy to use. The first version of the WORQ was interview administered. The initial psychometric evaluation of the WORQ showed high test-retest reliability and good internal consistency [3]. To improve the practicality of its use, a self-report version was developed [7]. This version has been translated in many languages [8] and showed good usability and test-retest reliability and promising construct validity on relatively small samples with varying pathologies [9,10,11]. Furthermore, in a validity study in a different population (musculoskeletal disorders), professionals stated that they gained significant information on work-related functioning but suggested that the WORQ would be most valuable when used in complex diagnoses, possibly because the low median sum score of the WORQ in their study [12].
A previous study on the content validity of the WORQ for use in persons with SCI in the early post-acute phase suggested evidence for content validity, but also some gaps and the authors recommended further research on the utility of the WORQ considering concepts identified as lacking in their study (d410 changing basic body position, d445 hand and arm use, d460 moving around in different locations, d640 doing housework, d760 family relationships, e115 products and technology for personal use in daily living, e120 products and technology for personal indoor and outdoor mobility and transportation, e150 design, construction and building products and technology of buildings for public use, e155 design, construction and building products and technology of buildings for private use) [7]. They also suggested to further evaluate its validity in persons with post-acute and chronic SCI [7].
To our knowledge, no study on content validity of the WORQ from the perspective of SCI rehabilitation professionals has been published. The aim of our study was to investigate the content validity of the Dutch self-report version of the WORQ for use in persons with post-acute and chronic SCI, by identifying relevant, irrelevant and lacking categories.
Methods
Design
This study used a mixed-methods design by combining findings from semi-structured interviews with persons with SCI and two group meetings with SCI professionals with a list of SCI specific ICF categories (qualitative study) and with findings from the WORQ filled out by the persons with SCI (quantitative study). The study was performed at a university-based Rehabilitation Center in The Netherlands.
Participants
Persons with SCI and active paid employment (≥1 h per week [13]) prior to sustaining the SCI were included in this study. Potential participants were purposefully selected according to SCI level and American Spinal Injury Association (ASIA) Impairment Scale [14]. Four subgroups were selected: ≥C8 AIS A or B (tetraplegia-motor complete), ≥C8 AIS C or D (tetraplegia-motor incomplete), ≤T1 AIS A or B (paraplegia-motor complete), or ≤T1 AIS C or D (paraplegia-motor incomplete). We aimed to include 4 participants in every subgroup (16 persons in total): 2 still undergoing inpatient rehabilitation; 1 at least 1-year post-SCI who had finished rehabilitation and returned to work and 1 at least 1-year post-SCI who had finished rehabilitation and did not return to work.
Eight professionals with ≥6 years of experience in SCI rehabilitation were selected to represent all relevant disciplines. The following team members participated in the 2 group meetings: vocational counselor, occupational therapist, physiotherapist, social worker, psychologist, rehabilitation nurse, activity therapist, and rehabilitation physician.
Procedures
Potential participants with SCI were selected by the first author, rehabilitation physician on the SCI ward, and invited for the study between May and October 2019. They received written and oral information and provided written informed consent. Those who agreed to participate completed the WORQ questionnaire first and participated in a semi-structured interview afterwards. The first six interviews were performed by both the physician and a master-level medical student to assure a similar method of interviewing. The subsequent interviews were performed by either the student or the physician.
An ICF-list was introduced to the SCI rehabilitation professionals who also received the WORQ beforehand as background information. A group meeting was held in which the professionals were asked to fill out on the ICF-list which ICF categories they considered important or irrelevant for use in VR in persons with SCI without consulting others. Responses were analyzed afterwards by the first author. Categories were considered as important if 75% or more of the professionals identified the item as important. To decide on the categories that were affirmed by more than 40% and less than 75% of the professionals, a second group meeting was held under presence of an additional rehabilitation professional who is a practicing VR counselor and also expert in research (3rd author) to reach consensus on the ICF categories by having a second voting round [15].
Instruments
The Dutch self-report version of the WORQ was developed by adapting the Flemish self-report version of the WORQ [10], in collaboration with the Flemish researchers [11]. The instrument consists of a part 1 with 17 socio-demographic and work situation-related questions and a part 2 with 40 functioning questions based on the ICF categories of the ICF Core Set for VR. Questions in part 2 were posed as ‘Overall in the past week, to what extent did you have problems with …?’. Answers were scored from 0 (no problem) to 10 (complete problem).
The open-ended questions of the patient interview (see supplementary appendix) were based on a previous study and aim to explore the vocational-related experiences of participants by investigating body functions, activity and participation, environmental factors (barriers and facilitators), and personal factors [7].
An ICF-list with a total of 72 ICF categories was composed by merging the categories represented by the items of the self-report WORQ based on the WORQ development study and a previous content validity study [3, 7], the brief ICF core set for SCI for post-acute care [16] and the brief ICF core set for SCI for chronic situation [17] into one list. To keep the level of detail consistent across all categories, only second-level ICF categories were included in this list. Consequently, several items of the WORQ representing more detailed ICF categories within the same second-level category were merged:
-
items Temper, Self-confidence and Irritable, into b126 Personality functions,
-
items Sad/depressed and Worried/anxious into b152 Emotional functions,
-
items Seeing Object and Seeing Person into b210 Seeing functions,
-
items Lifting and carrying objects weighing up to 5 kg and Lifting and carrying objects weighing more than 5 kg into d430 Lifting and carrying objects
-
items Walking a short distance and Walking a long distance into d450 Walking.
Data collection about the persons with SCI included socio-demographics (age, sex), the date of interview, SCI subgroup and time since SCI onset (less or more than 1 year), employment characteristics at the time of interview (performing paid work or not) and their scores on the WORQ part 2. Regarding the professionals, data collection included years of experience in SCI rehabilitation and their rating of the importance of the ICF categories.
Data analysis
The aim of our study was to investigate the content validity of the Dutch self-report version of the WORQ for use in persons with post-acute and chronic SCI, by identifying relevant, irrelevant, and lacking categories.
To identify a category as important we used several criteria: included in the Brief ICF Core Sets for SCI, mentioned in interviews by persons with SCI and consensus on importance by the SCI professional team.
Qualitative data analysis
The interview was audio-recorded and transcribed. Two researchers independently coded text fragments of the first three transcripts using the ICF list, compared the results of their coding, discussed the differences, and reached consensus on how to finally code the text. Further transcripts were first read and coded by one researcher and afterwards checked by the second researcher. For 2 ICF categories on this list, the Dutch translation of these 2 ICF categories is not completely aligned with the English version and therefore the coding was performed according to the English version. The codes mentioned during the interviews that did not match a category of the ICF-list were listed as “not included”. These codes were linked to another ICF category if possible. The number of interviews in which the ICF category occurred was reported. The frequency of the ICF categories per interview was not analyzed.
Furthermore, we determined what ICF categories were quoted as important or irrelevant in ≥ 75% by the SCI professionals.
Quantitative data analysis
Quantitative data were analyzed using SPSS for Windows, version 23.0 (IBM Corp., Armonk, NY). Distributions of scores from the participants on the functioning categories of the WORQ part 2 were expressed as Median (IQR). If a participant provided 2 scores per question, the average of the two scores was used. If item 34 (driving) was marked as not applicable, this was treated as a missing value. If less than 25% of the item scores were missing, the missing values were replaced by the mean of the valid WORQ item scores.
Results
At the time of performing the interviews, only one person with motor-complete SCI in inpatient rehabilitation could be included in this study and therefore this group was extended by recruiting persons with similar but less recent SCI. Fourteen of the intended 16 interviews were conducted. The sociodemographic and injury-related characteristics of the participants are presented in Table 1.
Table 2 shows the ICF list and the number of interviews in which the category occurred; the median scores of the ICF categories as filled out on the WORQ part 2 by the persons with SCI; and the categories that are important according to 75% or more of the SCI professionals. One participant did not return the WORQ-questionnaire on time but several months after the interview, the analysis of this questionnaire was considered missing. Some ICF categories (e.g., d850 remunerative employment, e310 immediate family, e330 people in positions of authority) are included in the WORQ part 1 by multiple-choice questions and those concepts do not have a median score.
Thirty seven out of 46 categories (80%) of the WORQ were confirmed by both sources: mentioned in interviews with persons with SCI and considered important by the SCI professional team. Those categories are marked in bold in Table 2. Of the other 9 ICF categories, four were mentioned by three or more persons with SCI, but were not identified by consensus as important by the professionals. Moving around (d455) obtained the maximum median score of 10 and was mentioned by 4 participants with SCI, but not considered important by the professionals. Five categories were considered important by the SCI professionals, but not mentioned in the interviews with persons with SCI. Of those categories the median score (if applicable) was rather low (≤3).
Table 3 shows the ICF categories, part of the brief ICF core set for SCI for acute care and/or chronic situation but not of the WORQ, that were mentioned in the interviews with persons with SCI or considered relevant by the SCI professionals. The categories in bold were mentioned by both sources. The ICF categories that were mentioned in the interviews with persons with SCI only (marked with * in the text box) were mentioned in one (s120 and d510) or 6 interviews (e155).
Table 4 shows the frequency of concepts mentioned in the interview with the persons with SCI but not included in the ICF-list used in this study. Some concepts can be linked to an existing ICF category and some categories remain undefined. Some are part of the WORQ part 1 or are covered in questions 41 and 42 of the WORQ (“Overall in the past week, how long did it take you...”).
Discussion
The aim of our study was to examine the content validity of the WORQ for use in persons with post-acute and chronic SCI. In our study, 80% of the ICF categories of the WORQ were mentioned by both sources. A previous study with persons with SCI in an early post-acute context showed similar results with 80% of WORQ ICF categories mentioned during interviews with persons with SCI [7]. In this previous study seven categories of the WORQ were not contained in any source [7], whereas in our study all categories were contained in at least one source. If an ICF category is part of the WORQ-NL, mentioned during the interview with a person with SCI and considered important by the SCI professionals, we can consider it undoubtedly as relevant for VR in persons with SCI. Four categories (d455 moving around, d570 looking after one’s health, d870 economic self-sufficiency, and e310 immediate family) were not mentioned by the SCI professional team but were mentioned in at least 3 interviews. The rehabilitation professionals considered categories such as “moving around” irrelevant. Potentially this can be explained as this category includes “running, climbing, jumping, etc.”, activities not directly related to employment, and does not include “walking or moving around using equipment”. The ICF categories mentioned by the SCI professionals only are categories often not considered problematic in SCI (e.g., “seeing functions”). In fact, all of those categories had a low median score (≤3). This does not necessarily mean that those categories are redundant for use in VR in persons with SCI. Professionals deemed these categories important possibly because they have a comprehensive view at the return to work process including the importance of underlying conditions e.g., seeing, sleeping and education whilst the persons with SCI were mainly focusing on their (previous) employment activities solely.
In our study furthermore, 14 ICF categories of the brief ICF core set for acute and/or chronic situation were mentioned in interviews with persons with SCI and considered important by the SCI professionals; however, they are not part of the WORQ, and therefore, they might be considered lacking for use in SCI. This is in line with a previous study where 9 categories were identified as lacking in the WORQ-SELF for use in SCI [7]. The common 5 lacking categories from both studies are: d410 changing basic body position; d445 arm and hand use; e115 products and technology for personal use in daily living; e120 products and technology for personal mobility and transportation and e150 design, construction, and building products and technology of buildings for public use. In this previous study the category “e155 design, construction, and building products and technology of buildings for private use” was furthermore mentioned as lacking in the WORQ. This item was mentioned in 6 interviews with persons with SCI, but was not considered important to the SCI professional team in our study. In hindsight the professionals in our study probably quoted this item as related to the “own private house” not considering the fact that possibly some persons work (partially) from home. Knowing that SCI often involves significant levels of impairment resulting in limited physical functioning and aid- and wheelchair dependence, our findings suggest that the above mentioned categories (see also Table 3) have to be considered when using the WORQ in VR for persons with SCI. The lacking categories describe typical aspects relevant for persons with SCI or severe physical disability. The WORQ may have been developed for the gross, not necessarily physically limited, population needing VR. In addition, the interviews in our study revealed several concepts not included in the ICF list but worthwhile considering for use in VR in persons with SCI. Some of those concepts belong to an existing ICF category and seem very specific to spinal cord injury e.g., b265 touch function, d415 sitting, and e135 products and technology for employment. The concept ‘time’ has been mentioned in 11 of the 14 interviews, however this concept is already present in the WORQ (questions 41 and 42 that are not part of the total score). Interestingly, the category “e425 Individual attitudes of acquaintances, peers, colleagues, neighbors, and community members” was mentioned in 6 interviews and, even if not SCI specific, could potentially be relevant for adding to the WORQ.
Some limitations were encountered during this study. Firstly, only fourteen of the 16 interviews were conducted, two persons with recent low complete SCI and one person with recent high complete SCI were missing. To compensate, one person with semi-recent low complete SCI was included. This resulted in a small sample of recent complete SCI and therefore the aimed variance in participants regarding level of SCI and time since injury was potentially not fully achieved. In addition, coding of the interviews was done in second-level ICF categories, which is less specific.
In conclusion, our study confirms that most categories of the WORQ are important to consider for VR in persons with SCI, however, there are ICF categories that are absent in the WORQ and deemed relevant for use in VR in persons with SCI. Consequently, the content validity of the WORQ without additional items is insufficient for persons with SCI.
For future use in clinical practice in VR of persons with SCI, we advise adding an SCI-specific appendix with the above-described lacking categories. In this way, additions can be developed to serve needs specific for diagnostic groups, and the WORQ core remains intact and can still benefit from compiling evidence resulting from future studies. Also, further research is warranted to investigate the value of using the additional categories described as lacking for the SCI population in this study.
Data availability
The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.
References
Meade MA, Reed KS, Saunders LL, Krause JS. It’s all of the above: Benefits of working for individuals with spinal cord injury. Top Spinal Cord Inj Rehabil. 2015;21:1–9.
DeVivo MJ. Causes and costs of spinal cord injury in the United States. Spinal Cord 2013;35:809–13.
Finger ME, Escorpizo R, Bostan C, De Bie R. Work rehabilitation questionnaire (WORQ): Development and preliminary psychometric evidence of an icf-based questionnaire for vocational rehabilitation. J Occup Rehabil. 2014;24:498–510.
World Health Organisation. International Classification of Functionning, Disability and Health:ICF. Geneva, Switzerland.; 2001.
Finger M, Escorpizo R, Glässel A, Gmünder H, Lückenkemper M, Chan C, et al. ICF Core Set for vocational rehabilitation: Results of an international consensus conference. Disabil Rehabil. 2012;34:429–38.
Selb M, Finger ME, Escorpizo R. Applying the Work Rehabilitation Questionnaire WORQ: A case illustrating its use in evaluating functioning of a person after a traumatic brain injury in an interprofessional vocational rehabilitation setting. In: Escorpizo R, Brage S, Homa D, Stucki G, editors Handbook of vocational rehabilitation and disability evaluation: application and implementation of the ICF (Dordrecht, Springer). 2015. p. 521–43.
Portmann Bergamaschi R, Escorpizo R, Staubli S, Finger ME. Content validity of the work rehabilitation questionnaire-self-report version WORQ-SELF in a subgroup of spinal cord injury patients. Spinal Cord. 2014;52:225–30.
Finger ME, Wicki-Roten V, Leger B, Escorpizo R. Cross-cultural adaptation of the WORK Rehabilitation Questionnaire (WORQ) to French: A valid and reliable instrument to assess work functioning. J Occup Rehabil. 2019;29:350–60.
Vermeulen K, Woestyn M, Oostra K, Geers S, Ryngaert K, Van De Velde K, et al. Cross-cultural adaptation and psychometric evaluation of the Dutch version of the work rehabilitation questionnaire (WORQ-VL). J Occup Rehabil. 2019;29:514–25.
Roels EH, Reneman MF, Post MWM. Measurement properties of the full and brief version of the WORK Rehabilitation Questionnaire in persons with physical dysabilities. J Occup Rehabil. 2021;31:886–94.
Husmann A, Escorpizo R, Finger ME. Examining work-related functioning in a physical therapy outpatient clinic: Validity and reliability of the work rehabilitation questionnaire (WORQ). J Occup Rehabil. 2020;30:156–66.
Burns S, Biering-Sørensen F, Donovan W, Graves D, Jha A, Johansen M, et al. International standards for neurological classification of spinal cord injury, Revised 2011. Top Spinal Cord Inj Rehabil. 2012;18:85–99.
Selb M, Escorpizo R, Kostanjsek N, Stucki G, Üstün B, Cieza A. A guide on how to develop an International classification of functioning, disability and health core set. Eur J Phys Rehabil Med. 2015;51:105–17.
Kirchberger I, Cieza A, Biering-Sørensen F, Baumberger M, Charlifue S, Post MW, et al. ICF Core Sets for individuals with spinal cord injury in the early post-acute context. Spinal Cord. 2010;48:297–304.
Cieza A, Kirchberger I, Biering-Sørensen F, Baumberger M, Charlifue S, Post MW, et al. ICF Core Sets for individuals with spinal cord injury in the long-term context. Spinal Cord. 2010;48:305–12.
Acknowledgements
Special thanks to M Hund for typing out the interviews and the interviewed members of the SCI rehabilitation team: G.G. van Laar, S.M. Teeninga, W.K.S. Smid, P.M.B. Habers, H. IJbema, V. Leseman and F.M. de Goei.
Author information
Authors and Affiliations
Contributions
EHR and CCI performed the data collection and analysis. EHR drafted the paper. MFR and MWMP provided feedback on the data collection, analyses, and paper.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Ethics statement
The Medical Ethics Committee of the University Medical Centre Groningen declared that this protocol did not need formal ethical approval under the Dutch law regulating medical research in human beings (reference number METc2019/243). We certify that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed during the course of this research. Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Roels, E.H., Schneider, C.C.I., Reneman, M.F. et al. Content validity of the Work Rehabilitation Questionnaire (WORQ) for persons with spinal cord injury: A mixed methods study. Spinal Cord 60, 354–360 (2022). https://doi.org/10.1038/s41393-021-00738-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41393-021-00738-x