Patient-reported outcomes are increasingly collected in clinical trials and in routine clinical practice, but strategies must be taken to include underserved groups to avoid increasing health disparities.
Patient-reported outcomes (PROs) collected in clinical trials can provide valuable evidence of the risks and benefits of treatment from a patient perspective, to inform regulatory approvals, clinical guidelines and health policy. PROs are increasingly collected routinely in clinical settings, at an aggregate level for audit and benchmarking, for real-world evidence generation, and as an input or predicted output for clinical decision tools and artificial intelligence (AI) in health1,2. At an individual patient level, PROs can be used to facilitate shared decision making, screen or monitor symptoms, and provide timely care tailored to individual needs3. PROs are also increasingly used in value-based healthcare initiatives4.
Efforts to capture and report PRO data should be inclusive and equitable, addressing the diverse needs of all patients with the condition of interest, including groups historically and currently underserved by research5,6. Issues of diversity, equity and inclusion (Box 1) have recently been highlighted in PRO ethical guidelines, which have identified a number of concerns to be addressed in PROs research5.
Lack of representation
Underserved groups are often poorly represented in research and may receive suboptimal clinical care due to a range of cultural, socioeconomic and logistical reasons, in addition to narrowly defined inclusion criteria for research. Lack of representation is compounded by historical mistrust of research and medical institutions that persists in many groups.
PROs can provide valuable evidence of the efficacy and safety of drugs and biologics, which can vary depending on intrinsic and extrinsic factors, including sex, race, ethnicity and age. Clinical trials should provide information that informs the use of therapeutic agents within the target population. However, despite regulatory guidance and public expectations, the composition of study populations in most clinical trials does not always reflect such characteristics, which limits analysis of treatment outcome by subgroup. This failure to achieve meaningful diversity limits information about drug response and measures of safety and efficacy, which may result in health data poverty (Box 1)7. In this context, clinical trial results — and PRO data specifically — become biased as they are limited to those populations involved in research, with sectors of the population excluded, or even harmed, as a result. Lack of representative PRO data collection limits understanding of the impact of disease or treatment on patients’ symptoms and quality of life, and thus the evidence base on which to provide clinical care, make regulatory decisions and inform health policy. This Comment considers current challenges related to PRO data collection in underserved groups and identifies approaches for greater inclusion.
Barriers to completion
With an increasing focus on PRO data collection to support patient-centered care, it is essential that the needs of underserved groups are addressed (Box 1). A key barrier to PRO data collection in underserved groups is a lack of valid and reliable measures that have been developed in, or are salient to, the target population. Many PRO measures are developed with limited patient input and may not address concepts that matter to underserved groups. Even when individuals from underserved groups are invited to complete PRO measures, they may experience significant barriers to PRO data completion. Individuals with disabilities, such as sight impairment, arthritis or cognitive function, and those in poor health may find completing the measures burdensome or challenging6. People with learning disabilities and low literacy have experienced exclusion from the routine monitoring of their health and wellbeing afforded by PROs8.
Importantly, the move to electronic PRO collection, while helpful for some, has created new barriers for others. Barriers to digital inclusion are widespread in underserved populations, with poor accessibility arising from a range of issues (Box 1). Estimates suggest that 37% of the world’s 7.8 billion population are digitally excluded, with older people, people on low incomes and other marginalized groups most likely to be affected9.
A recent study investigating the incorporation of PROs in clinical trials demonstrated that certain patient groups are not represented10. Investigators examined PRO capture across ten National Clinical Trials Network oncology trials and found that 24.7% of study participants declined to complete the PROs, and that 62.2% of the participants who agreed to the PRO component declined electronic PRO capture. Racial or ethnic minorities, those with less education and older patients were less likely to consent to electronic PRO collection.
AI health technologies trained and tested on PRO datasets that do not include members of these underserved populations are increasingly being utilized in healthcare. There is a risk that individuals from these groups may systemically receive suboptimal care as a result11.
Racial and ethnic disparities
Specific challenges have been identified in the inclusion of minority ethnic groups in research and with the use of translated and culturally validated PROs12,13. A review of ethnicity reporting and PRO use of cancer trials registered in the National Institute for Health Research portfolio found that only 14 out of 84 (17%) trials collecting PROs reported ethnicity data. Eight (57%) studies were multicentered, multinational trials and the remaining were UK based (43%), suggesting a diverse target population; however, none reported using translated PRO measures even when available13.
Online collection of PROs may lead to profound racial disparities, as highlighted by Mass General Brigham’s PRO data collection, which spans 10 hospitals, 200 clinics and more than 75 specialties in the United States14. Before the COVID-19 pandemic, only 17% of PROs were collected using an online patient portal, with the remainder collected via tablet in clinic14. PRO completion rates were equitable, irrespective of self-identified race or ethnicity as recorded within the electronic health record. In March 2020, all tablets used for PRO collection were removed from clinics to limit the spread of COVID-19. This rapid transition prompted a shift in the capture of PROs, from primarily in-clinic to use of the online portal; this shift introduced profound disparity in data collection. Patients who self-identified as Black provided PROs at half the rate of white patients, and patients who identified as Hispanic almost stopped completing PROs altogether14.
Low- and middle-income countries
Further consideration should be given to PRO data collection in low- and middle-income countries (LMICs). Participants from LMICs tend to be underrepresented in the development of PRO measures and there are also indications of a correlation between economic development and research participation, whereby PRO research is more likely to be conducted in upper-middle-income economies, such as Brazil, Russia, India, China and South Africa, than in low-income economies15. The challenges of conducting PRO research in LMIC settings include lower literacy levels, which require the use of interview administered questionnaires that can in turn introduce bias; variable adherence to standardized protocols for conducting randomized clinical trials; and cultural diversity. Such challenges require particular attention from research funders and investigators when designing, budgeting and conducting research. Outcomes should be culturally relevant and practical aspects of data collection must be carefully considered for each context.
A growing number of LMICs are proactively looking at collecting and using local evidence to strengthen their healthcare decision-making processes, as a core strategy for progressing towards universal health coverage. A stronger focus on collecting PRO data in LMICs presents a valuable opportunity to entrench patients’ perspectives in the health policy discourse.
Barriers to participation in PRO completion, such as access to technology, disability, language and cultural requirements, should be addressed both in the interests of fairness and to ensure results are as accurate and generalizable as possible. Resources required to widen participation should be considered, for example, costs of alternative modes of PRO administration, addressing accessibility requirements, and development of culturally relevant translations.
Existing good practice guidance, such as minimizing participant burden, streamlining PRO administration and using PRO alerts, can be effectively used to promote inclusion and accessibility5. Communication of the rationale for PRO assessment (who will access the data and how it will be used) to potential participants may address the concerns of those wary of participating in research or providing information in a routine care setting. The representation and participation of underserved groups in PROs can be increased through the actions in Table 1.
Involvement promotes recruitment
Patient and public input are central to ensuring that PRO research is inclusive, equitable and meets the needs of diverse groups. Input can be facilitated by engaging diverse patient partners in co-design, and the involvement of study cohorts that are representative of the full breadth of the target population. Patients that are representative of the target population should be involved in the identification of concepts that matter to them and should contribute to the selection and/or development of PRO measures16.
Representativeness in involvement activities can be achieved by addressing barriers that reduce the diversity of contributors, including engagement through community groups, charities and support groups; ensuring that opportunities to get involved are appropriately timed and located; and reimbursement for reasonable expenses. In drug development, a commitment to incorporate diversity and inclusiveness as part of patient-focused drug development efforts is necessary. Early engagement with regulatory agencies is recommended as they can offer advice and support to promote inclusivity.
The aims and benefits of completing PRO measures should be conveyed to participants, with flexibility in the modes of delivery, to increase engagement and participation of individuals from diverse groups8. An equity checklist, such as Benkhalti and colleagues’ checklist to guide equity considerations in health technology assessment, can be an effective tool17.
Empowering participants from underserved groups to inform the design and delivery of PROs allows for the identification and mitigation of barriers to successful PRO implementation17. PRO measures must be accessible if individuals are to accurately communicate information about their health18. User-centred design (Box 1), including usability testing, can help identify the needs of the target group(s) and create functional tools for patients and providers6.
User-centered design principles can also accommodate people with visual impairment, limited mobility, learning disabilities, low health literacy or numeracy, including the ability to interpret graphical representations of data6. Digital inclusion should always be considered, including alternative modes of delivery such as bring your own device, assistive technologies or alternative modes of administration — for example, mail or telephone, including interviewer or interactive voice response (Box 1). Participants may need physical help to turn pages or hold a pen, or assistance with a telephone or computer keyboard. PRO collection that involves participants with different languages requires the availability of validated language and culturally adapted PRO questionnaires.
Practitioners must be sensitive to recognizing when proxy-reported measures may be needed, for example with advancing cognitive decline, to ensure accurate representation of a person’s health and functioning18. However, it is important to note that in a regulatory setting, use of such measures is discouraged and so early engagement and advice from regulatory agencies is recommended.
Improve clinical care for all
PRO measures and data collection must be reflective of diverse and multicultural societies, to improve research and promote equitable clinical care for the benefit of all patients and the public as a whole. Representative diversity in clinical trials is vital to ensure that all new medicines and technologies that reach the market are applicable to all the population subgroups that they are intended to serve. Targeted initiatives are needed to ensure that no groups are excluded from participation in PRO data collection, both in research settings and routine clinical care.
Inclusion of underserved populations in PRO data collection will help promote equitable healthcare and reduce health data poverty. Co-design of systems with representative patient input will be central to their successful realization. Resource implications must be considered, and novel approaches evaluated, to promote shared learning and best practice.
Virji, A. Z. et al. NEJM Catalyst Innov. Care Deliv. 2, https://doi.org/10.1056/CAT.21.0043 (2021).
Wu, A. W. et al. J. Clin. Epidemiol. 66, S12–S20 (2013).
Calvert, M. et al. BMJ 364, k5267 (2019).
Stamm, T. et al. NEJM Catalyst Innov. Care Deliv. 2, https://doi.org/10.1056/CAT.21.0146 (2021).
Cruz Rivera, S. et al. Nat. Med. 27, 572–573 (2021).
Lavallee, D. C. et al. Health Affairs 35, 575–582 (2016).
Ibrahim, H. et al. Lancet Digit. Health 3, e260–e265 (2021).
Jahagirdar, D. et al. BMC Health Serv. Res. 12, 431 (2012).
Kunonga, T. P. et al. J. Med. Internet Res. 23, e25887 (2021).
Pugh, S. L. et al. Int. J. Radiat. Oncol. Biol. Phys. 108, 950–959 (2020).
Seyyed-Kalantari, L. et al. Nat. Med. 27, 2176–2182 (2021).
Braveman, P. et al. J. Epidemiol. Commun. Health 57, 254–258 (2003).
Slade, A. L. et al. Trials 22, 306 (2021).
Sisodia, R. C. et al. J. Am. Med. Inform. Assoc. 28, 2265–2268 (2021).
Masyuko, S. et al. PLoS ONE 16, e0245269 (2021).
Patient-Focused Drug Development: Collecting Comprehensive and Representative Input (FDA, 2018).
Benkhalti, M. et al. Int. J. Technol. Assess. Health Care 37, e17 (2021).
Hughes, S. et al. J. R. Soc. Med. 114, 381–388 (2021).
Wild, D. et al. Value Health 8, 94–104 (2005).
FDA. Enhancing the Diversity of Clinical Trial Populations — Eligibility Criteria, Enrollment Practices, and Trial Designs Guidance for Industry. (2020).
NHS. What We Mean by Digital Inclusion https://digital.nhs.uk/about-nhs-digital/our-work/digital-inclusion/what-digital-inclusion-is (2021).
INVOLVE. Strategies for Diversity and Inclusion in Public Involvement: Supplement to the Briefing Notes for Researchers (2012).
FDA. Guidance for Industry — Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (2009).
NIHR. Improving Inclusion of Under-Served Groups in Clinical Research: Guidance from INCLUDE Project (2020).
Hurst, L. et al. Defining Value-based Healthcare in the NHS (CEBM, 2019).
M.J.C. receives funding from the National Institute for Health Research (NIHR), UK Research and Innovation (UKRI), NIHR Birmingham Biomedical Research Centre, the NIHR Surgical Reconstruction and Microbiology Research Centre, NIHR ARC West Midlands, UK SPINE, European Regional Development Fund – Demand Hub and Health Data Research UK at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Innovate UK (part of UKRI), Macmillan Cancer Support, UCB Pharma, Janssen, GSK and Gilead. M.C. has received personal fees from Astellas, Aparito Ltd, CIS Oncology, Takeda, Merck, Daiichi Sankyo, Glaukos, GSK and the Patient-Centered Outcomes Research Institute (PCORI) outside of the submitted work. In addition, a family member owns shares in GSK. O.L.A. receives funding from the NIHR Birmingham BRC, West Midlands, Birmingham, NIHR ARC West Midlands, UKRI, Health Foundation, Janssen, Gilead and GSK; and declares personal fees from Gilead Sciences Ltd, Merck and GSK outside of the submitted work. S.E.H. receives funding from the NIHR Applied Research Collaboration (ARC) West Midlands and UKRI; and declares personal fees from Cochlear Ltd and Aparito Ltd outside of the submitted work. A.M.S. has received unrelated consulting fees or speaking honoraria in the last 24 months from Navigating Cancer, Association of Community Cancer Centers, Pfizer, Genentech, Purchaser Business Group on Health and Henry Ford Cancer Center; and receives unrelated research funding from PCORI, NIH, AHRQ, Bladder Cancer Advocacy Network, Hematology/Oncology Pharmacy Association (HOPA), Cancer and Aging Research Group (CARG), Sivan Innovation and UroGen Pharma Ltd. C.M. receives funding from NIHR Surgical Reconstruction and Microbiology Research Centre (SRMRC), UKRI, and declares personal fees from Aparito Ltd outside the submitted work. G.T. receives funding from a NIHR Postdoctoral Fellowship Award. N.E.A. receives funding from a NIHR Clinical Doctoral Research Fellowship Award; and declares personal fees from GSK outside of the submitted work. G.V. was past president of ISOQOL, the European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Group; and has received consultancy fees from Roche, Eisai, Novartis, Seattle Genetics and Sanofi related to her clinical work in breast cancer and grants from the Yorkshire Cancer Research, Pfizer, IQVIA, Breast Cancer Now and the EORTC Quality of Life Group. P.K. is an employee of Merck Healthcare KGaA, Darmstadt, Germany. All other authors declare no competing interests. The views expressed in this article are those of the author(s) and not necessarily those of the NIHR, Medicines and Healthcare products Regulatory Agency, Health Research Authority or the Department of Health and Social Care.
About this article
Cite this article
Calvert, M.J., Cruz Rivera, S., Retzer, A. et al. Patient reported outcome assessment must be inclusive and equitable. Nat Med 28, 1120–1124 (2022). https://doi.org/10.1038/s41591-022-01781-8
This article is cited by
Levels of health literacy among people living with HIV in outpatient care: a cross-sectional study from Denmark
AIDS Research and Therapy (2023)
Journey from an Enabler to a Strategic Leader: Integration of the Medical Affairs Function in ESG Initiatives and Values
Pharmaceutical Medicine (2023)
International Society for Quality of Life Research commentary on the US Food and Drug Administration draft guidance for industry on core patient-reported outcomes in cancer clinical trials
Quality of Life Research (2023)
Using Patient Perspectives to Inform Better Clinical Trial Design and Conduct: Current Trends and Future Directions
Pharmaceutical Medicine (2023)
The Assessment of Patient-Reported Outcomes for the Authorisation of Medicines in Europe: A Review of European Public Assessment Reports from 2017 to 2022
Applied Health Economics and Health Policy (2023)