Introduction

Across all areas of health care, policy decision makers are in pursuit of what Berwick and colleagues1 have called the “triple aim”: i.e., improving patient experiences with care, improving health outcomes in the population, and managing health system impacts. Pursuing these goals can be challenging in the context of rare diseases. Challenges relate to the development of and access to new therapies, the generation of knowledge to understand the epidemiology and natural history of the diseases, and the design of high-quality research needed to support effective and appropriate patient care.2,3 These challenges are highly relevant to the field of inborn errors of metabolism (IEMs).4,5,6

Toward the triple aim, health system investments to improve and protect health for people with IEMs and other rare diseases include (i) secondary prevention aimed at early disease detection and intervention (notably newborn screening); (ii) study of the biological basis of rare diseases to inform treatment, and development of facilities for their diagnosis (e.g., gene and biomarker discovery for rare diseases and diagnostic laboratories); (iii) programs to promote the development of, and ensure access to, treatments that would otherwise be prohibitively expensive; and (iv) coordination of primary and specialist care, including services such as multidisciplinary team care and the development of regional genetic treatment centers ( Figure 1 ). These system-level strategies are often predicated on the effectiveness of interventions at the level of individual patient care. For IEMs, these individual-level interventions may include orphan drugs, medical foods, dietary supplements, and organ replacement. Just as health system–level interventions are dependent on the existence of effective interventions at the patient level, clinical interventions by themselves cannot lead to improved outcomes unless patients have access to them in the context of an appropriate system of care ( Figure 1 ).

Figure 1
figure 1

Integrated system-level and clinical interventions for inborn errors of metabolism.

Full size image

Practice-Based Evidence

The interdependency of health system and clinical care interventions points to the potential usefulness of evaluating the effectiveness of health care for IEMs in an integrative way. Currently, recommendations for providing care to patients with IEMs often consist of consensus guidelines developed through expert working groups, e.g., refs. 7 and 8 or through Delphi processes.9 Such guidelines are directly informed by existing evidence and by the opinions of experienced clinicians. However, they are limited by the scarcity of empirical data to support the effectiveness of interventions at both health system and clinical levels, in part, due to the challenges associated with gathering such evidence in the context of rare diseases.6,10 For example, randomized controlled trials in rare diseases require significant resources and multicenter collaboration11,12 yet, particularly with frequentist approaches, often remain underpowered to provide the evidence sought. Of note, traditional randomized controlled trials do not account well for clinical heterogeneity, a key feature of many IEMs.13,14 Alternative experimental and quasiexperimental study designs have been developed for rare diseases, but there are methodological challenges associated with each.15,16,17 Moreover, experimental designs do not allow the simultaneous evaluation of multiple interventions at the levels of the health-care system and individual patient care.

Observational studies may provide important knowledge by taking advantage of natural experiments, the opportunities afforded by differences in clinical practices, and/or policy-level interventions over time and across providers or jurisdictions, to investigate associations between different interventions and outcomes. Although such studies are susceptible to confounding and to bias due to differential ascertainment of cases, investigators can mitigate the impact of these biases by standardizing measures of key variables, achieving near complete case ascertainment, and evaluating the representativeness of the included cases. Indeed, in recognition of the need to conduct clinical evaluative research in real-world settings to understand the complex nature of care, several authors have described the concept of “practice-based evidence,”18,19 recently defined as research that “… accommodates multiple concurrent interventions and patient characteristics that reflect actual clinical practice, using data from natural settings to describe the content and timing of treatments that are associated with better outcomes (including patient reported outcomes) for patients with specific characteristics.”18

This definition of practice-based evidence is highly relevant to the study of IEMs. Its consideration of interventions within a system of care accommodates the need for an integrative approach, and it is broad enough to incorporate relevant outcomes across the triple aim, including patient/family perspectives and impact on the health system in addition to clinical end points. We suggest that a practice-based evidence approach in outcomes research for IEMs should not be restricted to observational research, given that randomized controlled trials and other experimental and quasiexperimental approaches remain fundamental to understanding treatment efficacy. Rather, observational research can provide detailed information on the epidemiology of IEMs and help to inform priorities for the development of trials. And in turn experimental methods can identify interventions that are efficacious in a controlled environment, which then need to be evaluated in an embedded system of care across a range of patients and settings, to determine who may benefit, in what ways, and under which circumstances.19

An Outcomes Research Network

A practice-based evidence approach to evaluative research for IEMs requires, at a minimum, the development of a network of practice settings with standardized collection of data capturing relevant outcomes and interventions.18 Such a network would support the strategy of multicenter collaboration and pooling of data that has been advocated to improve the robustness of research for rare diseases,6,12,20,21 incorporating the strengths typically associated with patient registries, such as complete coverage of patient populations (to avoid selection bias), standardized case definitions, and longitudinal data collection. The resources that are needed to develop this network and data collection system highlight the importance of ensuring that sufficient information is collected to answer the most important research questions while also having the flexibility to accommodate emerging questions in a rapidly evolving field. Thus, we suggest conceptualizing a practice-based evidence network not as a traditional registry but as an endeavor that is explicitly focused on outcomes evaluation, guided by a research framework to identify key questions and ensure that relevant data are captured.

In Canada, the federal government, through a Canadian Institutes of Health Research grant has recently invested in an Emerging Team to develop such a national network and the information system needed to support it. The Canadian Inherited Metabolic Diseases Research Network (CIMDRN) includes investigators with expertise in clinical care of patients with IEMs, as well as in epidemiology, health services and policy research, and health economics. We are supported by an advisory board whose members include patient and family representatives and scientists in related fields. All children diagnosed with IEMs in Canada receive care at one of 16 regional metabolic treatment centers. Members of our network are located at nearly all these centers, which should permit complete ascertainment of Canadian cases. We will collect longitudinal clinical data, patient/family-reported data, and health services use data for all children enrolled in our research program.

The CIMDRN joins a number of other initiatives that have been developed to improve health care and outcomes for patients with IEMs and related rare diseases internationally, for example, the projects within the Europe-based Network for Genetic Metabolic Diseases Detectable by Newborn Screening (Metabnet),22 and several collaboratives based in North America ( Table 1 ). Of note, the consortia established by the Rare Diseases Clinical Research Network focus on the creation of disease-specific clinical contact registries that primarily act as tools to support the enrollment of patient cohorts in research studies for specific diseases.23,24,25 Other groups have focused on longitudinal surveillance of outcomes among patients with a broader range of IEMs diagnosed through newborn screening, for example, the Inborn Errors of Metabolism Information System developed by Region 4 of the Genetic and Newborn Screening Service Collaboratives.26 This work is now connected to the Inborn Errors of Metabolism Collaborative, a research project supported by the Newborn Screening Translational Research Network.27 Still other groups investigate specific interventions, for example, the Nutrition and Dietary Supplement Interventions for Inborn Errors of Metabolism;28 or study the development of new or newly applied therapies, such as the collaborative National Center for Advancing Translational Sciences.29

Table 1 Key North American examples of related initiatives that support research to improve outcomes for inborn errors of metabolism
Full size table

The CIMDRN shares with these initiatives an evaluative perspective and, in common with registries and newborn screening long-term follow-up programs, a focus on prospective data collection from a complete cohort of patients. However, our network brings forward several innovative components. For example, our emphasis on the interdependence of system-level and clinical-level interventions and on outcomes that encompass medical, patient-centered, and system perspectives (the triple aim) broadens our work beyond individual therapies for individual diseases toward the generation of knowledge that will generalize across IEMs and to other rare diseases. We have also developed a novel practice-based evidence framework (described below) that will serve to guide both the data we collect and the research questions that our network will pursue.

A Practice-Based Evidence Framework

The CIMDRN research framework emphasizes the full range of outcomes relevant to the triple aim ( Figure 2 ), so as to make explicit the trade-offs that are inevitable in a rare disease environment. For example, some interventions with modest clinical benefits may be highly valued by patients and families yet expensive for the health-care system (e.g., enzyme replacement therapy). Other interventions may be relatively inexpensive from the health system perspective, yet experienced as burdensome or anxiety-provoking for patients and families (e.g., severe dietary restrictions may fall into this category). Studying these impacts simultaneously will help us to understand their implications with respect to the value assigned to outcomes in clinical and policy decisions.

Figure 2
figure 2

A practice-based evidence research framework for inborn errors of metabolism. IEM, inborn error of metabolism; MCADD, medium-chain acyl-CoA dehydrogenase deficiency; MPS1, mucopolysaccharidosis 1; PKU, phenylketonuria; VLCADD, very long-chain acyl-CoA dehydrogenase deficiency.

Full size image

We have also identified three themes that capture the types of evaluative research questions that are most relevant at both the level of individual patient care and the health system for IEMs and that can serve to integrate research at these levels ( Figure 2 ). To demonstrate how these themes may be used to identify and pursue high-priority research questions both within and across IEMs, two illustrative case studies or examples for each theme are described below and in Figure 2 . Because phenylalanine hydroxylase deficiency (OMIM 261600) is a more common and particularly well-studied IEM, one example for each theme is based on this disease.

Theme 1: Tailoring care in the context of clinical heterogeneity

IEMs, like many rare diseases, are frequently characterized by heterogeneity in clinical presentation, ranging from mild forms that may require no intervention to severe forms that are associated with significant morbidity and/or mortality. For many IEMs, recognition of this heterogeneity has increased since their inclusion in newborn screening programs: their apparent prevalence has increased, possibly due to undiagnosed early fatal cases in nonscreened populations and the diagnosis of mild or asymptomatic cases in screened populations.30 There is uncertainty about the natural history of many IEMs. Correlations among genotype, biochemical phenotype, and clinical symptoms are often weaker than predicted, highlighting the roles of other factors, including social characteristics, in influencing outcomes. Clinical providers face challenges in predicting patients’ prognoses and in developing ways to customize care. Heterogeneity and the need for individualized care also create challenges in designing effective health system investments that can meet the needs of a diverse population of patients and target therapies to those most likely to benefit. In this way, IEMs exemplify the challenges of personalized medicine.31 Research questions suggested by this theme, therefore, relate to identifying predictors of disease prognosis (including biomarkers that have been validated using outcomes that are important to patients), understanding whether and how care is tailored to account for differences in disease severity, and generating evidence to facilitate decisions about personalizing care.

Clinical heterogeneity, example 1: When does non-PKU hyperphenylalaninemia require treatment? Phenylalanine hydroxylase deficiency is associated with intolerance to dietary intake of the amino acid phenylalanine and results in a spectrum of disorders ranging from classical PKU to non-PKU hyperphenylalaninemia (non-PKU HPA) ( Figure 2 ).32 Classical phenylketonuria (PKU) is usually defined as an untreated blood phenylalanine concentration of >1,200 µmol/l or low tolerance to dietary phenylalanine.32,33 Non-PKU HPA describes individuals whose phenylalanine levels are above normal (>120 µmol/l) but typically <600 µmol/l (although some centers define the upper limit for non-PKU HPA as <360 µmol/l, with higher levels defined as PKU). The vast majority of untreated children with classical PKU show impaired cognitive development. The generally accepted goal of treatment, primarily through dietary therapy, is reduction of blood phenylalanine into the therapeutic range (often defined as 120–360 µmol/l) and thus prevention of cognitive deficits. However, an issue of debate is whether patients with mild non-PKU HPA need to be treated.34,35 Specifically, it is unclear whether those whose blood phenylalanine levels are elevated but consistently below 600 µmol/l are at risk of intellectual impairment. There is variation in the classification and treatment of non-PKU HPA,36 with implications for patients, families, and the health-care system. For example, in Canada, costs for therapy per patient can be $20,000 to >$100,000/year. There is a need for research to better understand the association between the choice of blood phenylalanine cutoff for initiation of treatment and outcomes that include neuropsychological development, health-related quality of life, and costs of care.

Clinical heterogeneity, example 2: What patient characteristics and interventions are associated with outcomes among individuals with “nonclassic” medium-chain acyl-CoA dehydrogenase deficiency or very long-chain acyl-CoA dehydrogenase deficiency? Medium-chain acyl-CoA dehydrogenase deficiency (OMIM 607008) and very long-chain acyl-CoA dehydrogenase deficiency (OMIM 609575) are fatty acid β-oxidation disorders characterized by impairments in energy production, particularly during times of increased energy demands and/or decreased dietary intake ( Figure 2 ).37 Patients are at risk of acute metabolic decompensation under conditions of metabolic stress (often caused by viral infection), with significant morbidity and risk of mortality.37,38 Individuals with very long-chain acyl-CoA dehydrogenase deficiency may also experience chronic sequelae (e.g., cardiac and neuromuscular morbidity).39 Treatment includes avoidance of fasting and may also include specific diet modifications and/or the use of supplements and medications. Newborn screening has resulted in the identification of a higher proportion of mild and potentially asymptomatic cases for both diseases.30,40,41 For “nonclassic” cases, there is uncertainty about how best to establish the diagnosis, predict prognosis, and manage care,9,42,43 with evidence of variation in treatment practices.44 Research priorities include developing a better understanding of the distribution of outcomes (e.g., metabolic decompensation, cardiac manifestations for very long-chain acyl-CoA dehydrogenase deficiency, parental stress) and appropriate treatment for children with known high-risk disease versus those with a less certain prognosis, and identifying the ways in which patient characteristics and interventions interact to predict outcomes in these groups.

Theme 2: A shift from “urgent care” to “opportunity for improvement”

Traditional treatment goals for IEMs focused on preventing catastrophic outcomes such as death, irreversible neurologic injury, and mental retardation. Similar to more common chronic diseases (e.g., diabetes, HIV), an improved understanding of the biology of IEMs and corresponding development of new treatments has meant that many patients are surviving with fewer severe sequelae. New priorities include identifying optimal long-term outcomes and determining the intermediate indicators of disease management (e.g., validated biomarkers of metabolic control) that best predict such outcomes and can be used to monitor treatment. In parallel, there is a need to consider the potential for late adverse effects of interventions, recognize new natural histories (revealed due to longer survival), and identify ineffective interventions. Thus, within this theme, we suggest prioritizing research that illuminates the appropriate goals of therapy when existing evidence suggests that established treatments improve but do not optimize outcomes, explores ways to define and monitor successful disease management in the short-term with longer term outcomes in mind, and addresses the benefits and risks of therapies over a long-term follow-up period.

Shift, example 1: Among patients with PKU, what individual characteristics and interventions are associated with patient-centered outcomes beyond intelligence quotient? Early identification by newborn screening with initiation of dietary therapy has dramatically reduced the prevalence of mental retardation among children with PKU ( Figure 2 ). However, despite measured intelligence quotient in the normal range, individuals with treated PKU may have less than optimal psychosocial and neurodevelopmental outcomes,45,46,47 and there is little evidence regarding the impact of PKU on physical development.48 The goals of modern therapy, beyond preventing mental retardation, thus remain uncertain.49 Blood phenylalanine levels correlate well with intelligence quotient in individuals with PKU,50 but there is a need to better understand the complex relationships among patient characteristics, adherence to dietary and other interventions, blood phenylalanine levels, and additional patient-centered outcomes, including executive function and quality of life.45,47

Shift, example 2: In patients with mucopolysaccharidosis 1, what are the long-term outcomes associated with early hematopoietic stem cell transplantation? Mucopolysaccharidosis 1 (Hurler disease, OMIM 607014) is a lysosomal storage disorder characterized by impaired metabolism of glycosaminoglycans, leading, in classic cases, to severe progressive neurocognitive, cardiorespiratory, and musculoskeletal impairments and shortened life expectancy ( Figure 2 ).51,52 There is evidence of clinical benefit with early hematopoietic stem cell transplantation.52,53 Enzyme replacement therapy is also used for some patients, although neurodevelopmental benefit remains unclear.52 However, there are unresolved questions about the long-term benefits and safety of hematopoietic stem cell transplantation,54,55,56 including the association of hematopoietic stem cell transplantation with outcomes including musculoskeletal impairment, neuropsychological development, child health-related quality of life, and possible late effects.

Theme 3: Need to evaluate comparative effectiveness of emerging and established therapies

Challenges in determining the effectiveness of treatments for IEMs relate to their rarity and heterogeneity, the complex nature of care, and the rapid pace of development of new treatments. There is a need for ongoing evaluation of new treatments while comparing them to established treatments using metrics capturing patient experiences, clinical outcomes, and health system impacts, including cost effectiveness. Linked to strategic investments in patient-oriented research, comparative effectiveness is a current priority for both rare and common diseases.57 As described, multiple complementary approaches to comparative effectiveness research will be needed to provide the best evidence to inform care for IEMs. Priority research questions could therefore address the effectiveness and cost effectiveness of emerging therapies as compared with standard care, both in experimental settings and in a clinical context that considers cointerventions, patient-centered outcomes, and system-level policies regarding access to care. Because existing frameworks for assessing the effectiveness and cost effectiveness of interventions are unlikely to be favorable toward rare diseases, practice-based research within this theme could incorporate alternative frameworks developed specifically for the rare disease context.58

Comparative effectiveness, example 1: Among patients with PKU, how does sapropterin compare with diet therapy alone in its association with a range of outcomes? An emerging therapy for individuals with PKU is the drug sapropterin dihydrochloride (tetrahydrobiopterin) ( Figure 2 ).59 Evidence to date, including clinical trial data, suggests that a subset of individuals with PKU respond to sapropterin with decreases in blood phenylalanine and/or an increase in dietary tolerance to protein.59,60,61,62,63,64 However, there are unresolved questions about predictors of response to sapropterin, the degree of cointervention with dietary therapy needed (although sapropterin to date has been licensed only as an adjuvant therapy and not as a replacement for dietary intervention), and, in particular, the association of sapropterin use with clinical outcomes,60,65 including neuropsychological outcomes, quality of life, and experiences with care. There is also a need for research to understand the costs of care to families and to the health-care system.

Comparative effectiveness, example 2: For hepatic IEMs, what are the comparative outcomes associated with early liver transplantation versus aggressive nutritional management? There is a diverse group of IEMs for which liver transplantation is being used in some cases because the deficient enzyme for the implicated metabolic process is hepatic ( Figure 2 ). For example, urea cycle defects are characterized by impaired excretion of nitrogen from protein catabolism, resulting in hyperammonemia.66 Maple syrup urine disease (OMIM 248600) is caused by impairment of the degradation pathway for branched-chain amino acids and associated ketoacids.67 Patients with classic forms of urea cycle defects and maple syrup urine disease are at a high risk of metabolic decompensation during periods of catabolic stress and may experience additional manifestations including death and/or neuropsychological impairment.67,68 Emerging evidence supports the effectiveness of liver transplantation in stabilizing metabolite levels and/or increasing dietary tolerance.69,70,71,72 However, there is a need to systematically compare the benefits and risks of early transplantation versus aggressive nutritional and medical interventions, which are also rapidly evolving.72,73 In addition to informing health care for hepatic IEMs, such research would inform policy debates regarding the prioritization of patients for liver transplantation and optimal use of this expensive and high-risk treatment.74

Conclusion

We have described a practice-based evidence approach to outcomes research in the field of IEMs, including a research framework that can be used to identify and pursue high-priority questions related to the effectiveness of interventions at multiple levels. The tools we have described and that will be used by the CIMDRN have the potential to promote evidence-informed care that is sensitive to current challenges in the care of patients with IEMs (for example, clinical heterogeneity, the rapid pace of development of new interventions) and that focuses on the most relevant outcomes from the perspectives of patients and families, providers, and health-care systems. The CIMDRN’s information system will initially emphasize the collection of existing observational data relevant to our research framework. However, the network is also well positioned to initiate new clinical intervention studies with designs tailored for rare disease settings, and to collect biological samples that could be used to identify biomarkers for use in predicting patient prognosis and as surrogate end points in monitoring disease management. These biomarkers could be validated using outcome information we collect across the triple aim, emphasizing end points valued by patients and families while accounting for important patient and environmental characteristics. Because the Canadian health-care system provides universal insurance coverage for most health services, the CIMDRN is also uniquely positioned to study IEMs using a combination of clinical and health-care administrative data, which we will supplement with patient- and family-reported information. Finally, the ethnic diversity of the Canadian population and the existence of a number of isolated communities will permit the study of disease prevalence in different population subgroups, which may inform the development of targeted interventions.

Ultimately, the CIMDRN aims to (i) produce clinical and policy guidance to improve patient experiences and outcomes for individuals with specific IEMs and (ii) generate new knowledge to inform the design and delivery of health services for patients with a broad range of IEMs and other rare diseases and to provide insights into patient-centered and personalized care. We suggest that the ideas and framework presented here are also generalizable internationally and can inform other groups pursuing similar objectives (e.g., through the clinical registries and newborn screening long-term follow-up programs described in Table 1 ). Personalized health care has been defined as “…a forward-looking, consumer-centric system that features customized diagnostic, treatment, and management plans based on a variety of patient factors, including culture, personal behaviors, preferences, family health history, and genetic/genomic makeup.”75 It is about identifying the best interventions at the appropriate times for the appropriate patients; this is our vision for the future of care for IEMs.

Disclosure

The authors declare no conflict of interest.