Introduction

Risk assessment based on the combination of personal and family health history is an effective and essential component of preventive health and is recommended by numerous medical organizations and guideline-making bodies.1 Examples of risk-management strategies predicated on risk assessment include breast magnetic resonance imaging (MRI), early (age <50 years) colon cancer screening, and BRCA genetic testing. Unfortunately, widespread adoption of risk assessment in clinical care has been hindered by challenges associated with conducting and acting on risk assessment, including data collection, risk calculation, synthesis of an action plan, and an underappreciation of the value of family health history.2,3

Several computerized data-collection and risk-assessment programs have been developed to help combat these challenges. Examples include Health Heritage,4 MyFamily,5 Family Healthware,6 HughesRiskApps,7 and MeTree.8 Studies of these and other nonelectronic risk-assessment aids have consistently shown the ability of such tools to increase the identification of individuals who meet guideline criteria for “increased” risk disease management strategies. For the purposes of this article, “increased risk” includes both those at high risk and those between population and high risk. Studies of these tools have shown that 42–82% of the general population are at increased risk for at least one condition depending on the number and type of diseases included in the risk assessment, and most do not know that they are at risk.4,6,9,10,11

The improved identification of increased-risk individuals suggests that there could be a clinically useful role for risk-assessment tools like these, particularly if they are coupled with clinical decision support (CDS). However, prior to promoting widespread integration within primary care, questions about integration into clinical workflow and clinical utility should be addressed. Workflow questions should address patients’ willingness to enter their data (for patient-facing tools), the quality of the data entered, and the accessibility of the CDS output for physicians (particularly important in electronic medical record–enabled practices), whereas clinical utility questions should address the ability to identify previously unidentified increased-risk patients and the change in the proportion of these receiving appropriate risk management. This last measure is complex because it depends on a change in how both the provider and the patient view the patient’s risk and the steps they take to mitigate that risk.

We have previously published articles on the clinical validity, workflow, and some clinical utility–related outcomes for MeTree,8,9,12,13,14 a Web-based patient-facing risk-assessment and CDS program integrated into a community-based primary-care setting. The clinical utility outcomes previously reported focused on MeTree’s ability to identify formerly unidentified primary-care patients at increased risk. In this article, we further assess clinical utility by evaluating its impact on the risk-management care that an increased-risk patient receives. To do so, we evaluated the agreement between a patient’s risk level and his or her risk management before versus after completing MeTree.

Materials and Methods

MeTree

MeTree collects personal and family health history data directly from patients through a Web-based patient-facing interface, performs risk stratification, and generates patient- and provider-oriented CDS reports for risk management of increased risk patients as supported by evidence-based guidelines. Patients are encouraged to talk with relatives prior to entering data and are given educational documents, brochures, and a worksheet to facilitate data collection. MeTree’s CDS reports address five conditions—breast cancer, ovarian cancer, colon cancer, thrombosis, and hereditary cancer syndromes—and their associated evidence-based risk-management strategies for increased risk patients (i.e., prevention, surveillance, and/or consultation) ( Table 1 ). Both reports were designed, through several iterations of stakeholder feedback, to facilitate decision making, engagement, and education and to encourage appropriate risk-reducing actions. To avoid “reminder fatigue,” no recommendations are made for a CDS condition when patients are not at increased risk because most providers are familiar with routine screening recommendations. More details regarding MeTree’s development, risk algorithms/calculations, educational documents/brochures, implementation outcomes, and CDS are available in the referenced papers.8,13,15

Table 1 MeTree’s clinical decision support conditions and risk-management recommendations
Full size table

Setting and participants

Three community-based primary-care practices in the Cone Health System (Greensboro, NC) were identified by the Cone Health administration (owing to their larger size and the similarities in the populations they served) and agreed to participate in the study. All adult patients scheduled for an upcoming well-visit appointment at one of the two assigned intervention clinics between 15 October 2009 and 11 November 2010 were mailed an invitation to participate. In response to the providers’ and patients’ enthusiasm about having access to MeTree, enrollment was extended beyond this period to 14 April 2012. However, because patients enrolled after 11 November 2010 were not part of the original study design and did not undergo the 1-year follow-up to ascertain changes in their clinical care, they are not included in this clinical utility analysis. In the third clinic, patients and providers did not have access to MeTree. This clinic served as a concurrent control to account for any changes in routine clinical care that might occur during the study period. The types of practice in the three clinics were as follows: intervention clinic 1, internal medicine physicians only; intervention clinic 2, multispecialty with internal medicine, one gastroenterologist, and one endocrinologist; and control clinic, half internal medicine and half family practice. Only primary-care providers participated in the study, and none had a prior relationship with genetic counselors in their health system or in our study. In addition, to our knowledge none had experience in genetic testing.

Study design

This was a controlled type II hybrid implementation-effectiveness study.16 MeTree was integrated into the clinical workflow of the intervention clinics. Invitees who enrolled were consented in person by the study coordinator 1 week prior to their provider appointment and given the educational materials along with a paper-based survey to complete at home. They arrived 1 hour prior to their appointment to complete MeTree on a single dedicated computer in the clinic. Once data entry was completed, reports were printed by the study coordinator. The patient’s report was then immediately given to the patient and the provider’s CDS report was placed in the patient’s chart before the patient entered the examination room. A chart abstraction to evaluate clinical outcomes was performed 12 months after the patient completed MeTree. The published protocol describes each measure in detail.17

The study flow is displayed in Figure 1 . Because participants entered their data in a dedicated clinic computer prior to their appointment, only one person per hour per clinic could be recruited; this recruitment bottleneck is shown as computer time conflicts in Figure 1 .

Figure 1
figure 1

Study flow diagram.

Full size image

Outcomes and measures

Measures related to risk management for CDS conditions. Charts were abstracted using detailed abstraction guides by six genetic counseling students who entered data into REDcap.18 Test cases were used to train the students, and questions or discrepancies during abstraction were resolved by the study’s principal investigator. For each patient, all orders for mammography, breast MRI, colonoscopy, ovarian ultrasound, CA-125, tamoxifen, raloxifene, thrombosis-related genetic testing, gynecology referral, and genetic counseling were abstracted from the chart during the period prior to the date of enrollment (i.e., prior to completing MeTree) and the year after. Data entered into MeTree included age, gender, demographics, education, medical history, and family health history.

Primary outcome. The primary outcome was agreement between patient risk level and risk management before MeTree intervention versus after. The agreement between risk level and risk management was determined in two mutually exclusive groups of patients: those who met evidence-based guideline criteria for a CDS condition’s increased-risk risk-management strategy (i.e., MeTree’s CDS report recommended that strategy) and those who did not. Agreement in the former group of patients was defined as the proportion of patients receiving the recommended risk-management strategy. Agreement in the latter group of patients was defined as the proportion of those not receiving the risk-management strategy. Because of complexities in ascertaining appropriate colon cancer risk management (e.g., differentiating diagnostic from screening procedures and accounting for screening intervals that exceeded study follow-up), we report colon cancer risk management only among those who should not yet have started routine screening (i.e., those <50 years of age) before using MeTree.

Secondary outcome. Because the agreement between risk level and risk management involves two steps—providers ordering a risk-management strategy consistent with the patient’s risk level (provider agreement with risk management) and patients following through on those orders (patient agreement with risk management)—we assessed each of these steps. Providers were considered in “agreement” if they placed an order for the recommended evidence-based risk-management strategy (i.e., for a MeTree recommended action). Patients were considered in “agreement” when there was documentation of completion of a provider-ordered risk-management strategy in the chart. Genetic counseling was performed by a study genetic counselor who tracked genetic counseling completion in a study database.

Patients’ risk perception and locus of control were measured on the baseline survey to account for potential differences in patient agreement (with provider orders). Questions were from the National Health Interview Survey19: “Compared to other people your own age, race and sex, what do you think your chances are of getting [condition] in your lifetime?” (5-point Likert scale from “much higher” to “much lower”) and “There is not much you can do to lower your chances of getting cancer” (“yes” or “no”). Data were entered into REDcap18 by study coordinators.

Control-clinic measures were used to account for temporal trends in orders for risk-management strategies in routine clinical practice. One hundred random adults with well-visit appointments from 15 October 2010 to 2011 were selected for chart abstraction from 15 October 2008 forward using the methodology described previously.20 The differences in the proportion of orders during the period before 15 October 2009 (when MeTree was integrated into the intervention clinics) and the period after were compared with the differences in the frequencies for the intervention clinics.

Statistical analyses

Data were analyzed using R statistical software. All calculations were assessed at a significance level of P < 0.05 using standard statistical methods.21 Pearson’s χ2 statistic was used to assess differences in risk-management frequencies in the intervention and control clinics at times before and after MeTree implementation in the intervention clinic. Logistic regression z-scores were used to test screening rates in the control versus treatment clinic after adjusting for covariates. Differences in provider and patient agreement according to baseline presence of a CDS condition and demographic variables (including age, gender, insurance, ethnicity, education, risk perception, and locus of control) were assessed using Pearson’s χ2 statistic. Analysis of variance F-statistics were used to test the relationship between the following variables: risk perception, Gail score, and proportion of relatives with cancer. The relationship between risk perception and locus of control was tested using Pearson’s χ2 statistic.

To evaluate the possibility of data clustering by clinic, we performed two analyses. First, we compared demographic (age, education, ethnicity, and gender) and response (evidence-based risk-management practices for CDS conditions prior to using MeTree, MeTree recommendations, evidence-based risk-management practices for CDS conditions after using MeTree, and patient agreement) variables between intervention clinics. Practice 1 had an older patient population (mean +7.7 years) and higher baseline screening for colon and breast cancer, which could be accounted for by the age difference; other variables were not significantly different. Second, in regression modeling, we estimated a design effect for clinics; estimates were very close to 1, suggesting no significant clustering.22 Because neither analysis showed a statistically significant effect for clinics above and beyond the observed difference in age distribution, the overall covariate-adjusted model was used for analysis.

Results

Participants

Of the 693 patients enrolled, 488 were from the original cohort and underwent the 12-month chart review. Of these, 41% (N = 203) met criteria for one of the evidence-based risk-management strategies for increased-risk patients (i.e., received at least one MeTree CDS recommendation) ( Figure 1 ). Table 2 shows characteristics of the study population, the subgroup receiving a MeTree CDS recommendation, and the control-clinic population. Ethnicity and age of participants reflect those of the underlying clinic populations in both the intervention and control clinics, although more participants were female (58.2 vs. 42.7%).12 The subgroup with a MeTree recommendation had more females (due to the breast and ovarian cancer recommendations), higher Gail scores, and larger families.

Table 2 Demographics for the study population, those with MeTree recommendations, and the control population
Full size table

Risk perception and locus of control

More patients at baseline perceived their risk as higher if they were ultimately found to meet guideline criteria for an increased-risk risk-management strategy (received a MeTree recommendation), had a greater proportion of relatives with cancer, or had higher Gail scores. Those who perceived their risk of cancer as higher had a mean Gail score of 2.4%, compared with Gail scores of 1.6% or 1.5% for those with a risk perceived as being the “same” or “less,” respectively (PANOVA = 9.64 × 1010). Likewise, patients rating their risk as “higher” had a greater proportion of relatives with cancer. Means ranged from 18.4 to 20.3%, depending on the cancer, compared with 12.9–15.1% and 11.2–12.5% for those with risks perceived as being the “same” and “less,” respectively (PANOVA = 4.72 × 109–2.61 × 1018). Locus of control did not vary across groups by presence or absence of an evidence-based risk-management recommendation.

Agreement between risk management and risk level

As previously described, agreement between risk level is presented for two groups: those who met evidence-based criteria for a CDS condition’s increased-risk risk-management strategy and those who did not. Table 3 presents the agreement in both groups for the time periods before using MeTree and after for each risk-management strategy. Across all CDS conditions, there is an improvement in agreement for both groups after using MeTree. In addition, when looking at data only for participants who had been receiving an increased-risk risk-management strategy before MeTree, only 2 of 19 (10.5%) met evidence-based criteria (received a MeTree CDS recommendation), while after MeTree 28 of 39 (71.8%) met evidence-based criteria. For colon cancer, which, as previously described, entails an evaluation only of those <50 years old, none of the patients meeting guideline criteria for early colon cancer screening (N = 76) was screened by the recommended age before MeTree. Adding this group to those at increased risk who did not receive increased-risk management before using MeTree reduces those with a match between risk level and risk management before MeTree to 0.8% (N = 2/250).

Table 3 Agreement between risk level and evidence-based risk management before and after Using MeTree for each risk-management strategyc
Full size table

Comparisons between intervention and control clinics

In the control clinic, breast cancer screening rates during the periods before and after the study start date were not significantly different: breast MRI 0–1.8% (Pchisq = 0.3151) and mammography 62.5–48.2% (Pchisq = 0.1284). After adjusting for age differences between the control and intervention clinics, mammography rates after the study start date were significantly lower in the control clinic ((48.2%) than in intervention clinics (76.0%), Plogistic regression = 0.00317), whereas breast MRI rates were similar in the intervention (0.74%) and control clinics (1.8%) (Plogistic regression = 0.371), suggesting that temporal changes did not result in increased breast cancer screening in the intervention clinics.

Provider agreement

The degree of provider agreement with recommended evidence-based risk-management strategies is presented in Table 4 . Agreement with cancer or thrombosis-related recommendations was unaffected by patient gender (Pchisq = 0.475, pchisq = 0.1), ethnicity (Pchisq = 0.101, Pchisq = 0.814), education (Pchisq = 0.201, Pchisq = 0.709), insurance (Pchisq = 0.706, Pchisq = 0.250), risk perception (P = 0.260–0.973), or locus of control (P = 0.268–0.799). It was also unaffected by the presence of a baseline pre-existing condition such as cancer or prior thrombosis that might heighten the provider’s awareness of a patient’s ongoing risk (P = 0.933, P = 1.00). Only younger patient age was significantly associated with an increased agreement for cancer-related recommendations (mean age: 54.25 (SD: 14.22) vs. 60.74 (SD: 10.46), P = 0.004).

Table 4 Provider and patient agreement with evidence-based risk-management strategy recommendationsc
Full size table

Patient agreement

Patient agreement with recommended evidence-based risk-management strategies is presented in Table 4 . There was no difference in agreement between those who had experienced a pre-existing cancer or prior thrombosis at baseline: Pchisq = 0.397 and 0.361, respectively. Agreement with cancer and thrombosis-related orders was unaffected by patient gender (Pchisq = 0.970, Pchisq = 0.171), ethnicity (Pchisq = 0.152, Pchisq = 0.576), education (Pchisq = 0.196, Pchisq = 0.392), risk perception (Pchisq = 0.105–0.576), or locus of control (Pchisq = 0.105, Pchisq = 1.00). However, agreement with cancer-related orders was higher in patients >60 years of age (Pchisq = 0.0472).

Discussion

Risk stratification and risk-stratified guidelines are important for improving quality and cost-effectiveness of medical care. Interventions targeted to risk level allow the right care to be matched to the right patient at the right time.23,24 However, constraints in primary-care environments limit collection and use of risk information.25 In this pilot study, we demonstrate that MeTree, a risk-assessment and CDS tool designed to overcome these constraints, can improve uptake of risk-stratified guidelines compared with current practice, decrease use of potentially unnecessary increased-risk services in patients not at increased risk, and improve the use of increased-risk services in those at increased risk. Importantly, this effect, although variable across conditions, had an equally positive uptake and impact across ethnicities and education levels, suggesting that tools like MeTree may help mitigate health-related disparities. To our knowledge, this is the first time that a risk-assessment tool has been shown to change clinical care.

Our assessment of MeTree’s impact on adoption of risk-stratified guidelines is based on the assumption that providers’ perceptions of patients’ risk levels can be captured by the prevention and risk-management strategies patients receive. For example, providers typically recommend services such as genetic counseling, breast MRIs, or colon cancer surveillance before age 50 if they believe the patients for whom they recommend such services are at increased risk. Using this criterion, we found, as have others, that there are discrepancies between physicians’ risk estimates and guideline-derived risk estimates in both directions (i.e., overestimation and underestimation).26,27,28,29 Underestimation was most pronounced in the group at risk for hereditary cancer syndromes and was also evident among the 76 participants <50 years of age who should have started colonoscopies early but had not. This is not surprising given the well-documented barriers to collecting and synthesizing family health history.3,20,30,31,32 Overestimation was also evident, with 17 average-risk patients receiving increased-risk care. It is reassuring that, given the high number of average-risk patients, this percentage was quite small; however, as more conditions are included in risk assessments, the baseline proportion is likely to increase. MeTree reduced the discrepancy between risk level and risk management by improving the use of evidence-based risk-management services in 52 of 203 (25.6%) patients and reduced potentially unnecessary service use in 6 of 17 (14.8%) patients.

A 17% improvement in the concordance between patient risk level and patient care in a real-world environment is a clinically significant improvement but is not optimal. This is important to explore because, based on current knowledge, MeTree—which meets all the proposed criteria for an effective risk-assessment program and is widely accepted among patient and provider users—should have been able to overcome many of the barriers to changing care.12,33 In this study, the greatest determinant of whether a patient received the recommended risk-management strategy was not patients following through with providers’ orders but, rather, providers placing orders.

Two factors affecting provider agreement with risk-management recommendations are context and agreement with guidelines. In a survey of our provider population, contextual reasons—including patients’ life expectancy, patients’ economic stability, patients’ concerns about the risk-management strategy, providers’ validation of variables triggering assignment to the increased-risk group, and providers’ knowledge about the impact of environmental or other explanatory factors not assessed by MeTree—were cited as the rationale for not placing a genetic counseling referral.34 This is not uncommon, and our referral rate is not dissimilar to that found in other studies.35 An example in our study of disagreement with a guideline affecting orders for a risk-management strategy is breast cancer chemoprevention. During end-of-study face-to-face discussions, providers attributed their failure to place a single order for tamoxifen or raloxifen to discomfort with chemoprevention recommendations, specifically as relating to thrombosis risk. This finding is also consistent with results from other studies.36 Given that provider agreement is a key factor in guideline uptake, and that our understanding is still quite limited, future qualitative interviews to identify barriers and solutions will be a critical step in optimizing risk-assessment programs for real-world implementation.

Our observed patient agreement rates were reasonably high. For example, genetic counseling (50%, N = 22/44) was higher than that seen in an independent study (36% ref. 37). As with provider adherence, patient gender, ethnicity, risk perception, insurance, baseline presence of a CDS condition, and education had no discernible effect on patient agreement. However, more patients in the increased-risk group perceived their risk to be high, suggesting that they were aware of their risk but not always ready to act on it. An additional finding is that MeTree did not lead to an unwarranted increase in increased-risk services such as genetic counseling (e.g., the Angelina Jolie effect38). In this study, there was only one instance when a patient’s concern led to a referral that was not recommended by MeTree. In fact, risk-assessment tools may do just the opposite; they may serve to alleviate concerns among those at average risk and decrease overscreening of those for whom it is not clinically indicated.

Our study has several limitations. First, our method of assessing agreement is limited (e.g., orders may not have been documented or patients might have undergone screening outside the institution). However, these inaccuracies would result in underestimation of agreement, and thus our findings likely represent a minimum effect. In addition, we were unable to assess agreement in colon cancer, although given that there was a substantial number of young participants who were at increased risk and had not received recommended management (N = 76), including them would likely have both reduced the match between risk level and risk management before MeTree and improved it after MeTree. Second, in identifying cases for which there may be potential overuse of a high-risk service, we emphasize “potential” because there are many contexts in which a patient with a low lifetime breast cancer risk score may still benefit from a breast MRI. Providers have clinical intuition that MeTree will never be able to emulate; therefore, we use this only as a marker for change rather than as an indicator of inappropriate care. Third, our patient and provider population may not be generalizable to the US population, although we have previously shown that our study population is similar to both the underlying clinic population and selected populations based on risk recurrence ratios.23,39 Fourth, we cannot be certain that the study population is completely representative of the underlying clinic populations. Several factors contribute to this: the inability to obtain information on ethnicity and education level for the control-clinic patients (although their random allocation should reduce the likelihood of bias), the fact that the clinics were not randomly selected but chosen by the hospital’s chief executive officer, and the possibility that participants who enrolled in the study were more motivated to pursue increased-risk services than those who did not. However, our population was similar to those in comparable studies, which suggests that the population may be generalizable.4,6,9,10,11 Based on the results of this pilot study, MeTree is now being studied in five diverse national health-care settings.

In conclusion, integrating MeTree into primary-care practice can improve uptake of evidence-based risk-stratified guidelines and in the process potentially reduce “overuse” and “underuse” of increased-risk services. In addition, it may be a helpful tool for reducing disparities in preventive health. Further study of MeTree in diverse populations and qualitative interviews will help us to better understand how to improve this tool and broaden its impact.

Disclosure

The authors declare no conflict of interest.