Main

Colorectal cancer is a major health burden with 1.24 million new cases diagnosed worldwide in 2008 (Ferlay et al, 2010; Jemal et al, 2010). It is a major cause of cancer deaths with an estimated 608 000 deaths attributed to colorectal cancer in 2008, accounting for 8% of all cancer deaths (Ferlay et al, 2010). In the United Kingdom, nearly 40 000 new cases were identified in 2008 (info.cancerresearchuk.org). In the United Kingdom, the National Awareness and Early Diagnosis Initiative (NAEDI) has been established to improve the public’s awareness of signs and symptoms of cancer, and to seek advice earlier that will ultimately prevent avoidable deaths (Richards, 2009). It is estimated that if survival rates in England matched best rates found in Europe, then about 1700 deaths could be avoided per year (Abdel-Rahman et al, 2009).

The diagnostic utility of individual so-called ‘red-flag’ alarm symptoms (i.e., rectal bleeding) are limited and not necessarily specific to colorectal cancer (Jones et al, 2007). Current guidance in the United Kingdom from the National Institute for Health and Clinical Excellence (NICE) recommends the decision for urgent referral to a specialist based on a series factors such as age, menopausal status, change of bowel habit, stool frequency, rectal bleeding, abdominal mass and anaemia (National Institute for Health and Clinical Excellence, 2005). However, there are suggestions that current guidance from NICE can only identify (the minority) high-risk symptomatic patients, and that the small improvement in survival in the United Kingdom (which falls behind much of Europe (Berrino et al, 2007)) implies the current approach is not working (Hamilton, 2009a, 2009b). While we do not necessarily share these views in the absence of evidence, there are clearly opportunities and a need to provide a more consistent and objective approach to assist primary care physicians in identifying patients with undiagnosed colorectal cancer.

While the usefulness of individual symptoms is limited (Ford et al, 2008; Hamilton et al, 2008; Hamilton et al, 2009; Olde Bekkink et al, 2010; Adelstein et al, 2011; Astin et al, 2011), a more appropriate approach would be to base the decision for referral using a combination of multiple risk factors, by means of a multivariable prediction model. Existing approaches based on multivariable prediction models have recently been derived that indicate improved performance over single symptoms can be achieved (Hamilton, 2009a; Marshall et al, 2012). QCancer (Colorectal) are a pair of multivariable prediction models (one for men and one for women) that have recently been developed to predict the risk of having undiagnosed colorectal cancer (Hippisley-Cox and Coupland, 2012).

QCancer (Colorectal) was developed and internally validated on a large cohort of 3.6 million patients from the QRESEARCH (www.qresearch.org) database (Hippisley-Cox and Coupland, 2012). The QRESEARCH database is a large database comprising over 12 million anonymised health records from 602 general practices throughout the United Kingdom using the EMIS computer system. QCancer (Colorectal) was developed on 2.35 million patients aged between 30 and 84 years, contributing 4798 incident cases of colorectal cancer from 4 110 382 person-years of observation between 01 January 2000 and 30 September 2010. The final prediction models included seven risk factors for women and nine risk factors for men (Table 1). Open source code to calculate the QCancer (Colorectal) scores are available from www.qcancer.org/colorectal/ released under the GNU Lesser General Public Licence, version 3. The performance of the QCancer (Colorectal) was assessed on a separate sample of 1.2 million patients from the same QRESEARCH database with good discriminative ability and calibration (Hippisley-Cox and Coupland, 2012).

Table 1 Risk factors in QCancer (Colorectal)
Full size table

The aim of this article is to describe the results from an independent evaluation of QCancer (Colorectal) on a large dataset of general practice records in the United Kingdom not used to derive the prediction model.

Materials and methods

Cohort selection

Study participants were patients registered between 01 January 2000 and 30 June 2008, and recorded on the THIN database (www.thin-uk.com). The same exclusion criteria as the original development paper were adopted (Hippisley-Cox and Coupland, 2012). Patients were excluded if they had a prior diagnosis of colorectal cancer, were registered <12 months with the general practice, had invalid dates, were under the age of 30 years or were aged 85 years or over. Entry to the cohort was defined, as for the original development study (Hippisley-Cox and Coupland, 2012), defined as the latest of (1) the study start date, (2) date the patient registered with the practice and (3) for those patients with symptoms (abdominal pain, appetite loss, rectal bleeding, weight loss, anaemia or change in bowel habit), the date of the first recorded onset of any symptom within the study period.

Outcome measure

The primary outcome was diagnosis of colorectal cancer, which was defined as incident diagnosis of colorectal cancer during the 2 years after study (using relevant UK diagnostic Read codes). Patients without the study outcome were censored at the earliest of the date of death, date of leaving the practice study or 2 years of follow-up.

Statistical analysis

The 2-year predicted risk of colorectal cancer for every patient in the THIN cohort was calculated using QCancer (Colorectal) risk score (www.qcancer.org/colorectal). Multiple imputation using all predictors plus the outcome variable was used to replace missing values for alcohol consumption for men. This involves creating multiple copies of the data and imputing the missing values with sensible values randomly selected from their predicted distribution. Ten imputed data sets were generated and results from analyses on each of the imputed data sets were combined using Rubin’s rules to produce estimates and confidence intervals that incorporate the uncertainty of imputed values.

Predictive performance of the QCancer (Colorectal) risk score on the THIN cohort was assessed by examining measures of calibration and discrimination. Calibration refers to how closely the predicted 2-year colorectal cancer risk agrees with the observed proportions with a diagnosis of colorectal cancer within 2 years. This was assessed for each tenth of predicted risk, ensuring 10 equally sized groups, and each 5-year age band by plotting observed proportions vs predicted risk.

Discrimination is the ability of the risk score to differentiate between patients who experience an event during the study period and those who do not. This measure is quantified by calculating the area under the receiver operating characteristic curve c-statistic; a value of 0.5 represents chance and 1 represents perfect discrimination. We also calculated the D-statistic (Royston and Sauerbrei, 2004) and R2statistic (Royston, 2006) that are measures of discrimination and explained variation, respectively, and are tailored for censored survival data.

We used decision curve analysis (accounting for censored observations) to describe and compare the clinical effects of QCancer (Colorectal) (Vickers and Elkin, 2006). Briefly, the net benefit of a model is the difference between the number of true positives and the number of false positives weighted by the odds of the selected threshold for high-risk designation. It is interpreted as the additional number of colorectal cancer cases identified without increasing the number treated unnecessarily across a range of clinically relevant treatment/referral thresholds. QCancer (Colorectal) is assessed by comparing the gain of the model over a strategy of ‘investigate all’ (i.e., assume all high risk) and ‘investigate none’ (i.e., all low risk). On the resulting curve, the ‘investigate all’ strategy is depicted as the line that crosses the x axis at the value that corresponds to the incidence rate of the outcome, while the ‘investigate none’ is depicted by a horizontal line at 0.

All statistical analyses were carried out in R (version 2.14.1) (R Development Core Team, 2011) and the ICE (multiple imputation) procedure in Stata (version 11.2) (StataCorp, 2009).

Results

Between 01 January 2000 and 30 June 2008, 2 135 540 eligible patients from 364 general practices in the United Kingdom were registered in the THIN database. The 2 135 540 eligible patients contributed 3 701 761 person-years of observation (median follow-up was 2 years), among whom there were 3712 cases of colorectal cancer (1676 women and 2036 men). Table 2 details the characteristics of eligible patients.

Table 2 Characteristics of participants aged 30 to 84 years in the QRESEARCH development and THIN validation cohorts
Full size table

Complete data on all risk predictors were available for all women, while complete data on alcohol consumption were available for 39.4% of men (n=417 560). Compared with the original development cohort, the THIN cohort had more patients reporting abdominal pain, fewer patients reporting a family history of gastrointestinal cancer and more patients with missing data on alcohol consumption (22% in the development cohort compared with 60% in the THIN cohort).

Table 3 reports the age–sex incidence rates of each symptom included in the QCancer (Colorectal) prediction models. All the symptoms apart from abdominal pain tended to become more common with age. During the follow-up, the crude rate of colorectal cancer was 100 per 100 000 person-years of observation with 90 per 100 000 person-years for women and 111 per 100 000 person-years for men. As per the original development cohort, incidence rates of colorectal cancer increased sharply with age.

Table 3 Incidence rates of appetite loss, weight loss, rectal bleeding and abdominal pain per 100 000 person-years by sex and age in the THIN cohort
Full size table

Performance data for QCancer (Colorectal) from the original development cohort (Hippisley-Cox and Coupland, 2012) and the THIN cohort (multiple imputation (for men) and complete-case) are presented in Table 4. The R2-statistics (percentage of explained variation) and the D-statistic were marginally higher in the THIN cohort, 65.8% and 2.84 (women) and 68.3% and 3.00 (men), compared with those reported in the original development paper, 64.8% and 2.78 (women) and 66.7% and 2.90 (men) (Hippisley-Cox and Coupland, 2012). The high values of the D-statistic and also the c-statistic 0.91 (women) and 0.92 (men) indicate very good discrimination properties of QCancer (Colorectal). The usefulness of using individual symptoms to identify patients with colorectal cancer is compared with QCancer (Colorectal) and is reported in Table 5. Using a cutoff (0.4% women and 0.5% men) that identified the 10% of women and men at the highest risk, QCancer (Colorectal) clearly outperformed with 71 and 74% of all new colorectal cancers identified for women and men, respectively. Using the presence of rectal bleeding, only 34 and 40% of new colorectal cancers were identified in women and men, respectively. Similarly, using only abdominal pain, only 36% of colorectal cancer cases in women and 31% in men were identified.

Table 4 Performance data in the original development cohort and the external validation (THIN) cohort
Full size table
Table 5 Comparison of strategies to identify patients of having a diagnosis of colorectal cancer in the next 2 years
Full size table

Calibration plots of QCancer (Colorectal) for women and men by tenth of risk are presented in Figure 1. Model calibration is very good with close agreement between predicted and observed colorectal cancer risks across all tenths of risk. Similarly, Figure 2 displays the calibration plots of QCancer (Colorectal) for women and men by 5-year age bands. Again, the QCancer (Colorectal) shows very good agreement across all age groups, with very slight overprediction in the older age groups in both sexes, though the overprediction is small.

Figure 1
figure 1

Observed vs predicted colorectal cancer risks (triangles=predicted risk and circles=observed risk).

Full size image
Figure 2
figure 2

Observed vs predicted colorectal cancer risks by sex and age (triangles=predicted risk and circles=observed risk).

Full size image

Figure 3 displays the net benefit curves for QCancer (Colorectal), which clearly show that QCancer (Colorectal) has higher net benefit compared with the ‘investigate all’ (line crossing the x axis at 0.2–0.25) and ‘investigate none’ (horizontal line at 0) strategies.

Figure 3
figure 3

Decision curve analysis (net benefit=TP−wFP)/N; where TP=number of true positives, FP=number of false positive and w=ratio of harm to benefit).

Full size image

Discussion

QCancer (Colorectal) is a new risk score to identify individuals with undetected colorectal cancer in a primary care setting. The risk score was developed and internally validated on a large primary care electronic database (QRESEARCH) of 3.6 million patients contributing 7401 cases of colorectal cancer between 01 January 2000 and 30 September 2008 (Hippisley-Cox and Coupland, 2012).

QCancer (Colorectal) was designed to be a prediction model based on risk factors that are recorded in patients’ health records or which patients themselves are likely to know. Thus, QCancer (Colorectal) has potential to be a useful tool in the primary care setting to identify patients who are at an increased risk of having undiagnosed colorectal cancer who would strongly benefit from further investigation.

In the United Kingdom, the CAPER score is an alternative prediction model that uses multiple symptoms to identify patients with undiagnosed colorectal cancer that is currently being considered by the Department of Health NAEDI framework (Hamilton et al, 2005; Hamilton, 2009a). However, there are several methodological concerns about how the score was derived. The handling of missing data is questionable in that patients with missing data who appear to have been omitted, thereby suggesting a complete-case analysis that has been shown to be a methodologically flawed and biased in the development of prediction models (Clark and Altman, 2003; Burton and Altman, 2004; Moons et al, 2006). Using data from a single location in the United Kingdom (Exeter), the authors screened 121 risk factors using univariate associations with colorectal cancer to reduce the number of predictors, a procedure that is not recommended (Sun et al, 1996). Furthermore, with only 349 cases of colorectal cancer the number of events per variable (<3) (Hamilton et al, 2005; Hamilton, 2009a) well below the rule-of-thumb of 10 is suggested to minimise the chance of over fitting. Lack of detailed reporting of how the score was derived and how to implement the model are also concerns (Khan, 2009). More importantly owing to the design of the study (a case–control study matched on age and sex), age and sex are not included as predictors in the CAPER score despite age being a major risk factor for colorectal cancer and the incidence of symptoms varying by age. Thus, an individual with or without symptoms will have the same predicted risk using the CAPER score regardless of age or sex. There is also currently insufficient performance (validation) data conducted by independent researchers to demonstrate the predictive performance of the CAPER score.

Our independent evaluation of the performance of QCancer (Colorectal) was carried out on the large separate database (THIN) of general practice records recording clinical data using the INPS Vision system that is used in 20% of UK general practices. The database comprised 2.1 million patients between 01 January 2000 and 30 June 2008 contributing 3.7 million years of observation and 3712 cases of colorectal cancer. The performance data presented in this article on the THIN cohort provide strong evidence to support the external validity of QCancer (Colorectal) in identifying patients with suspected colorectal cancer with marginal better performance as that seen in the internal validation data. QCancer (Colorectal) also clearly outperformed the use of using individual symptoms for identifying new cases of colorectal cancer.

A limitation of our study is that the recording of symptoms might be less complete, as individuals with mild symptoms may not visit their general practitioner or even report such mild symptoms if they do consult their general practitioner.

To date, the development, internal validation and our external validation of QCancer (Colorectal) has used 5.7 million patients contributing 10 million person-years of observation and 11 113 cases of colorectal cancer during the observation periods to develop and evaluate QCancer (Colorectal) to predict the risk of colorectal cancer in adults aged 30 to 84 years. In this study, we have provided an independent and external evaluation of the QCancer (Colorectal) risk score on a large cohort of patients in the United Kingdom. We have assessed the performance of QCancer (Colorectal) against performance metrics presented in the internal validation of QCancer (Colorectal), and have provided evidence to support the use of QCancer (Colorectal) in routine clinical practice.

Implementation of QCancer (Colorectal) or indeed any other risk prediction models used in the clinical decision making and patient managed should be evaluated by independent investigators using an appropriately large data set. Ideally, any prediction model should also be evaluated in an impact study to assess whether the prediction model changes clinician behaviour and ultimately improves patients outcomes (Moons et al, 2009; Wallace et al, 2011; Hamilton, 2012).

Conclusions

In this study, we have provided an independent and external validation of the QCancer (Colorectal) risk score on a large cohort of patients in the United Kingdom. The performance of QCancer (Colorectal) was similar to that in the internal validation with comparable results to indicating good predictive ability for identifying patients with undiagnosed colorectal cancer.

Ethical approval

Ethical approval was provided by Trent multicentre research ethics committee.