Clinical Study

BJC Open article

British Journal of Cancer (2012) 107, 260–265. doi:10.1038/bjc.2012.266
Published online 14 June 2012

Identifying patients with undetected colorectal cancer: an independent validation of QCancer (Colorectal)

G S Collins1 and D G Altman1

1Centre for Statistics in Medicine, Wolfson College Annexe, University of Oxford, Linton Road, Oxford OX2 6UD, UK

Correspondence: Dr GS Collins, E-mail:

Received 1 March 2012; Revised 15 May 2012; Accepted 18 May 2012
Advance online publication 14 June 2012





Early identification of colorectal cancer is an unresolved challenge and the predictive value of single symptoms is limited. We evaluated the performance of QCancer (Colorectal) prediction model for predicting the absolute risk of colorectal cancer in an independent UK cohort of patients from general practice records.



A total of 2.1 million patients registered with a general practice surgery between 01 January 2000 and 30 June 2008, aged 30-84 years (3.7 million person-years) with 3712 colorectal cancer cases were included in the analysis. Colorectal cancer was defined as incident diagnosis of colorectal cancer during the 2 years after study entry.



The results from this independent and external validation of QCancer (Colorectal) prediction model demonstrated good performance data on a large cohort of general practice patients. QCancer (Colorectal) had very good discrimination with an area under the ROC curve of 0.92 (women) and 0.91 (men), and explained 68% (women) and 66% (men) of the variation. QCancer (Colorectal) was well calibrated across all tenths of risk and over all age ranges with predicted risks closely matching observed risks.



QCancer (Colorectal) appears to be a useful tool for identifying undetected cases of undiagnosed colorectal cancer in primary care in the United Kingdom.


colorectal cancer; diagnosis; risk prediction; QCancer; validation

Colorectal cancer is a major health burden with 1.24 million new cases diagnosed worldwide in 2008 (Ferlay et al, 2010; Jemal et al, 2010). It is a major cause of cancer deaths with an estimated 608000 deaths attributed to colorectal cancer in 2008, accounting for 8% of all cancer deaths (Ferlay et al, 2010). In the United Kingdom, nearly 40000 new cases were identified in 2008 ( In the United Kingdom, the National Awareness and Early Diagnosis Initiative (NAEDI) has been established to improve the public’s awareness of signs and symptoms of cancer, and to seek advice earlier that will ultimately prevent avoidable deaths (Richards, 2009). It is estimated that if survival rates in England matched best rates found in Europe, then about 1700 deaths could be avoided per year (Abdel-Rahman et al, 2009).

The diagnostic utility of individual so-called ‘red-flag’ alarm symptoms (i.e., rectal bleeding) are limited and not necessarily specific to colorectal cancer (Jones et al, 2007). Current guidance in the United Kingdom from the National Institute for Health and Clinical Excellence (NICE) recommends the decision for urgent referral to a specialist based on a series factors such as age, menopausal status, change of bowel habit, stool frequency, rectal bleeding, abdominal mass and anaemia (National Institute for Health and Clinical Excellence, 2005). However, there are suggestions that current guidance from NICE can only identify (the minority) high-risk symptomatic patients, and that the small improvement in survival in the United Kingdom (which falls behind much of Europe (Berrino et al, 2007)) implies the current approach is not working (Hamilton, 2009a, 2009b). While we do not necessarily share these views in the absence of evidence, there are clearly opportunities and a need to provide a more consistent and objective approach to assist primary care physicians in identifying patients with undiagnosed colorectal cancer.

While the usefulness of individual symptoms is limited (Ford et al, 2008; Hamilton et al, 2008; Hamilton et al, 2009; Olde Bekkink et al, 2010; Adelstein et al, 2011; Astin et al, 2011), a more appropriate approach would be to base the decision for referral using a combination of multiple risk factors, by means of a multivariable prediction model. Existing approaches based on multivariable prediction models have recently been derived that indicate improved performance over single symptoms can be achieved (Hamilton, 2009a; Marshall et al, 2012). QCancer (Colorectal) are a pair of multivariable prediction models (one for men and one for women) that have recently been developed to predict the risk of having undiagnosed colorectal cancer (Hippisley-Cox and Coupland, 2012).

QCancer (Colorectal) was developed and internally validated on a large cohort of 3.6 million patients from the QRESEARCH ( database (Hippisley-Cox and Coupland, 2012). The QRESEARCH database is a large database comprising over 12 million anonymised health records from 602 general practices throughout the United Kingdom using the EMIS computer system. QCancer (Colorectal) was developed on 2.35 million patients aged between 30 and 84 years, contributing 4798 incident cases of colorectal cancer from 4110382 person-years of observation between 01 January 2000 and 30 September 2010. The final prediction models included seven risk factors for women and nine risk factors for men (Table 1). Open source code to calculate the QCancer (Colorectal) scores are available from released under the GNU Lesser General Public Licence, version 3. The performance of the QCancer (Colorectal) was assessed on a separate sample of 1.2 million patients from the same QRESEARCH database with good discriminative ability and calibration (Hippisley-Cox and Coupland, 2012).

The aim of this article is to describe the results from an independent evaluation of QCancer (Colorectal) on a large dataset of general practice records in the United Kingdom not used to derive the prediction model.


Materials and methods

Cohort selection

Study participants were patients registered between 01 January 2000 and 30 June 2008, and recorded on the THIN database ( The same exclusion criteria as the original development paper were adopted (Hippisley-Cox and Coupland, 2012). Patients were excluded if they had a prior diagnosis of colorectal cancer, were registered <12 months with the general practice, had invalid dates, were under the age of 30 years or were aged 85 years or over. Entry to the cohort was defined, as for the original development study (Hippisley-Cox and Coupland, 2012), defined as the latest of (1) the study start date, (2) date the patient registered with the practice and (3) for those patients with symptoms (abdominal pain, appetite loss, rectal bleeding, weight loss, anaemia or change in bowel habit), the date of the first recorded onset of any symptom within the study period.

Outcome measure

The primary outcome was diagnosis of colorectal cancer, which was defined as incident diagnosis of colorectal cancer during the 2 years after study (using relevant UK diagnostic Read codes). Patients without the study outcome were censored at the earliest of the date of death, date of leaving the practice study or 2 years of follow-up.

Statistical analysis

The 2-year predicted risk of colorectal cancer for every patient in the THIN cohort was calculated using QCancer (Colorectal) risk score ( Multiple imputation using all predictors plus the outcome variable was used to replace missing values for alcohol consumption for men. This involves creating multiple copies of the data and imputing the missing values with sensible values randomly selected from their predicted distribution. Ten imputed data sets were generated and results from analyses on each of the imputed data sets were combined using Rubin’s rules to produce estimates and confidence intervals that incorporate the uncertainty of imputed values.

Predictive performance of the QCancer (Colorectal) risk score on the THIN cohort was assessed by examining measures of calibration and discrimination. Calibration refers to how closely the predicted 2-year colorectal cancer risk agrees with the observed proportions with a diagnosis of colorectal cancer within 2 years. This was assessed for each tenth of predicted risk, ensuring 10 equally sized groups, and each 5-year age band by plotting observed proportions vs predicted risk.

Discrimination is the ability of the risk score to differentiate between patients who experience an event during the study period and those who do not. This measure is quantified by calculating the area under the receiver operating characteristic curve c-statistic; a value of 0.5 represents chance and 1 represents perfect discrimination. We also calculated the D-statistic (Royston and Sauerbrei, 2004) and R2statistic (Royston, 2006) that are measures of discrimination and explained variation, respectively, and are tailored for censored survival data.

We used decision curve analysis (accounting for censored observations) to describe and compare the clinical effects of QCancer (Colorectal) (Vickers and Elkin, 2006). Briefly, the net benefit of a model is the difference between the number of true positives and the number of false positives weighted by the odds of the selected threshold for high-risk designation. It is interpreted as the additional number of colorectal cancer cases identified without increasing the number treated unnecessarily across a range of clinically relevant treatment/referral thresholds. QCancer (Colorectal) is assessed by comparing the gain of the model over a strategy of ‘investigate all’ (i.e., assume all high risk) and ‘investigate none’ (i.e., all low risk). On the resulting curve, the ‘investigate all’ strategy is depicted as the line that crosses the x axis at the value that corresponds to the incidence rate of the outcome, while the ‘investigate none’ is depicted by a horizontal line at 0.

All statistical analyses were carried out in R (version 2.14.1) (R Development Core Team, 2011) and the ICE (multiple imputation) procedure in Stata (version 11.2) (StataCorp, 2009).



Between 01 January 2000 and 30 June 2008, 2135540 eligible patients from 364 general practices in the United Kingdom were registered in the THIN database. The 2135540 eligible patients contributed 3701761 person-years of observation (median follow-up was 2 years), among whom there were 3712 cases of colorectal cancer (1676 women and 2036 men). Table 2 details the characteristics of eligible patients.

Complete data on all risk predictors were available for all women, while complete data on alcohol consumption were available for 39.4% of men (n=417560). Compared with the original development cohort, the THIN cohort had more patients reporting abdominal pain, fewer patients reporting a family history of gastrointestinal cancer and more patients with missing data on alcohol consumption (22% in the development cohort compared with 60% in the THIN cohort).

Table 3 reports the age–sex incidence rates of each symptom included in the QCancer (Colorectal) prediction models. All the symptoms apart from abdominal pain tended to become more common with age. During the follow-up, the crude rate of colorectal cancer was 100 per 100000 person-years of observation with 90 per 100000 person-years for women and 111 per 100000 person-years for men. As per the original development cohort, incidence rates of colorectal cancer increased sharply with age.

Performance data for QCancer (Colorectal) from the original development cohort (Hippisley-Cox and Coupland, 2012) and the THIN cohort (multiple imputation (for men) and complete-case) are presented in Table 4. The R2-statistics (percentage of explained variation) and the D-statistic were marginally higher in the THIN cohort, 65.8% and 2.84 (women) and 68.3% and 3.00 (men), compared with those reported in the original development paper, 64.8% and 2.78 (women) and 66.7% and 2.90 (men) (Hippisley-Cox and Coupland, 2012). The high values of the D-statistic and also the c-statistic 0.91 (women) and 0.92 (men) indicate very good discrimination properties of QCancer (Colorectal). The usefulness of using individual symptoms to identify patients with colorectal cancer is compared with QCancer (Colorectal) and is reported in Table 5. Using a cutoff (0.4% women and 0.5% men) that identified the 10% of women and men at the highest risk, QCancer (Colorectal) clearly outperformed with 71 and 74% of all new colorectal cancers identified for women and men, respectively. Using the presence of rectal bleeding, only 34 and 40% of new colorectal cancers were identified in women and men, respectively. Similarly, using only abdominal pain, only 36% of colorectal cancer cases in women and 31% in men were identified.

Calibration plots of QCancer (Colorectal) for women and men by tenth of risk are presented in Figure 1. Model calibration is very good with close agreement between predicted and observed colorectal cancer risks across all tenths of risk. Similarly, Figure 2 displays the calibration plots of QCancer (Colorectal) for women and men by 5-year age bands. Again, the QCancer (Colorectal) shows very good agreement across all age groups, with very slight overprediction in the older age groups in both sexes, though the overprediction is small.

Figure 1.
Figure 1 - Unfortunately we are unable to provide accessible alternative text for this. If you require assistance to access this image, please contact or the author

Observed vs predicted colorectal cancer risks (triangles=predicted risk and circles=observed risk).

Full figure and legend (60K)

Figure 2.
Figure 2 - Unfortunately we are unable to provide accessible alternative text for this. If you require assistance to access this image, please contact or the author

Observed vs predicted colorectal cancer risks by sex and age (triangles=predicted risk and circles=observed risk).

Full figure and legend (65K)

Figure 3 displays the net benefit curves for QCancer (Colorectal), which clearly show that QCancer (Colorectal) has higher net benefit compared with the ‘investigate all’ (line crossing the x axis at 0.2–0.25) and ‘investigate none’ (horizontal line at 0) strategies.

Figure 3.
Figure 3 - Unfortunately we are unable to provide accessible alternative text for this. If you require assistance to access this image, please contact or the author

Decision curve analysis (net benefit=TP−wFP)/N; where TP=number of true positives, FP=number of false positive and w=ratio of harm to benefit).

Full figure and legend (48K)



QCancer (Colorectal) is a new risk score to identify individuals with undetected colorectal cancer in a primary care setting. The risk score was developed and internally validated on a large primary care electronic database (QRESEARCH) of 3.6 million patients contributing 7401 cases of colorectal cancer between 01 January 2000 and 30 September 2008 (Hippisley-Cox and Coupland, 2012).

QCancer (Colorectal) was designed to be a prediction model based on risk factors that are recorded in patients’ health records or which patients themselves are likely to know. Thus, QCancer (Colorectal) has potential to be a useful tool in the primary care setting to identify patients who are at an increased risk of having undiagnosed colorectal cancer who would strongly benefit from further investigation.

In the United Kingdom, the CAPER score is an alternative prediction model that uses multiple symptoms to identify patients with undiagnosed colorectal cancer that is currently being considered by the Department of Health NAEDI framework (Hamilton et al, 2005; Hamilton, 2009a). However, there are several methodological concerns about how the score was derived. The handling of missing data is questionable in that patients with missing data who appear to have been omitted, thereby suggesting a complete-case analysis that has been shown to be a methodologically flawed and biased in the development of prediction models (Clark and Altman, 2003; Burton and Altman, 2004; Moons et al, 2006). Using data from a single location in the United Kingdom (Exeter), the authors screened 121 risk factors using univariate associations with colorectal cancer to reduce the number of predictors, a procedure that is not recommended (Sun et al, 1996). Furthermore, with only 349 cases of colorectal cancer the number of events per variable (<3) (Hamilton et al, 2005; Hamilton, 2009a) well below the rule-of-thumb of 10 is suggested to minimise the chance of over fitting. Lack of detailed reporting of how the score was derived and how to implement the model are also concerns (Khan, 2009). More importantly owing to the design of the study (a case–control study matched on age and sex), age and sex are not included as predictors in the CAPER score despite age being a major risk factor for colorectal cancer and the incidence of symptoms varying by age. Thus, an individual with or without symptoms will have the same predicted risk using the CAPER score regardless of age or sex. There is also currently insufficient performance (validation) data conducted by independent researchers to demonstrate the predictive performance of the CAPER score.

Our independent evaluation of the performance of QCancer (Colorectal) was carried out on the large separate database (THIN) of general practice records recording clinical data using the INPS Vision system that is used in 20% of UK general practices. The database comprised 2.1 million patients between 01 January 2000 and 30 June 2008 contributing 3.7 million years of observation and 3712 cases of colorectal cancer. The performance data presented in this article on the THIN cohort provide strong evidence to support the external validity of QCancer (Colorectal) in identifying patients with suspected colorectal cancer with marginal better performance as that seen in the internal validation data. QCancer (Colorectal) also clearly outperformed the use of using individual symptoms for identifying new cases of colorectal cancer.

A limitation of our study is that the recording of symptoms might be less complete, as individuals with mild symptoms may not visit their general practitioner or even report such mild symptoms if they do consult their general practitioner.

To date, the development, internal validation and our external validation of QCancer (Colorectal) has used 5.7 million patients contributing 10 million person-years of observation and 11113 cases of colorectal cancer during the observation periods to develop and evaluate QCancer (Colorectal) to predict the risk of colorectal cancer in adults aged 30 to 84 years. In this study, we have provided an independent and external evaluation of the QCancer (Colorectal) risk score on a large cohort of patients in the United Kingdom. We have assessed the performance of QCancer (Colorectal) against performance metrics presented in the internal validation of QCancer (Colorectal), and have provided evidence to support the use of QCancer (Colorectal) in routine clinical practice.

Implementation of QCancer (Colorectal) or indeed any other risk prediction models used in the clinical decision making and patient managed should be evaluated by independent investigators using an appropriately large data set. Ideally, any prediction model should also be evaluated in an impact study to assess whether the prediction model changes clinician behaviour and ultimately improves patients outcomes (Moons et al, 2009; Wallace et al, 2011; Hamilton, 2012).



In this study, we have provided an independent and external validation of the QCancer (Colorectal) risk score on a large cohort of patients in the United Kingdom. The performance of QCancer (Colorectal) was similar to that in the internal validation with comparable results to indicating good predictive ability for identifying patients with undiagnosed colorectal cancer.

Ethical approval

Ethical approval was provided by Trent multicentre research ethics committee.



This work is published under the standard license to publish agreement. After 12 months the work will become freely available and the license terms will switch to a Creative Commons Attribution-NonCommercial-Share Alike 3.0 Unported License.



  1. Abdel-Rahman M, Stockton D, Rachet B, Hakulinen T, Coleman MP (2009) What if cancer survival in Britain were the same as in Europe: how many deaths are avoidable? Br J Cancer 101(Suppl 2): S115–S124 | Article | PubMed | ISI |
  2. Adelstein BA, Macaskill P, Chan SF, Katelaris PH, Irwig L (2011) Most bowel cancer symptoms do not indicate colorectal cancer and polyps: a systematic review. BMC Gastroenterol 11: 65 | Article | PubMed |
  3. Astin M, Griffin T, Neal RD, Rose P, Hamilton W (2011) The diagnostic value of symptoms for colorectal cancer in primary care: a systematic review. Br J Gen Pract 61(586): e231–e243 | Article | PubMed |
  4. Berrino F, De Angelis R, Sant M, Rosso S, Bielska-Lasota M, Coebergh JW, Santaquilani M (2007) Survival for eight major cancers and all cancers combined for European adults diagnosed in 1995–99: results of the EUROCARE-4 study. Lancet Oncol 8: 773–783 | Article | PubMed | ISI |
  5. Burton A, Altman DG (2004) Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines. Br J Cancer 91(1): 4–8 | Article | PubMed | CAS |
  6. Clark TG, Altman DG (2003) Developing a prognostic model in the presence of missing data: an ovarian cancer case study. J Clin Epidemiol 56(1): 28–37 | Article | PubMed | ISI |
  7. Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM (2010) GLOBOCAN 2008 v1.2, Cancer Incidence and Mortality Worldwide: IARC CancerBase No. 10. International Agency for Research on Cancer: Lyon, France, Available from
  8. Ford AC, Veldhuyzen van Zanten SJ, Rodgers CC, Talley NJ, Vakil NB, Moayyedi P (2008) Diagnostic utility of alarm features for colorectal cancer: systematic review and meta-analysis. Gut 57(11): 1545–1553 | Article | PubMed | CAS |
  9. Hamilton W (2009a) The CAPER studies: five case-control studies aimed at identifying and quantifying the risk of cancer in symptomatic primary care patients. Br J Cancer 101(Suppl 2): S80–S86 | Article | PubMed | CAS |
  10. Hamilton W (2009b) Five misconceptions in cancer diagnosis. Br J Gen Pract 59(563): 441–445, 447; discussion 446 | Article | PubMed |
  11. Hamilton W (2012) Computer assisted diagnosis of ovarian cancer in primary care. BMJ 344: d7628 | Article | PubMed |
  12. Hamilton W, Lancashire R, Sharp D, Peters TJ, Cheng K, Marshall T (2009) The risk of colorectal cancer with symptoms at different ages and between the sexes: a case-control study. BMC Med 7: 17 | Article | PubMed |
  13. Hamilton W, Lancashire R, Sharp D, Peters TJ, Cheng KK, Marshall T (2008) The importance of anaemia in diagnosing colorectal cancer: a case-control study using electronic primary care records. Br J Cancer 98(2): 323–327 | Article | PubMed | ISI | CAS |
  14. Hamilton W, Round A, Sharp D, Peters TJ (2005) Clinical features of colorectal cancer before diagnosis: a population-based case-control study. Br J Cancer 93(4): 399–405 | Article | PubMed | CAS |
  15. Hippisley-Cox J, Coupland C (2012) Identifying patients with suspected colorectal cancer in primary care: derivation and validation of an algorithm. Br J Gen Pract 62: e29–e37 | Article | PubMed |
  16. Jemal A, Center MM, DeSantis C, Ward EM (2010) Global patterns of cancer incidence and mortality rates and trends. Cancer Epidemiol Biomarkers Prev 19(8): 1893–1907 | Article | PubMed | ISI |
  17. Jones R, Latinovic R, Charlton J, Gulliford MC (2007) Alarm symptoms in early diagnosis of cancer in primary care: cohort study using General Practice Research Database. BMJ 334: 1040 | Article | PubMed |
  18. Khan NF (2009) Implementation of a diagnostic tool for symptomatic colorectal cancer in primary care: a feasibility study. Prim Health Care Res Dev 10(01): 54 | Article |
  19. Marshall T, Lancashire R, Sharp D, Peters TJ, Cheng KK, Hamilton W (2012) The diagnostic performance of scoring systems to identify symptomatic colorectal cancer compared to current referral guidance. Gut 60: 1242–1248 | Article |
  20. Moons KG, Donders RA, Stijnen T, Harrell FE (2006) Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 59(10): 1092–1101 | Article | PubMed | ISI |
  21. Moons KGM, Altman DG, Vergouwe Y, Royston P (2009) Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338: b606 | Article | PubMed |
  22. National Institute for Health and Clinical Excellence (2005) Referral Guidelines for Suspected Cancer. National Institute for Health and Clinical Excellence: London
  23. Olde Bekkink M, McCowan C, Falk GA, Teljeur C, Van de Laar FA, Fahey T (2010) Diagnostic accuracy systematic review of rectal bleeding in combination with other symptoms, signs and tests in relation to colorectal cancer. Br J Cancer 102(1): 48–58 | Article | PubMed |
  24. R Development Core Team (2011) R: A Language and Environment for Statistical Computing. R Development Core Team: Vienna, Austria, ISBN 3-900051-07-0,
  25. Richards MA (2009) The national awareness and early diagnosis initiative in England: assembling the evidence. Br J Cancer 101: S1–S4 | Article | PubMed | ISI |
  26. Royston P (2006) Explained variation for survival models. Stata J 6(1): 83–96
  27. Royston P, Sauerbrei W (2004) A new measure of prognostic separation in survival data. Stat Med 23(5): 723–748 | Article | PubMed |
  28. StataCorp (2009) StataCorp. Stata statistical software: release 11
  29. Sun GW, Shook TL, Kay GL (1996) Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 49(8): 907–916 | Article | PubMed | ISI | CAS |
  30. Vickers AJ, Elkin EB (2006) Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 26: 565–574 | Article | PubMed | ISI |
  31. Wallace E, Smith SM, Perera-Salazar R, Vaucher P, McCowan C, Collins G, Verbakel J, Lakhanpaul M, Fahey T (2011) Framework for the impact analysis and implementation of clinical prediction rules (CPRs). BMC Med Inform Decis Mak 11: 62 | Article | PubMed |
BJC Open article

From twelve months after its original publication, this work is licensed under the Creative Commons Attribution-NonCommercial-Share Alike 3.0 Unported License.
To view a copy of this license, visit