Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Clinical Studies

Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery

Abstract

Background

Several diagnostic prediction models to help clinicians discriminate between benign and malignant adnexal masses are available. This study is a head-to-head comparison of the performance of the Assessment of Different NEoplasias in the adneXa (ADNEX) model with that of the Risk of Ovarian Malignancy Algorithm (ROMA).

Methods

This is a retrospective study based on prospectively included consecutive women with an adnexal tumour scheduled for surgery at five oncology centres and one non-oncology centre in four countries between 2015 and 2019. The reference standard was histology. Model performance for ADNEX and ROMA was evaluated regarding discrimination, calibration, and clinical utility.

Results

The primary analysis included 894 patients, of whom 434 (49%) had a malignant tumour. The area under the receiver operating characteristic curve (AUC) was 0.92 (95% CI 0.88–0.95) for ADNEX with CA125, 0.90 (0.84–0.94) for ADNEX without CA125, and 0.85 (0.80–0.89) for ROMA. ROMA, and to a lesser extent ADNEX, underestimated the risk of malignancy. Clinical utility was highest for ADNEX. ROMA had no clinical utility at decision thresholds <27%.

Conclusions

ADNEX had better ability to discriminate between benign and malignant adnexal tumours and higher clinical utility than ROMA.

Clinical trial registration

clinicaltrials.gov NCT01698632 and NCT02847832.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1
Fig. 2: Summary forest plot of the area under the receiver operating characteristic curve (AUC) after Bayesian meta-analysis of centre-specific results (n = 894).
Fig. 3: Overall calibration curves for the estimated risk of malignancy.
Fig. 4: Overall decision curves.

Similar content being viewed by others

Data availability

The dataset generated and analysed during the current study is available in the KU Leuven Research Data Repository (RDR), https://doi.org/10.48804/TXL95Z. The dataset is not publicly available because this was not part of the informed consent. However, the dataset may be obtained following permission of AC and DT and after fulfilling all data transfer requirements.

References

  1. Engelen MJA, Kos HE, Willemse PHB, Aalders JG, de Vries EGE, Schaapveld M, et al. Surgery by consultant gynecologic oncologists improves survival in patients with ovarian carcinoma. Cancer. 2006;106:589–98.

    Article  PubMed  Google Scholar 

  2. Earle CC, Schrag D, Neville BA, Yabroff KR, Topor M, Fahey A, et al. Effect of surgeon specialty on processes of care and outcomes for ovarian cancer patients. J Natl Cancer Inst. 2006;98:172–80.

    Article  PubMed  Google Scholar 

  3. Woo YL, Kyrgiou M, Bryant A, Everett T, Dickinson HO. Centralisation of services for gynaecological cancers—a Cochrane systematic review. Gynecol Oncol. 2012;126:286–90.

    Article  PubMed  Google Scholar 

  4. Bristow RE, Chang J, Ziogas A, Anton-Culver H. Adherence to treatment guidelines for ovarian cancer as a measure of quality care. Obstet Gynecol. 2013;121:1226–34.

    Article  CAS  PubMed  Google Scholar 

  5. Jacobs I, Oram D, Fairbanks J, Turner J, Frost C, Grudzinskas JG. A risk of malignancy index incorporating CA 125, ultrasound and menopausal status for the accurate preoperative diagnosis of ovarian cancer. Br J Obstet Gynaecol. 1990;97:922–9.

    Article  CAS  PubMed  Google Scholar 

  6. Geomini P, Kruitwagen R, Bremer GL, Cnossen J, Mol BWJ. The accuracy of risk scores in predicting ovarian malignancy: a systematic review. Obst Gynecol. 2009;113:384–94.

    Article  Google Scholar 

  7. Kaijser J, Sayasneh A, Van Hoorde K, Ghaem-Maghami S, Bourne T, Timmerman D, et al. Presurgical diagnosis of adnexal tumours using mathematical models and scoring systems: a systematic review and meta-analysis. Hum Reprod Update. 2014;20:449–62.

    Article  PubMed  Google Scholar 

  8. Meys EMJ, Kaijser J, Kruitwagen RFPM, Slangen BFM, Van Calster B, Aertgeerts B, et al. Subjective assessment versus ultrasound models to diagnose ovarian cancer: a systematic review and meta-analysis. Eur J Cancer. 2016;58:17–29.

    Article  CAS  PubMed  Google Scholar 

  9. Ortiz-Muñoz B, Aznar-Oroval E, García AG, Peris AC, Ballestero PP, Yepes MS, et al. HE4, Ca125 and ROMA algorithm for differential diagnosis between benign gynaecological diseases and ovarian cancer. Tumor Biol. 2014;35:7249–58.

    Article  Google Scholar 

  10. Moore RG, McMeekin DS, Brown AK, DiSilvestro P, Miller MC, Allard WJ, et al. A novel multiple marker bioassay utilizing HE4 and CA125 for the prediction of ovarian cancer in patients with a pelvic mass. Gynecol Oncol. 2009;112:40–6.

    Article  CAS  PubMed  Google Scholar 

  11. Huang J, Chen J, Huang Q. Diagnostic value of HE4 in ovarian cancer: a meta-analysis. Eur J Obstet Gynecol Reprod Biol. 2018;231:35–42.

    Article  CAS  PubMed  Google Scholar 

  12. Van Calster B, Van Hoorde K, Valentin L, Testa AC, Fischerova D, Van Holsbeke C, et al. Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study. BMJ. 2014;349:g5920.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Westwood M, Ramaekers B, Lang S, Grimm S, Deshpande S, de Kock S, et al. Risk scores to guide referral decisions for people with suspected ovarian cancer in secondary care: a systematic review and cost-effectiveness analysis. Health Technol Assess. 2018;22:1–264.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Timmerman D, Planchamp F, Bourne T, Landolfo C, du Bois A, Chiva L, et al. ESGO/ISUOG/IOTA/ESGE consensus statement on pre-operative diagnosis of ovarian tumors. Int J Gynecol Cancer. 2021;31:961–82.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Collins GS, Moons KGM. Comparing risk prediction models. BMJ. 2012;344:e3186.

    Article  PubMed  Google Scholar 

  16. Qian L, Du Q, Jiang M, Yuan F, Chen H, Feng W. Comparison of the diagnostic performances of ultrasound-based models for predicting malignancy in patients with adnexal masses. Front Oncol. 2021;11:673722.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Meys EMJ, Jeelof LS, Achten NMJ, Slangen BFM, Lambrechts S, Kruitwagen RFPM, et al. Estimating risk of malignancy in adnexal masses: external validation of the ADNEX model and comparison with other frequently used ultrasound methods. Ultrasound Obstet Gynecol. 2017;49:784–92.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Stukan M, Badocha M, Ratajczak K. Development and validation of a model that includes two ultrasound parameters and the plasma D-dimer level for predicting malignancy in adnexal masses: an observational study. BMC Cancer. 2019;19:564.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Van Calster B, Valentin L, Froyman W, Landolfo C, Ceusters J, Testa AC, et al. Validation of models to diagnose ovarian cancer in patients managed surgically or conservatively: multicentre cohort study. BMJ. 2020;370:m2614.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Czekierdowski A, Stachowicz N, Smolen A, Łoziński T, Guzik P, Kluz T. Performance of IOTA simple rules risks, ADNEX model, subjective assessment compared to CA125 and HE4 with ROMA algorithm in discriminating between benign, borderline and stage I malignant adnexal lesions. Diagnostics. 2023;13:885.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594.

    Article  PubMed  Google Scholar 

  22. Timmerman D, Valentin L, Bourne TH, Collins WP, Verrelst H, Vergote I. Terms, definitions and measurements to describe the sonographic features of adnexal tumors: a consensus opinion from the International Ovarian Tumor Analysis (IOTA) group. Ultrasound Obstet Gynecol. 2000;16:500–5.

    Article  CAS  PubMed  Google Scholar 

  23. Installé AJ, Van den Bosch T, De Moor B, Timmerman D. Clinical data miner: an electronic case report form system with integrated data preprocessing and machine-learning libraries supporting clinical diagnostic model research. JMIR Med Inform. 2014;2:e28.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Kurman RJ, Carcangiu M, Herrington CS. World Health Organisation classification of tumours of the female reproductive organs, 4th ed. Lyon: International Agency for Research on Cancer; 2014.

  25. Prat J. FIGO Committee on Gynecologic Oncology. Staging classification for cancer of the ovary, fallopian tube, and peritoneum. Int J Gynecol Obstet. 2014;124:1–5.

    Article  Google Scholar 

  26. Timmerman D, Testa AC, Bourne T, Ferrazzi E, Ameye L, Konstantinovic ML, et al. Logistic regression model to distinguish between the benign and malignant adnexal mass before surgery: a multicenter study by the International Ovarian Tumor Analysis Group. J Clin Oncol. 2005;23:8794–801.

    Article  PubMed  Google Scholar 

  27. Debray TP, Damen JA, Riley RD, Snell K, Reitsma JB, Hooft L, et al. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res. 2019;28:2768–86.

    Article  MathSciNet  PubMed  Google Scholar 

  28. Demler OV, Pencina MJ, D’Agostino RB. Misuse of DeLong test to compare AUCs for nested models. Stat Med. 2012;31:2577–87.

    Article  MathSciNet  PubMed  PubMed Central  Google Scholar 

  29. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167–76.

    Article  PubMed  Google Scholar 

  30. Wynants L, Vergouwe Y, Van Huffel S, Timmerman D, Van Calster B. Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study. Stat Methods Med Res. 2018;27:1723–36.

    Article  MathSciNet  CAS  PubMed  Google Scholar 

  31. Reitsma JB, Glas AS, Rutjes AWS, Scholten RJPM, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005;58:982–90.

    Article  PubMed  Google Scholar 

  32. Wynants L, Timmerman D, Verbakel JY, Testa A, Savelli L, Fischerova D, et al. Clinical utility of risk models to refer patients with adnexal masses to specialized oncology care: multicenter external validation using decision curve analysis. Clin Cancer Res. 2017;23:5082–90.

    Article  PubMed  Google Scholar 

  33. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6. https://doi.org/10.1136/bmj.i6.

  34. Wynants L, Riley RD, Timmerman D, Van Calster B. Random-effects meta-analysis of the clinical utility of tests and prediction models. Stat Med. 2018;37:2034–52.

    Article  MathSciNet  CAS  PubMed  Google Scholar 

  35. Van Calster B, Vergouwe Y, Looman CWN, Van Belle V, Timmerman D, Steyerberg EW. Assessing the discriminative ability of risk models for more than two outcome categories. Eur J Epidemiol. 2012;27:761–70.

    Article  PubMed  Google Scholar 

  36. van Buuren S, Groothuis-Oudshoorn K. mice: multivariate imputation by chained equations in R. J Stat Softw. 2011;45:1–67.

    Article  Google Scholar 

  37. Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res. 2019;3:18.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Tso E, Elson P, VanLente F, Markman M. The “real-life” variability of CA-125 in ovarian cancer patients. Gynecol Oncol. 2006;103:141–4.

    Article  CAS  PubMed  Google Scholar 

  39. Sayasneh A, Ferrara L, De Cock B, Saso S, Al-Memar M, Johnson S, et al. Evaluating the risk of ovarian cancer before surgery using the ADNEX model: a multicentre external validation study. Br J Cancer. 2016;115:542–8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17:230.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Van Calster B, Steyerberg EW, Wynants L, van Smeden M. There is no such thing as a validated prediction model. BMC Med. 2023;21:70.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Youssef A, Pencina M, Thakur A, Zhu T, Clifton D, Shah NH. External validation of AI models in health should be replaced with recurring local validation. Nat Med. 2023;29:2686–7.

    Article  CAS  PubMed  Google Scholar 

  43. Tuxen MK, Sölétormos G, Petersen PH, Schiøler V, Dombernowsky P. Assessment of biological variation and analytical imprecision of CA 125, CEA, and TPA in relation to monitoring of ovarian cancer. Gynecol Oncol. 1999;74:12–22.

    Article  CAS  PubMed  Google Scholar 

  44. Braga F, Ferraro S, Mozzi R, Panteghini M. The importance of individual biology in the clinical use of serum biomarkers for ovarian cancer. Clin Chem Lab Med. 2014;52:1625–31.

    Article  CAS  PubMed  Google Scholar 

  45. Ferraro S, Borille S, Carnevale A, Frusciante E, Bassani N, Panteghini M. Verification of the harmonization of human epididymis protein 4 assays. Clin Chem Lab Med. 2016;54:1635–43.

    Article  CAS  PubMed  Google Scholar 

  46. Barr CE, Funston G, Mounce LTA, Pemberton PW, Howe JD, Crosbie EJ. Comparison of two immunoassays for the measurement of serum HE4 for ovarian cancer. Pract Lab Med. 2021;26:e00235.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Engelen MJA, de Bruijn HWA, Hollema H, ten Hoor KA, Willemse PHB, Aalders JG, et al. Serum CA 125, carcinoembryonic antigen, and CA 19-9 as tumor markers in borderline ovarian tumors. Gynecol Oncol. 2000;78:16–20.

    Article  CAS  PubMed  Google Scholar 

  48. Gotlieb WH, Soriano D, Achiron R, Zalel Y, Davidson B, Kopolovic J, et al. CA 125 measurement and ultrasonography in borderline tumors of the ovary. Am J Obstet Gynecol. 2000;183:541–6.

    Article  CAS  PubMed  Google Scholar 

  49. Sevinc A, Adli M, Kalender ME, Camci C. Benign causes of increased serum CA-125 concentration. Lancet Oncol. 2007;8:1054–5.

    Article  PubMed  Google Scholar 

  50. Van Calster B, Timmerman D, Bourne T, Testa AC, Van Holsbeke C, Domali E, et al. Discrimination between benign and malignant adnexal masses by specialist ultrasound examination versus serum CA-125. J Natl Cancer Inst. 2007;99:1706–14.

    Article  PubMed  Google Scholar 

  51. Huhtinen K, Suvitie P, Hiissa J, Junnila J, Huvila J, Kujari H, et al. Serum HE4 concentration differentiates malignant ovarian tumours from ovarian endometriotic cysts. Br J Cancer. 2009;100:1315–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Sokalska A, Timmerman D, Testa AC, Van Holsbeke C, Lissoni AA, Leone FPG, et al. Diagnostic accuracy of transvaginal ultrasound examination for assigning a specific diagnosis to adnexal masses. Ultrasound Obstet Gynecol. 2009;34:462–70.

    Article  CAS  PubMed  Google Scholar 

  53. Valentin L. Pattern recognition of pelvic masses by gray-scale ultrasound imaging: the contribution of Doppler ultrasound. Ultrasound Obstet Gynecol. 1999;14:338–47.

    Article  CAS  PubMed  Google Scholar 

  54. Sayasneh A, Kaijser J, Preisler J, Smith AA, Raslan F, Johnson S, et al. Accuracy of ultrasonography performed by examiners with varied training and experience in predicting specific pathology of adnexal masses. Ultrasound Obstet Gynecol. 2015;45:605–12.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We thank Gitte Thirion, Julie Oosterlynck, Katja Vandenbrande for processing the serum samples. We thank all medical specialists, data and case managers, secretaries, and all other people who collected data necessary for completing the database.

Funding

This research was funded by Kom Op Tegen Kanker (Stand up to Cancer), the Flemish cancer society (2016/10728/2603). The IOTA5 study is supported by the Research Foundation-Flanders (FWO) (projects G049312N, G0B4716N, 12F3114N, G097322N), and Internal Funds KU Leuven (projects C24/15/037 and C24M/20/064). DT is senior clinical investigator of FWO. TVG is a Senior Clinical Investigator of FWO (18B2921N). TBo is supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare UK National Health Service (NHS) Trust and Imperial College London. CL is supported by Linbury Trust Grant LIN 2600. The views expressed in this article are those of the authors and not necessarily those of the NHS, NIHR, or UK Department of Health. LV is supported by the Swedish Research Council (grant K2014-99X-22475-01-3, Dnr 2013-02282), funds administered by Malmö University Hospital and Skåne University Hospital, Allmänna Sjukhusets i Malmö Stiftelse for bekmäpande av cancer (the Malmö General Hospital Foundation for fighting against cancer), and two Swedish Governmental grants (Avtal om läkarutbildning och forskning (ALF)-medel and Landstingsfinansierad Regional Forskning).TBo

Author information

Authors and Affiliations

Authors

Contributions

DT, TBo, CL, WF, AC, AT, LV, and BVC conceived and designed the study. DT, TBo, CL, WF, TVG, RH, FMo, FMa, AN, CVH, VC, DF, AT, LV, TBa, and AC enrolled patients and acquired data. CL, RW, AV, JB, AN, Tba, and AC worked on the lab processes. DT, BVC, WF, CL, and JC did the data cleaning, with support from ASVR. DT, LV, TBo, BVC, JC, CL, and WF wrote the statistical analysis plan. BVC and JC analysed the data. DT, LV, TBo, BVC, JC, WF, CL, AC, and TVG interpreted the data. DT, LV, TBo, BVC, WF, CL, and JC wrote the first draft of the manuscript, which was then critically reviewed and revised by all the other authors. All authors approved the final version of the manuscript for submission. AC, DT, and BVC had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Corresponding author

Correspondence to Ben Van Calster.

Ethics declarations

Competing interests

TBo reports grants, personal fees, and travel support from Samsung Medison; travel support from Roche Diagnostics; and personal fees from GE Healthcare; all outside the submitted work. RW is employed by Oncoinvent AS. AC is a contracted researcher for Oncoinvent AS and Novocure and a consultant for Sotio a.s. and Epics Therapeutics SA. BVC and DT report consultancy work done by KU Leuven to help implementing and testing the ADNEX model in ultrasound machines by Samsung Medison and GE Healthcare, outside the submitted work. Tba reports grants, personal fees, and travel support from Roche, Novartis, GSK, MSD, and AstraZeneca, all outside the submitted work. All other authors declare no competing interests.

Ethics approval and consent to participate

Trans-IOTA is a subproject of the IOTA phase 5 and phase 7 studies (clinicaltrials.gov NCT01698632 and NCT02847832). The trans-IOTA project was approved by the Research Ethics Committee of the University Hospitals KU Leuven (reference numbers S51375 and S59207), and by the ethics committees of all participating centres. All patients gave their informed consent before enrolment in the study. The study was performed in accordance with the Declaration of Helsinki.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Landolfo, C., Ceusters, J., Valentin, L. et al. Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery. Br J Cancer 130, 934–940 (2024). https://doi.org/10.1038/s41416-024-02578-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41416-024-02578-x

Search

Quick links