Precision oncology is currently based on pairing molecularly targeted agents (MTA) to predefined single driver genes or biomarkers. Each tumor harbors a combination of a large number of potential genetic alterations of multiple driver genes in a complex system that limits the potential of this approach. We have developed an artificial intelligence (AI)-assisted computational method, the digital drug-assignment (DDA) system, to prioritize potential MTAs for each cancer patient based on the complex individual molecular profile of their tumor. We analyzed the clinical benefit of the DDA system on the molecular and clinical outcome data of patients treated in the SHIVA01 precision oncology clinical trial with MTAs matched to individual genetic alterations or biomarkers of their tumor. We found that the DDA score assigned to MTAs was significantly higher in patients experiencing disease control than in patients with progressive disease (1523 versus 580, P = 0.037). The median PFS was also significantly longer in patients receiving MTAs with high (1000+ <) than with low (<0) DDA scores (3.95 versus 1.95 months, P = 0.044). Our results indicate that AI-based systems, like DDA, are promising new tools for oncologists to improve the clinical benefit of precision oncology.
Precision oncology, the molecularly targeted treatment of every cancer patient based on the individual genetic alterations of their tumor, is a highly anticipated, straightforward approach to beat cancer eventually. Efforts in precision oncology have been focusing on identifying single variables, predictive biomarkers, presence or absence of molecular alterations, able to predict alone the response to a molecularly targeted agent (MTA). This approach has led to the development of predictive companion diagnostic tests that are now part of the indication of many MTAs1,2. The introduction of predictive biomarkers in the drug discovery and clinical use of MTAs was a significant milestone in precision oncology, but using single biomarkers has severe limitations.
Basket trials to find successful predictive biomarker–MTA pairs are in progress. The NCI-MATCH trial is a very important example. In NCI-MATCH tumors are molecularly profiled, and patients are assigned to different treatment arms based on the presence of predefined single biomarkers. The trial is feasible, and in some cases, successful, but it also indicates the limitations of the single biomarker paradigm as only three of the first 11 completed subprotocols resulted in positive results3. TAPUR, the other large-scale precision oncology basket trial organized by ASCO (American Society of Clinical Oncology), has reported similar results4.
Other clinical trials have been designed to analyze the overall clinical benefit of treating patients based on their tumors’ molecular profile with matching “on-target” but off-label MTAs or with matching investigational MTAs. IMPACT01 was the longest-running precision oncology trial that evaluated the benefit and limitation of matching MTAs to single biomarkers5. The ten-year survival rate was 6% in case molecularly matching therapies versus 1% in non-matching therapies (P < 0.0001). The objective response rate and the clinical benefit rate (including stable disease) were higher in case of matched MTAs compared to the results of the non-matched treatments (16.4% versus 5.4% and 35% versus 20.3%, respectively)5. The other pioneering precision oncology trial, MOSCATO01, found an 11% response rate and 33% benefit based on >1.3 longer PFS2/PFS1 ratio6. These results indicate that matching MTAs to single biomarkers does provide benefit; however, SHIVA01, the first randomized precision oncology trial assessing the efficacy of matching MTAs compared to conventional chemotherapy, did not find any statistically significant advantage of this approach7.
ESCAT, the European Society for Medical Oncology (ESMO) Scale for Clinical Actionability of molecular Targets, defines tiers to prioritize MTAs based on the highest level of evidence linking the MTA to a genetic alteration or a biomarker present in the patient’s tumor8. Tier I includes molecular alterations, which have demonstrated sufficient and robust clinical evidence to predict response to an MTA. Tier II includes molecular alterations demonstrating less robust clinical evidence in the same tumor type. Tier III includes alterations with evidence generated in other tumor types. Within tier III, tier IIIB alterations are different genetic variants of the same gene. Tier IV alterations are supported by preclinical data or in silico models only.
We have retrospectively evaluated the efficacy of matched MTA given in SHIVA01, also according to ESCAT tiers9. Molecular alterations revealed in SHIVA01 were retrospectively classified into the ESCAT tiers, then the average PFS and OS of MTAs in different tiers were compared using log-rank tests. We did not find a significant difference in the clinical outcome of the MTAs in tiers II, IIIA, and IV. Tier IIIB MTAs were associated with significantly shorter PFS than the MTAs in the other ESCAT tiers. This result indicates that the correct functional classification of the molecular alteration was a critical factor of success, but the level of evidence that links the MTA to the molecular alteration was not a significant predictor of response to the MTAs in the SHIVA01 trial. COSMIC (Catalogue of Somatic Mutations in Cancer) contains more than six million different potential somatic genetic alterations in more than 600 cancer genes10. The functional significance of most of these alterations is unknown (variants of unknown significance, VUS) or is associated with conflicting or limited evidence in the literature. This is a significant obstacle that contributes to the limited success rate of precision oncology today.
Another critical factor in precision oncology is that each tumor harbors a combination of 3–4 genetic driver alterations on average, and additional driver alterations in the non-coding regions are also present, raising the average number of molecular driver alterations to an average of 4–5 in every tumor11.
The I-PREDICT study presented evidence that MTA treatments (mostly combinations) inhibiting more than half (>50%) of the driver genetic alterations present in the tumor (high “matching score”) achieved at least 30% longer PFS (PFS2/PFS1 ≥ 1.3) compared to the previous line of standard therapy in 75% of patients, in comparison to 36.6% success rate of patients treated with MTAs targeting less than 50% of drivers (low “matching score”)12. This result indicates that the efficacy of MTAs matching their biomarker or driver is limited by the presence of unmatched drivers in two-thirds of patients.
Due to the recent outstanding advancement of molecular diagnostics, especially next-generation sequencing (NGS), it is now possible to analyze multiple driver genes in parallel instead of one by one, even in routine clinical practice. The challenge of precision oncology today is to assess the functional significance of all detected genetic alterations of all potential driver genes (single-nucleotide variations (SNVs), indels, CNVs, structural changes), next choosing the right target and the matching MTA that can be effective in the presence of the unmatched drivers. The complexity is further increased by the different sensitivity of individual genetic alterations to different MTAs matching the same target.
A potential solution is the deployment of standardized computational methods, artificial intelligence (AI)-assisted molecular drug-assignment algorithms. Computational tools can handle multi-dimensional, multianalyte data, and provide reproducible results. Therefore, computational algorithms are feasible tools to forecast treatment response to different MTAs in the case of the multiple different combinations present in tumors, and their performance can be evaluated as medical devices.
In this study, we aimed to develop a computational method (“digital drug assignment,” DDA), to test its performance to identify the most likely effective targeted therapies for cancer patients based on the individual complex molecular profile of their tumor. Driver genes and their specific genetic alterations are not equally important in the survival of the cancer cell. Several driver genes can be associated with multiple MTAs, and many MTAs can be associated with multiple drivers directly or indirectly through associated indirect molecular targets in the same tumor. The presence and importance of the associations between the drivers, targets, and MTAs are supported by variable numbers and level of experimental evidence. The goal was to create a system that automatically prioritizes potential driver genes and associated druggable targets based on thousands of available functional interactions derived from published evidence. Next, identifies MTAs that inhibit the most important drivers or the most important druggable targets associated with the most important divers at the highest level of aggregated evidence.
In theory, a system like this could maximize the chance to identify the MTA, which most likely triggers a therapeutic response based on our current knowledge of cancer biology and molecular pharmacology.
The DDA system presented in this study consists of a curated evidence-based network of over 12,000 driver-target-MTA interactions implemented in a software system, the Realtime Oncology Treatment CalculatorTM. The DDA system automatically assigns a mathematical DDA score, the “aggregated evidence level” (AEL), to MTAs related to cancer patients’ complete molecular profile in milliseconds.
Next, we used the molecular profile and the clinical outcome data of patients treated in the SHIVA01 clinical trial to evaluate the clinical performance of DDA. In SHIVA01, all patients in the precision oncology arm were treated with an MTA matched to a predefined biomarker present in the patient’s tumor. However, we found that MTAs with higher DDA scores were associated with significantly higher clinical benefit.
Results presented here indicate that DDA can potentially overcome the limitations of single biomarker-based treatment decisions, address the molecular complexity of cancer to improve the clinical benefit of precision oncology.
Overview of the DDA
Today, AI often refers to only machine learning (ML). However, computer science and the US Food and Drug Administration (FDA) action plan consider AI any intelligent computer programs that improve diagnostics or treatment decisions13,14. It would require vast clinical databases to develop effective predictive algorithms for treatment decisions solely with ML due to the large number of parameters that can influence the response to MTAs. Therefore, we decided to use a complex (“if–then”) rule-based expert system to create the DDA system based on the functional knowledge (“what we already know”) available from published experimental data. Rule-based expert systems are transparent, open box systems, and provide consistent results. Later, parts of the system can be further improved with ML methods after the fractionization of current complexity.
DDA uses a network of functional associations among mutant driver genes harboring molecular alterations, including SNVs, CNVs, and gene expression, tumor mutation burden (TMB), microsatellite instability (MSI) and druggable targets, MTAs, and tumor type (localization and histology) based on published peer-reviewed evidence (PubMed) built into a software system, the Realtime Oncology Treatment CalculatorTM (Fig. 1). Information about the frequency of molecular alterations was also included (COSMIC)10.
DDA algorithm assigns a score to each gene harboring molecular alterations (“driver calculation”) based on the number and weight of all functional associations with druggable target genes and MTAs. Next, DDA assigns a score to druggable target genes (“target calculation”) based on the number and weight of associations with all potential driver genes in the tumor. Finally, scores of MTAs are calculated based on the number and weight of associations with all “driver” genes and all “target” genes and the score of these genes. The weight of any association is calculated using constants (“matching weights”) corresponding to the similarity between the biological parameters of the particular tumor and the same parameters in the evidence used to generate the association.
“Driver calculation”, prioritization of the driver genes by the DDA system
First, a mathematical DDA score, the AEL of each mutant gene present in a tumor, is calculated based on published evidence of the functional significance of the particular mutation and evidence of other mutations found in other patients in the same gene or the gene as a driver gene in general in the evidence database (“driver calculation”). Functional evidence is a piece of evidence that indicates the role of the gene in the carcinogenesis in the patient’s tumor type or another tumor type, or evidence for the functional significance of the mutant gene and molecular alterations in the gene in the sensitivity or resistance (preclinical or clinical) to MTAs directly targeting the gene or an indirect druggable target gene that is functionally linked to the mutant gene. Each association that supports the functional relevance of the mutant gene, molecular all potential alterations in the mutant gene, the particular molecular alteration present in the patient’s tumor, counts as a positive association and each association indicating the functional irrelevance of the mutant gene or the molecular alterations in the mutant gene, the particular molecular alteration present in the patient’s tumor, counts as a negative association.
The evidence level (weight) of the associations is calculated based on the type (clinical or preclinical), source of the evidence (journal), and relevance (matching tumor type and matching molecular alteration type/exact alteration) in the particular patient. Whole available supporting evidence is aggregated to calculate the AEL of each specific mutation and all mutant genes in the same patient. All mutant genes with an AEL score higher than zero are considered a potential driver in further calculations.
“Target calculation”, prioritization of the druggable targets by the DDA system
Next, the AEL of druggable genes (drivers or indirect targets) is calculated based on all available evidence which links each druggable gene to all potential driver genes present in the same patient (“target calculation”). Associations between the potential driver genes and a target gene, which indicate that drugs targeting the druggable target gene are more effective in the presence of a driver gene count as a positive association, associations which indicate that drugs targeting the target gene in the presence of the driver is less effective counts as a negative association. The evidence level of all druggable targets’ associations with all mutant genes in the same patient present is aggregated to calculate the AEL score of the druggable target.
“MTA calculation”: prioritization of the MTAs by the DDA system
Finally, the AEL score of potential MTAs is calculated based on the AEL of all associations between the MTA and all potential drivers and associated druggable targets (“MTA calculation”). Associations between the potential driver genes and an MTA indicating that the MTA is more effective in the presence of the potential driver gene, or the particular molecular alteration of the driver gene, are calculated as positive associations. Associations between the potential driver genes and an MTA indicating that the MTA is less effective in the presence of the potential driver gene or the particular molecular alteration of the driver gene are calculated as negative associations. The AEL scores of the targets with a positive aggregated score are added to the AEL score of the associated MTAs, while the AEL of targets of the MTA with a negative AEL score is deducted from the AEL of the associated MTA. The AEL of the potential drivers whose aggregated score of the associations with the MTA are positive is added to the AEL score of the drug, while AEL of the potential drivers whose aggregated score of the associations with the MTA is negative are deducted from the AEL score of the MTA.
The final AEL score of the MTA is based on the number and evidence level of the associations between the MTA and all potential drivers and targets, and the AEL of the associated potential driver genes and druggable targets of the same patient. Consequently, AEL of the MTA is based on the aggregated score of multiple pieces of evidence that links the MTA to the whole molecular profile of the tumor, instead of assigning a rank to the MTA based on the highest available piece of evidence, which links the MTA to one driver.
DDA based on the molecular profiles of patients treated in the SHIVA01 clinical trial
Version 1.64 of the system tested in this study connected 709 drivers and targets with 631 MTAs (registered and in clinical development). These drivers, targets, and MTAs were linked together with 12,620 connections (“if–then” rules), including 6089 clinical evidence and 6531 preclinical evidence-based connections deducted from 7306 publications.
In the randomized SHIVA01 trial, 11 MTAs were selected following a predefined treatment algorithm, molecular alterations–MTA pairs, based on clinically validated biomarker or supporting preclinical evidence. Of the 195 randomized patients, 170 were treated with MTAs based on SNVs in 50 genes and CNVs in 24 genes by NGS and expression level of three hormone receptors by immunohistochemistry (IHC) including patients after crossover from the chemotherapy treatment arm15. Both outcome data and complete molecular profiles were available for 113 patients7.
Molecular profiles of these 113 patients were uploaded into the DDA system. The system prioritized all driver alterations and identified associated targets and MTAs that are positive or negative connections with the molecular profiles of all patients. The system used multiple connections based on functional evidence (clinical, preclinical, and in silico) and frequency-based evidence for each patient. DDA identified, on average, 17 associated targets and 47 associated MTAs. DDA assigned an AEL score to all MTAs used to treat patients in the SHIVA01 trial (Supplementary Tables 1 and 2, Tables 1 and 2, and Fig. 2a, b).
Association between the DDA AEL scores and the disease control rates of MTAs used in the SHIVA01 trial
The calculated average AEL scores of the employed MTAs were around threefold higher in case of patients achieving disease control (partial response (PR) and stable disease (SD)) than in non-responders in the SHIVA01 trial7,15,16,17, and the difference was significant (1523 versus 580, P = 0.037) (Fig. 3).
Molecularly targeted drugs used in the SHIVA01 were arbitrarily classified into three groups according to their DDA AEL scores into low (AEL < 0), intermediate (0 < AEL < 1000), and high (1000 < AEL) DDA tiers (Fig. 4). The cut-off for the high DDA tier was chosen based on the distribution histogram of AEL values blind to the outcome data (Supplementary Fig. 1). In the low tier (n = 12, 11%), the calculated AEL values were negative, indicating that the calculated evidence level of associations predicting resistance (negative drug relations) was higher than the AEL of positive associations. In the intermediate (n = 65, 57%) and high tiers (n = 36, 32%), the AEL of positive associations was higher than the AEL of negative associations. The disease control rate (DCR) (PR and SD) was 56% in the total patient population, 17% in the low, 55% in the intermediate, and 69% in the high DDA tiers, respectively.
Association between the DDA AEL scores, progression-free survival (PFS), and overall survival (OS) of MTAs used in the SHIVA01 trial
The PFS and OS of the DDA tiers were compared using log-rank tests (Fig. 5). The median PFS of the whole population was 3.48 months (n = 113). The median PFS was significantly longer in patients treated with drugs of high AEL (1000 < AEL) than in the low AEL (AEL < 0) DDA tier (3.95 versus 1.95 months, P = 0.044); hazard ratio: (p(AEL < 0)/p(AEL > 1000)): 1.91 (95% CI 0.86–4.23) and (p(AEL > 1000)/p(AEL < 0)): 0.52 (95% CI 0.24–1.16). The median PFS of the intermediate tier (0 < AEL < 1000) was 3.11 months (Fig. 5a). To further evaluate the relationship between the PFS and the AEL scores of MTAs patients were grouped according to PFS by above and below the median PFS of 3.48 months, there is a statistically significant difference in the average AEL values of the two groups: 1625.9 (above median PFS) versus 611.9 (below median PFS) (P = 0.02336).
The median OS of the whole population was 11.15 months and the median OS of the intermediate tier (0 < AEL < 1000) was 11.63 months (Fig. 5b). Patients in the SHIVA01 trial were allowed to crossover between the MTA and chemotherapy arms, limiting the evaluation of the OS results. However, there was a trend of patients having worse OS in the low DDA tier; the difference did not reach statistical significance (8.78 months versus 12.09 months).
Concordance between the DDA-based therapy recommendations and MTAs chosen in the SHIVA01 trial
DDA would assign the highest AEL score to the same MTA used in SHIVA01 in 60 patients (53%) and assign a different MTA in 53 patients (47%) out of the 11 drugs used in the SHIVA01 trial. If the trial was today, DDA would still assign the highest AEL score to the same MTA in 28 patients (25%) and would assign a different MTA for 85 patients (75%) out of the 631 MTAs (registered and under development) available today in the system’s current database. For simplicity, letrozole and tamoxifen were grouped for this analysis (Fig. 6). DDA was more likely to assign the highest AEL to the same drug for patients in the higher DDA tiers (DDA Tier I: 0/12 (0%), DDA Tier II: 13/65 (20%), DDA Tier III: 15/36 (42%)) from the currently available 631 MTAs. In cases where patients responded to the MTA, the AEL score of the highest ranking MTA by the DDA out of the MTAs available today is the same or very similar to the AEL score of the MTA used in the SHIVA01.
Here we report a computational method, DDA, to prioritize MTAs for precision oncology. DDA is an “open box,” explainable AI system. The rationale and mechanism of action are described here in detail, and the concrete algorithm and the whole evidence database used for each case can be retrieved and reviewed for human supervision for quality assurance purposes. The system is a rule-based expert system that contains a large number of if–then relations and algorithms. This type of device is in the scope of the Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan of the FDA14.
We used the molecular information and outcome data of patients treated with MTAs in the SHIVA01 trial for the clinical performance analysis of the DDA-based software system, the Realtime Oncology Treatment CalculatorTM. It is important to note that the data of SHIVA01 were not used to train or optimize the AI system. This is important because this excludes the possibility of “overfitting,” the common issue of AI systems. The patients’ molecular profiles were uploaded into the software system, and the AEL scores MTAs associated with the profiles were calculated blindly to the outcome data prospectively on the retrospective data. Next, the clinical benefit of MTAs with different AEL scores was analyzed in different patient groups according to the treatment outcomes. There was a statistical difference between the AEL scores of MTAs according to the disease control and below or above median PFS. In addition, the average PFS was significantly longer in patients treated with drugs of high AEL (1000 < AEL) than in the low AEL (AEL < 0) DDA tiers. The thresholds for these tiers were arbitrary, selected based on the distribution of AELs (Supplementary Fig. 1). For example, setting the threshold at 500 increases the DCR to 71% in the high tier. Similarly, the Kaplan–Meier estimation is also slightly more significant when the high AEL is defined as >500 (P = 0.042) than with >1000 group (P = 0.044). There was a case with BRAFV600E mutation with an outlying high AEL value. If we omitted this case from the analysis, the difference between the AEL values of the DCR and PD group would have been even more significant (P = 0.015 versus P = 0.037).
There were several advantages of using the data from the SHIVA01 for the clinical performance analysis of the DDA system. In this clinical trial, the same standard molecular diagnostic tests were performed for all patients, and the outcome was assessed by the same methodology. The patients represented a relevant distribution of different types of advanced solid cancers. SHIVA01 was a prospective trial. Therefore, there was no bias toward reporting only the outstanding responders like in case studies.
The limitations of this study are that patients included in the SHIVA01 were heavily pretreated, which limited the assess of the full potential of MTAs. The focused 50 genes NGS panel used in the trial is still widely used in molecular diagnostics today. This panel was designed to detect the most frequent driver alterations (“hot spots”) in epithelial cancers primarily included in this trial. Therefore, it was sufficient to identify the expected 3–4 driver alterations per tumor. However, the possibility of choosing from more MTAs than the 11 MTAs used in SHIVA01, based on comprehensive molecular profiling (300–600 gene panels, WES or WGS) covering all potential driver alterations, would more likely increase than decrease the clinical performance of DDA.
Using combinations of two or three therapies is also a logical solution to fight the complexity of cancer. The I-PREDICT trial and real-world data show that combination therapies covering more than half of the drivers overall achieve superior results than monotherapies covering less than half of drivers12,18. However, combination therapies also carry the disadvantage of higher risk of toxicity and further increase of the targeted therapies’ financial burden.
In SHIVA01, MTAs were mainly used in monotherapies targeting predefined molecular alterations. The average number of driver alterations in the three DDA tiers was very similar (3.92 in the low, 3.26 in the intermediate, and 3.73 in the high DDA tier). Thus, DDA AEL scores of MTAs in monotherapy correlated with better clinical outcomes independently from the number of concurrent driver alterations in this study.
These results indicate that the DDA system can identify MTAs that are more likely to be effective despite unmatched driver alterations. This outcome can be achieved due to the network analysis system design of DDA. DDA identifies the MTA which inhibits the driver alteration(s) or indirect druggable molecular target(s), which have the highest number and level of positive associations—and least negative associations—with the highest number of all driver alterations of the same tumor (Figs. 1 and 2a, b).
The potential clinical utility of the current version of DDA based on the data presented here confines to the support of relative prioritization between MTAs in the absence of any randomized clinical trial evidence that would indicate the superiority of one MTA over the other. In SHIVA01, all MTAs were used in the presence of a predefined target chosen by the investigators’ molecular tumor board. MTAs associated with registered companion diagnostics (Level1/Tier I evidence by ESCAT) were excluded. The efficacy of MTAs used in the study was not different according to the lower ESCAT tiers9. Consequently, data presented in this study imply that choosing an MTA based on the higher DDA score will more likely lead to the same or better clinical benefit rather than worse in comparison to randomly choosing another MTA with a DDA lower score at the same evidence level.
In clinical practice, the DDA score can be useful if multiple MTAs are available at the same evidence levels. MTAs at the same evidence level should be registered in the same indication at the same treatment line, associated with the same companion diagnostic (ESCAT Tier I) molecular genetic alterations, or associated with multiple companion diagnostic alterations present in the same tumor or not linked to any companion diagnostic tests. DDA can also be useful when there are no registered treatment options available, but multiple MTAs in clinical use or clinical trials match molecular genetic alterations present in the patient’s tumor at the same evidence level. The ultimate goal of DDA is to help the work of molecular tumor boards planning of the personalized treatment strategy of all treatment lines for each patient. We provide three clinical examples for the clinical utility in the supplementary information (Supplementary Note 1).
It is important to note that the current DDA version is not intended to replace any on-label therapies with an MTA not registered in the same indication or at a higher clinical evidence level, based on the DDA score alone. We will need further retrospective and prospective clinical trials to evaluate the absolute clinical benefit of MTAs according to their DDA score, and we will need randomized trials to introduce new treatment options based on DDA in the registered treatment protocols. MTAs and corresponding DDAs can be co-developed and co-registered in the future. Based on data presented here, DDA can be potentially superior as a companion diagnostic method over most single biomarkers identifying patients who most benefit from an MTA, leading to the acceleration, reduced cost, and risk of drug development.
The DDA system can be used by physicians, pathologists, molecular biologists, or other professionals with relevant biomedical backgrounds who are specifically trained for the system’s utility and limitations. DDA is designed to be open for easy human supervision. The system generates the lists of published evidence used for the DDA score calculations and generates text descriptions of the information available on the molecular alterations and associated MTAs. DDA is linked to an online case management system designed for dynamic decision support for precision oncology. DDA can be updated as new molecular diagnostic test results are available for the patient. The system combines all available results from different laboratories and types of tests: NGS, fluorescent in situ hybridization (FISH), IHC (for example, ER, PDL-1), TMB, and MSI to generate a prioritized list of MTAs associated with all these results combined.
An essential feature of DDA that it is based on a continuously expanded extensive database of evidence-based associations between driver genes, targets, and MTAs. Therefore, DDA mirrors the improvement of our understanding of cancer biology and the availability of novel molecularly targeted treatments over time. As a result, the AEL scores of the same MTAs and drug assignments for the same patient can change over time. The other important feature of the system is that since the targets of MTAs are known by definition, experience (preclinical and clinical) with MTAs in the presence of different driver mutations increases our understanding of the functional relevance of specific mutations and teaches about the associations between drivers and druggable targets. This further teaches the network, which can better and better predict the outcome of any future MTA without previous experience based on the functional significance of its targets.
Since DDA uses standardized evidence databases and algorithms to aggregate evidence-based associations, the predictive performance of the computational system and the AEL score can be continuously tested on test databases and on real-world experience to ensure that the performance is reproducible, and the new version of the system is at least as good or better in predicting response to MTAs. Regarding this point, we have reproduced the analysis presented in this report with a newer 1.67 version of the Realtime Oncology Treatment CalculatorTM and found the same or slightly better results (data not shown).
The use of AI in precision medicine, especially in precision oncology, is highly anticipated19. The FDA has initiated public discussion on this subject and created an action plan on how to regulate AI-based software as medical devices (SaMD)14. The definition of AI used by the FDA is any “intelligent” software. AI tools transform information by algorithms to improve diagnostic or treatment decisions. Algorithms in AI-assisted treatment assignment algorithms can be developed manually as in the current version of the DDA or using ML on training datasets. A National Cancer Policy Forum has already reviewed this emerging field: Improving Cancer Diagnosis and Care: Clinical Application of Computational Methods in Precision Oncology20.
It is important to note that most software solutions currently used to interpret NGS results are reference tools. These systems are designed to store and retrieve information from structured databases and guidelines to automatize interpretation for electronic reporting. The expected performance of these reference tools is to help users to reach the same conclusion they would reach anyway, but faster and more conveniently. These systems are based on large databases of single gene/biomarker—MTA parings prioritized based on the evidence level of the evidence selected by a team of experts who perform the literature search and curation21. Some of these systems use AI for text recognition (natural language processing, NLP) to automatize the medical literature search before human curation, but this does not mean that they also use AI-based drug-assignment algorithms22.
There are ongoing efforts to develop more complex decision support systems. The “simplified interventional mapping system” (SIMS) developed by Worldwide Innovative Networking (WIN) consortium offers a systematic approach for the prioritization of druggable targets (“interventional points”) using a scoring system, which reflects the type and extent of changes in genes connected to a group of targets23. The approach was tested in the WINTHER trial24. Results indicated that besides the “matching score” of the DNA alterations, the ranking of targets using the WINTHER algorithm calculating the extent of mRNA expression in the tumor compared to the corresponding normal tissue also correlated with the clinical benefit of MTAs24. The WINTHER trial results also support the prioritization of targets using mathematical algorithms based on the complex biology of cancer as predictive diagnostic tools instead of using single biomarkers.
PreciGENETM (CureMatch Inc., San Diego, CA, USA) primarily focusing on the identification of effective combination therapies is an important example of rule-based AI-assisted computational tools like DDA with the Realtime Oncology Treatment CalculatorTM25.
We expect that more and more solutions similar to these systems will emerge in the next few years. We also expect that large randomized clinical trials will compare different AI-based treatment assignment algorithms against another26,27. The goal will be to find the right predictive AI algorithm for every MTA in each indication. AI systems will never “make decisions.” The predictive scores generated by these systems will eventually help us make better treatment decisions in the next, digital age of precision oncology.
Data collection and analysis
The software of the DDA system used in this study was the Realtime Oncology Treatment CalculatorTM version 1.64. PFS and OS data were represented by Kaplan–Meier estimation and the survival end points were compared using log-rank tests. Hazard ratio was computed by log-rank method. T-test was used to determine the significance of difference between the means of groups. Statistical analysis was performed using the NumPy28, SciPy29, and lifelines30 modules of Python 3.7.
The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of the National Institute of Pharmacy and Nutrition (approval ID: OGYEI/50268/2017). The SHIVA01 clinical trial was approved by the Ile-de-France ethics committee and informed consent was obtained from all human participants. The trial was carried out in accordance with the Declaration of Helsinki, the Good Clinical Practice guidelines of the International Conference on Harmonization, and relevant French and European laws and directives.
During the preparation, submission, conduct, and analysis of this study, we complied with all relevant ethical regulations.
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
The data generated and analyzed during this study are described in the following data record: https://doi.org/10.6084/m9.figshare.14414612 (ref. 31). The data are available within the paper and/or uploaded in reusable format to the figshare repository, and are openly available at https://doi.org/10.6084/m9.figshare.14331323 (ref. 32).
RealTime Oncology Treatment Calculator™ code is proprietary to Oncompass Medicine Inc. and it is available for collaborative purposes upon request.
Peták, I., Schwab, R., Orfi, L., Kopper, L. & Kéri, G. Integrating molecular diagnostics into anticancer drug discovery. Nat. Rev. Drug Discov. 9, 523–535 (2010).
Le Tourneau, C., Borcoman, E. & Kamal, M. Molecular profiling in precision medicine oncology. Nat. Med. 25, 711–712 (2019).
Flaherty, K. T. et al. NCI-MATCH Team. Molecular landscape and actionable alterations in a genomically guided cancer clinical trial: National Cancer Institute Molecular Analysis for Therapy Choice (NCI-MATCH). J. Clin. Oncol. 38, 3883–3894 (2020).
Mangat, P. K. et al. Rationale and design of the Targeted Agent and Profiling Utilization Registry (TAPUR) Study. JCO Precis. Oncol. 2018 https://doi.org/10.1200/PO.18.00122 (2018).
Tsimberidou, A. M. et al. Long-term overall survival and prognostic score predicting survival: the IMPACT study in precision medicine. J. Hematol. Oncol. 12, 145 (2019) .
Massard, C. et al. High-throughput genomics and clinical outcome in hard-to-treat advanced cancers: results of the MOSCATO 01 trial. Cancer Discov. 7, 586–595 (2017) .
Le Tourneau, C. et al. Molecularly targeted therapy based on tumour molecular profiling versus conventional therapy for advanced cancer (SHIVA): a multicentre, open-label, proof-of-concept, randomised, controlled phase 2 trial. Lancet Oncol. 16, 1324–1334 (2015).
Mateo, J. et al. A framework to rank genomic alterations as targets for cancer precision medicine: the ESMO Scale for Clinical Actionability of molecular Targets (ESCAT). Ann. Oncol. 29, 1895–1902 (2018).
Moreira, A. et al. Efficacy of molecularly targeted agents given in the randomised trial SHIVA01 according to the ESMO Scale for Clinical Actionability of molecular Targets. Eur. J. Cancer 121, 202–209 (2019).
Tate, J. G. et al. COSMIC: the catalogue of somatic mutations in cancer. Nucleic Acids Res. 47, D941–D947 (2019).
ICGC/TCGA Pan-Cancer Analysis of Whole Genomes Consortium. Pan-cancer analysis of whole genomes. Nature 578, 82–93 (2020) .
Sicklick, J. K. et al. Molecular profiling of cancer patients enables personalized combination therapy: the I-PREDICT study. Nat. Med. 25, 744–750 (2019).
McCarthy, D. Computers getting the drift. Philos. Trans. A Math. Phys. Eng. Sci. 365, 3019–3031 (2007).
U.S. Food and Drug Administration. Artificial Intelligence/Machine Learning (AI/ML) Software as a Medical Device Action Plan (U.S. Department of Health and Human Services, 2021).
Belin, L. et al. Randomized phase II trial comparing molecularly targeted therapy based on tumor molecular profiling versus conventional therapy in patients with refractory cancer: cross-over analysis from the SHIVA trial. Ann. Oncol. 28, 590–596 (2017).
Kamal, M. et al. Revisited analysis of a SHIVA01 trial cohort using functional mutational analyses successfully predicted treatment outcome. Mol. Oncol. 12, 594–601 (2018).
Servant, N. et al. Bioinformatics for precision medicine in oncology: principles and application to the SHIVA clinical trial. Front. Genet. 5, 152 (2014).
Kato, S. et al. Real-world data from a molecular tumor board demonstrates improved outcomes with a precision N-of-One strategy. Nat. Commun. 11, 4965 (2020).
Panagiotou, O. A. et al. Clinical application of computational methods in precision oncology: a review. JAMA Oncol. 6, 1282–1286 (2020).
National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Health Care Services; National Cancer Policy Forum. Improving cancer diagnosis and care: clinical application of computational methods in precision oncology, Proc. Workshop (eds. Nass, S. J. et al.) (National Academies Press, 2019).
Chakravarty, D. et al. OncoKB: A Precision Oncology Knowledge Base. JCO Precis Oncol. 2017 https://doi.org/10.1200/PO.17.00011 (2017).
Itahashi, K. et al. Evaluating clinical genome sequence analysis by Watson for genomics. Front. Med. (Lausanne) 5, 305 (2018).
Lazar, V. et al. A simplified interventional mapping system (SIMS) for the selection of combinations of targeted treatments in non-small cell lung cancer. Oncotarget 6, 14139–14152 (2015).
Rodon, J. et al. Genomic and transcriptomic profiling expands precision cancer medicine: the WINTHER trial. Nat. Med. 25, 751–758 (2019).
Boichard, A., Richard, S. B. & Kurzrock, R. The crossroads of precision medicine and therapeutic decision-making: use of an analytical computational platform to predict response to cancer treatments. Cancers (Basel) 12, 166 (2020).
Mittra, A. & Moscow, J. A. Future approaches to precision oncology-based clinical trials. Cancer J. 25, 300–304 (2019).
Topol, E. J. A decade of digital medicine innovation. Sci. Transl. Med. 11, eaaw7610 (2019).
Harris, C. R. et al. Array programming with NumPy. Nature 585, 357–362 (2020).
Virtanen, P. et al. and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261–272 (2020).
Davidson-Pilon, C. et al. CamDavidsonPilon/lifelines: v0.24.8 (Version v0.24.8). Zenodo. https://doi.org/10.5281/zenodo.3833188 (2020).
Petak, I. et al. Metadata record for the manuscript: a computational method for prioritizing targeted therapies in precision oncology: performance analysis in the SHIVA01 trial. figshare https://doi.org/10.6084/m9.figshare.14414612 (2021).
Petak, I. et al. Datasets for the article: A computational method for prioritizing targeted therapies in precision oncology: performance analysis in the SHIVA01 trial. figshare https://doi.org/10.6084/m9.figshare.14331323 (2021).
This work was supported by the Agence Nationale de le Recherche, Site de Recherche Intégrée contre le Cancer (SiRIC) (ERA PerMed (ERAPERMED2018-078), and the Hungarian Innovation Agency (NVKP_16-1-2016-0005, 2019-1.1.1-PIACI-KFI-2019-00367, TUDFO/51757/2019-ITM, TKP 2020-NKA-19).
I.P. and R.S. are employees and equity holders in Oncompass Medicine Inc. A.D., R.D., P.F., B.V., E.V., D.T., D.L., J.D., and C.H. are employees of Oncompass Medicine Inc. The remaining other authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Petak, I., Kamal, M., Dirner, A. et al. A computational method for prioritizing targeted therapies in precision oncology: performance analysis in the SHIVA01 trial. npj Precis. Onc. 5, 59 (2021). https://doi.org/10.1038/s41698-021-00191-2