# Prostate-specific antigen dynamics predict individual responses to intermittent androgen deprivation

## Abstract

Intermittent androgen deprivation therapy (IADT) is an attractive treatment for biochemically recurrent prostate cancer (PCa), whereby cycling treatment on and off can reduce cumulative dose and limit toxicities. We simulate prostate-specific antigen (PSA) dynamics, with enrichment of PCa stem-like cell (PCaSC) during treatment as a plausible mechanism of resistance evolution. Simulated PCaSC proliferation patterns correlate with longitudinal serum PSA measurements in 70 PCa patients. Learning dynamics from each treatment cycle in a leave-one-out study, model simulations predict patient-specific evolution of resistance with an overall accuracy of 89% (sensitivity = 73%, specificity = 91%). Previous studies have shown a benefit of concurrent therapies with ADT in both low- and high-volume metastatic hormone-sensitive PCa. Model simulations based on response dynamics from the first IADT cycle identify patients who would benefit from concurrent docetaxel, demonstrating the feasibility and potential value of adaptive clinical trials guided by patient-specific mathematical models of intratumoral evolutionary dynamics.

## Introduction

Prostate cancer (PCa) is the most common type of cancer in American men and the second leading cause of cancer mortality1. Following surgery or radiation, the standard treatment for hormone-sensitive PCa is continuous androgen deprivation therapy (ADT) at the maximum tolerable dose (MTD) with or without continuous abiraterone acetate (AA) until the tumor becomes castration resistant2. Importantly, advanced PCa is not curable because PCa routinely evolves resistance to all current treatment modalities. Continuous treatment approaches fail to consider the evolutionary dynamics of treatment response where competition, adaptation, and selection between treatment sensitive and resistant cells contribute to therapy failure3. In fact, continuous treatment, by maximally selecting for resistant phenotypes and eliminating other competing populations, may actually accelerate the emergence of resistant populations—a well-studied evolutionary phenomenon termed competitive release4.

In part to address this issue, prior trials have used intermittent ADT (IADT) to reduce toxicity and potentially delay time to progression (TTP). However, these trials were typically not designed with a detailed understanding of the underlying evolutionary dynamics. For example, a prospective Phase II trial of IADT for advanced PCa included an 9-month induction period in which patients were treated at MTD prior to beginning intermittent therapy5. We have previously postulated that only a small number of ADT-sensitive cells would typically remain after the induction period, thereby significantly reducing the potential of intermittent treatment to take advantage of the evolutionary dynamics3.

Fully harnessing the potential of intermittent PCa therapy requires identifying ADT resistance mechanisms, predicting individual responses, and determining potentially highly patient-specific, clinically actionable triggers for pausing and resuming IADT cycles. Progress in integrated mathematical oncology may make such analysis possible. Many mathematical models based on a variety of plausible resistance mechanisms have been proposed to simulate IADT responses3,6,7,8,9,10,11,12,13,14. Although these models can fit clinical data, they often rely on numerous model variables and parameters that in combination fail to adequately predict responses and outcomes for individual patients10. We hypothesize that PCa cells with stem-like properties (PCaSCs) may be, at least in part, responsible for tumor heterogeneity and treatment failure owing to their self-renewing, differentiating and quiescent nature15,16,17. Here, we define PCaSCs to be a population of cells that are less sensitive to low testosterone environments but still somewhat dependent on androgen receptor pathways. Simulating longitudinal prostate-specific antigen (PSA) levels in early IADT treatment cycles could help identify patient-specific PCaSC dynamics to computationally forecast individual disease dynamics and reliably predict IADT response or resistance in subsequent treatment cycles.

The first evidence of stem cells in the prostate was provided by Isaacs and Coffey18, who used androgen cycling experiments in rodents to show that castration resulted in involution of the prostate, whereas restored androgen levels resulted in complete regeneration of the prostate. These findings demonstrated that the normal prostate depends on androgens for maintenance. A small population of androgen-independent stem cells within the prostate epithelium divide to give rise to amplifying cells, which do not directly depend on androgen for their continuous maintenance, but respond to androgens by generating androgen-dependent transit cells. Approximately 0.1% of cells in prostate tumors express the stem cell markers CD44+/α2β1hi/CD133+19. A pre-clinical study by Bruchovsky et al. showed ADT selects for murine PCaSCs20. Analogously, Lee et al.21 demonstrated increased PCaSCs populations after ADT in patient-derived PCa cell lines, which can be reverted by the addition of functional AR. Combined, these results suggest evolution of or selection for pre-existing androgen-independent PCaSCs as a plausible explanation of the development of ADT resistance.

The purpose of this study is to evaluate individual PSA dynamics in early IADT treatment cycles as a predictive marker of response or resistance in subsequent treatment cycles. We hypothesize that patient-specific PCaSC division patterns underlie the measurable longitudinal PSA dynamics, and that a mathematical model of PCaSCs can be trained to predict treatment responses on a per-patient basis. Here, we present an innovative framework to simulate and predict the dynamics of PCaSCs, androgen-dependent non-stem PCa cells (PCaCs), and blood PSA concentrations during IADT. Our mathematical model of PCaSC enrichment is calibrated and validated with longitudinal PSA measurements in individual patients to identify model dynamics that correlate with treatment resistance. The model’s predictive power to accurately forecast individual patients’ responses to IADT cycles is evaluated in an independent patient cohort. These analyses suggest that PCaSC and PSA dynamics may potentially be used to personalize IADT, maximize TTP, and ultimately improve PCa outcomes. The calibrated and validated model is then used to generate testable hypotheses about patients that may benefit from concurrent chemotherapy.

## Results

### Model accurately simulates patient-specific response dynamics

The model was calibrated to and assessed for accuracy on longitudinal data from a prospective Phase II study trial conducted in 109 men with biochemically recurrent PCa treated with IADT22 (see Methods, Supplementary Fig. 1). Stratified random sampling23 was used to divide the data into training and testing cohorts. That is, the data were ordered according to number of cycles of treatment and then randomly sampled such that each cohort had a similar spread of number of cycles per patient. Assuming that uninhibited PCaSCs divide approximately once per day24, λ (day−1) was set to ln(2) and parameter estimation was used to find the remaining parameters. The model was calibrated to the training cohort data with two population uniform parameters (PSA production rate ρ = 1.87E-04 (µg/L day−1), decay rate φ = 0.0856 (day−1)) and two patient-specific parameters (median PCaSC self-renewal rate ps = 0.0278 (95% CI: [0.0265, 0.0611]) (non-dimensional), ADT cytotoxicity rate α = 0.0360 (95% CI: [0.0258, 0.1379]) (day−1)).

The model results captured clinically measured longitudinal PSA dynamics of individual responsive and resistant patients (Supplementary Fig. 2A–B) and the population as a whole (R2 = 0.74 Supplementary Fig. 2C). The corresponding PCaSC dynamics demonstrated a rapid increase in the PCaSC population in patients that became castration resistant compared with patients that remained sensitive throughout the trial. Simulations also showed that, in this model, the emergence of resistance is a result of selection for the PCaSCs during on-treatment phases. Coinciding with this transition to castration resistance, analysis of model parameters revealed that resistant patients had significantly higher PCaSC self-renewal rates than responsive patients (median ps = 0.0249 for responsive patients vs. ps = 0.0820 for resistant patients, p < 0.001, (95% CI: (0.0167, 0.0344) vs. (0.0472, 0.1460), respectively) Supplementary Fig. 2D). As such, we can conclude that those patients with a low ps are considered to be evolutionarily stable, whereas those with a higher ps are likely to develop resistance quicker. As shown in Supplementary Fig. 2D, analysis revealed that ps and α may be functionally related. Supplementary Table 1 shows the mean squared error comparison for possible relationships between these parameters. This analysis showed that multiple relationships could provide similar fits to the data. From this, we chose the exponential relationship, allowing PCaSC self-renewal rate ps to be the single, identifiable, independent patient-specific parameter.

To assess the accuracy of our model, we set the PSA production rate ρ and decay rate φ to the values obtained from the training cohort and identified patient-specific values for ps and corresponding α in the testing cohort. With these, the model was able to fit the data (R2 = 0.63) and the resulting parameter distributions and relationships were similar to those found in the training cohort (Fig. 1).

### PCaSC dynamics predict subsequent IADT cycle responses

To predict the evolution of resistance in subsequent treatment cycles, we fit the model to single treatment cycles for patients in the training set, again setting the PSA production rate ρ and decay rate φ to the values previously found (Fig. 2a). The self-renewal and ADT cytotoxicity rates maintained the exponential relationship, previously obtained when optimizing over all IADT cycles. The distributions of the self-renewal rate ps and the corresponding ADT cytotoxicity rate α, as well as their relative change from cycle to cycle, were used to predict responses in subsequent cycles of patients in the training cohort. To determine whether to categorize a patient as responsive or resistant based on our predictions, receiving operating characteristic curves25 from the results from the training cohort data were analyzed to find a threshold κi for each cycle (Fig. 2d). If P(Ω) > κi, then the patient was predicted as resistant in cycle i. For each cycle i, a value of κi was chosen that would maximize the sensitivity (predicting resistance when a patient is indeed resistant), and specificity (predicting resistance when a patient is actually responsive) of the training cohort.

Using these thresholds, model forecasting was completed on the testing set and the patient was classified as either responsive or resistant in the subsequent treatment cycle. Representative examples of second and third cycle predictions for one responsive patient and one resistant patient from the testing cohort are shown in Fig. 3. Patient 017 was a continuous responder who underwent three cycles of IADT before the end of the trial. The model correctly predicted responsiveness in cycles two and three based on the parameters fitted in cycles one and two, respectively. Patient 054 became resistant in the third IADT cycle and the model was able to correctly predict response in cycle two and resistance in the third cycle based on the thresholds learned in the training set. The model yielded a sensitivity of 57% and specificity of 94% over all subsequent IADT cycles for patients in the test cohort. The overall accuracy of the model was 90%.

To evaluate if the low sensitivity is an artifact of which resistant patients are stratified into the training cohort, we performed the parameter optimization as well as forecast analysis by re-randomizing the patients into training and testing cohorts five times. The 95% confidence intervals for the uniform parameters φ and ρ, the threshold values ki, as well as the overall accuracy are shown in Supplementary Table 2. As expected, the threshold values varied dependent upon the randomization. The overall accuracy had a relatively narrow 95% confidence interval ranging between 75.34% and 88.66%, with sensitivity between 43% and 61% and specificity between 75% and 91%. To better account for under-represented resistant patients in the training cohort, we utilized a bootstrapping leave-one-out design and performed the optimization and forecast analysis for each of the 70 patients individually. This resulted in training set-specific φ, ρ, κ2, κ3, and κ4 (95% confidence intervals shown in Supplementary Table 3), yielding a sensitivity of 73%, specificity of 91%, and overall accuracy of 89% (mean AUC = 0.80 ± 0.06). The ROC curve for each individual patient, as well as the average ROC curve for cycles 2 through 4 are shown in Fig. 4.

### IADT without induction period significantly increases TTP

In a study by Crook et al.5 continuous ADT was compared with IADT in localized PCa patients and they found that IADT was non-inferior to ADT using overall survival as the clinical endpoint. A similar study by Hussain et al.26 conducted in metastatic, hormone-sensitive PCa patients found that neither regimen proved superior. These findings are likely the result of the 7–8 months induction period, which resulted in the competitive release of the resistant phenotype once the androgen sensitive subpopulation was eliminated3. Administering IADT without such an induction period would allow sufficient time for the sensitive subpopulation to efficiently compete with the resistant subpopulation and prolong TTP and ultimately, overall survival.

### Alternative treatment decision thresholds also increase TTP

In the protocol used in the Bruchovsky study, treatment was paused after 36 weeks if PSA fell below 4 µg/L after both 24 and 32 weeks of treatment. To investigate if a patient-specific approach that depends on individual pre-treatment PSA levels would be more effective, we simulated IADT using a 50% decline in PSA as a treatment cessation trigger, as well as 90% and 30% decline, and compared the results against those obtained in the Bruchovsky trial. Pausing treatment once PSA fell below 30% and 90% of the pre-treatment PSA, checking every two weeks, resulted in significantly longer TTP when compared with the IADT protocol used in the trial (Fig. 6d–f). In addition, the average cumulative dose could be significantly reduced depending on the threshold used as shown in Fig. 6a–c. With a 90% decline from pre-treatment PSA, the average cumulative dose is ~64% of that of the standard IADT. A 50% decline results in ~30% of the cumulative dose used in standard IADT, whereas 30% results in ~27%. Controlling for the total time that each patient participated in the trial results in ~60%, 25%, and 22% of the cumulative dose used in standard IADT for thresholds of 90%, 50%, and 30%, respectively.

### Concurrent docetaxel administration provides favorable TTP

As PCaSCs are resistant to ADT, it may be beneficial to administer alternative therapies that do not work directly via the AR pathway. AA, enzalutamide, apalutamide, and docetaxel (DOC) have shown benefit when used concurrently with ADT in various PCa settings, including both high- and low-volume disease in metastatic hormone-sensitive PCa27,28,29,30,31. Thus, we sought to evaluate potential responses to combination therapy in the biochemically recurrent setting. We investigated the effect of concurrent DOC by simulating six cycles of DOC early during IADT (as defined in the Methods section). Model analysis showed that administering DOC early with concurrent ADT followed by IADT may be able to increase TTP (Fig. 7a, c). In particular, model simulations showed that the majority of patients may receive a benefit of >4.2 months (length of six cycles of DOC) when given induction DOC with IADT (Fig. 7b). Equally important, simulations can identify selected patients for whom early concurrent DOC may not provide gains in TTP (Fig. 7d).

### First cycle PCaSC self-renewal rate stratifies patients who can benefit from DOC

As shown above, the stem cell self-renewal rate plays a significant role in IADT and could accurately predict a patient’s response in subsequent cycles after just the first cycle. With this, we used the first cycle ps value to simulate six cycles of DOC with concurrent ADT followed by IADT and found that patients with ps ≥ med(ps) could benefit from DOC after the first cycle of IADT (Fig. 8a). There was a significant difference in progression free survival between patients with a high ps and those with a low ps, both when simulating IADT with and without DOC after the first cycle (Fig. 8b). Though the difference in TTP was not significantly higher with DOC given after the first cycle (Fig. 8c), median TTP increased from 35 to 45 months in those patients with ps ≥ med(ps) (Fig. 8b, d, e).

## Discussion

ADT is not curative for advanced PCa, as patients inevitably develop resistance. IADT is a promising approach to counteract evolutionary dynamics by reducing competitive release of the resistant subpopulation during treatment holidays. As IADT is highly dynamic, maximum efficacy requires continuous, accurate estimates of sensitive and resistant subpopulations.

Here, we present a simple mathematical model of evolutionary dynamics within biochemically recurrent PCa during IADT. The model has been trained with two parameters that are uniform across all patients and only two patient-specific parameters, which are interconnected, allowing us to further reduce them to a single, measurable parameter for each patient. Simulations suggest that, in this model, the evolution of PCaSCs is highly correlated with the development of resistance to IADT and may consequently be a plausible mechanism of resistance development. PSA dynamics suggest that resistant patients are likely to have higher PCaSC self-renewal rates than responsive patients, leading to increased production of PCaSCs and ultimately differentiated cells, thereby accelerating PSA dynamics with each treatment cycle. Our results are similar to prior studies in glioblastoma that cancer stem cell self-renewal is likely to increase during prolonged treatment32.

Using longitudinal PSA measurements and observed clinical outcomes from the IADT trial by Bruchovsky et al.22, the model was calibrated to clinical data and predicted the development of resistance with 89% accuracy. Study analysis produced four important clinical findings: (1) PSA dynamics provide actionable triggers for PCa treatment personalization, vis-à-vis static PSA values with highly debated clinical utility33. (2) IADT outcomes in prior studies may be adversely affected by the long induction periods that accelerate selection for treatment-resistant subpopulations. (3) Patient-specific PSA treatment thresholds relative to pre-treatment burden rather than a fixed value for all patients could significantly improve IADT responses. (4) Early treatment response dynamics during IADT can identify patients that may benefit from concurrent docetaxel treatment and, maybe more importantly, identify patients who are adequately treated with IADT alone.

Our study demonstrated the value of ongoing model simulations in predicting outcomes from each treatment cycle throughout the course of therapy. By integrating data that becomes available with each additional treatment cycle to adaptively inform tumor population dynamics, model simulations predict the response to the next cycle with a sensitivity and specificity of 73% and 91%, respectively and an overall accuracy of 89%. This ability to learn from early treatment cycles and predict subsequent responses adds an essential degree of personalization and flexibility to a cancer treatment protocol—a game theoretic strategy termed Bellman’s Principle of Optimality that greatly increases the physician’s advantage34.

For those patients predicted to become resistant in the next cycle of IADT, an actionable model would also predict alternative treatments that could produce better clinical outcomes. The role of docetaxel in metastatic, hormone-sensitive PCa has been investigated in three studies in the past five years. The GETUG-AFU15 study found a non-significant 20% increase in overall survival in high volume disease (HVD) patients who received DOC concurrently with continuous ADT, but no survival benefit in those with low volume disease (LVD)35. Subsequently, the results of the STAMPEDE trial found that DOC administration resulted in a >12 months overall survival benefit with a median follow-up of 43 months29. Finally, the CHAARTED trial showed a statistically significant overall survival benefit from adding DOC in patients with HVD; however, no statistically significant survival benefit was found in LVD patients30. Here we explored the option of adding docetaxel in the treatment of biochemically recurrent PCa. Coinciding with previous findings, we found that concurrent docetaxel did not provide a significant benefit for the entire patient cohort. However, we found that analyzing PSA dynamics from the first IADT cycle stratifies patients who may receive benefit from concurrent administration of docetaxel. More importantly, the model was able to identify patients who would not receive a significant benefit in TTP from concurrent DOC treatment. These results emphasize that heterogeneity in patient responses can be harnessed by quantitative models to personalize treatment protocols.

Model validation showed that with two uniform parameters learned from our training cohort, only 63% of the data were accurately captured in the testing cohort. Although this may be perceived as a limitation of the model, the primary objective of this study was not to fit data with highest accuracy but to develop a predictive model that could be clinically actionable. Allowing for more patient-specific parameters substantially increases data fit, but compromises the ability to predict cycle-by-cycle dynamics as insufficient data are collected to inform each model parameter on a per-patient basis.

Since only 21% of the 70 patients progressed before the conclusion of the trial22 (Supplementary Fig. 1), the clinical data included more responsive than resistant patients. This was overcome by bootstrapping the data, with more balanced data being poised to advance clinical utility. Of note, the presented model was calibrated and validated on patients with predominantly stage T2b and T3 disease, with Gleason scores ranging from 2 to 922. As additional data become available, the developed framework may be generalizable and able to predict how PCa patients of different stages will respond to IADT with comparable accuracy. Further work in patients with more advanced or metastatic PCa is needed.

The mathematical model was developed with the intent to make clinically actionable predictions. To balance model complexity and prediction accuracy, the presented simple model has been derived36. Although numerous additional biological mechanisms are contributing to the observed PSA dynamics, it will become impossible to calibrate and validate their contributions from the measured clinical data. For reproducibility, we report the trained parameter values for the presented model. However, modification of the functional form of the developed model terms or inclusion of additional mechanisms will change each parameter. Therefore, herein estimated parameter values should not be taken as biological ground truth, nor are they translatable to different mathematical models.

A perceived limitation of the quantitative model may be the use of PSA dynamics as the sole biomarker of progression. PCa can become aggressive and metastatic despite low levels of serum PSA37,38 with development of androgen independence, most notably in neuroendocrine PCa. Additional serum biomarkers such as circulating tumor cells (CTCs) and cell-free DNA (cfDNA) may prove useful in estimating intratumoral evolutionary dynamics in subsequent trials. With the CellSearch platform, higher CTCs enumeration more than five cells per 7.5 mL of peripheral blood has been shown to be prognostic and portend worse overall survival in metastatic CRPC patients39; however detecting CTCs in biochemically recurrent patients has been labor-intensive with low yield40.

In conclusion, our study demonstrates that a simple mathematical model based on cellular dynamics in PCa can have a high predictive power in a retrospective data set from patients with biochemically recurrent PCa undergoing IADT. In particular, we demonstrate the model can use data from each treatment cycle to estimate intratumoral subpopulations and accurately predict the outcomes in subsequent cycles. Furthermore, in patients who are predicted to fail therapy in the next cycle, alternative treatments for which a response is more likely can be predicted. We conclude that PSA dynamics can prospectively predict treatment response to IADT, suggesting ways to adapt treatment to delay TTP.

## Methods

The Bruchovsky prospective Phase II study trial was conducted in 109 men with biochemically recurrent PCa22. IADT consisted of 4 weeks of Androcur as lead-in therapy, followed by a combination of Lupron and Androcur, for a total of 36 weeks. Treatment was paused if PSA had normalized (< 4 µg/L) at both 24 and 32 weeks, and resumed when PSA increased above 10 µg/L. PSA was measured every 4 weeks. Patients whose PSA had not normalized after both 24 and 32 weeks of being on treatment were classified as resistant and taken off of the study. We analyzed the data of 79 patients who had completed more than one IADT cycle. One patient was omitted for inconsistent treatment, seven were omitted owing to the development of metastasis and/or local progression, and one was omitted due to taking multiple medications throughout the trial, resulting in 70 patients included in the analysis (Supplementary Fig. 1). To calibrate and assess the accuracy of our model, the data was divided into training (n = 35, 27 responsive, eight resistant) and testing (n = 35, 28 responsive, seven resistant) cohorts, respectively matched for clinical response to treatment.

### Mathematical model of IADT response

We developed a mathematical model of PCaSC (S), non-stem (differentiated) PCa cells (D), and serum PSA concentration (P) (Fig. 9). PCaSCs divide with rate λ (day−1) to produce either a PCaSC and a non-stem PCa cell with probability 1−ps (asymmetric division) or two PCaSCs at probability ps (symmetric division) with negative feedback from differentiated cells16, modeled as $$\frac{S}{{S + D}}$$. Differentiated cells exclusively produce PSA at rate ρ (µg/L day1), which decays at rate φ (day−1). Unlike androgen-independent PCaSCs, differentiated cells die in response to androgen removal at rate α (day−1)41. IADT on and off cycles are described with parameter Tx, where Tx = 1 when IADT is given and Tx = 0 during treatment holidays.

The coupled mathematical equations describing these interactions are shown below

$$\frac{{dS}}{{dt}} = \left( {\frac{S}{{S + D}}} \right)p_s\lambda S,\;\frac{{dD}}{{dt}} = \left( {1 - \;\frac{S}{{S + D}}p_s} \right)\lambda S - \alpha T_xD,\;\frac{{dP}}{{dt}} = \rho D - \varphi P.$$
(1)

### Mathematical model training and validation

Cell cycle duration is assumed to align with the evolved circadian rhythm42. Matsu-Ura et al. demonstrated that intercellular coupling helps stem cells to synchronize their cell divisions with local circadian pacemakers in secretory Paneth cells43. Thus, we set uninhibited PCaSCs to divide once per day at λ = ln(2). Sensitivity analysis44 was done to assess the variation in the model output in response to small perturbations in the model parameters. This was computed by solving the sensitivity matrix $$\chi = \frac{{\partial P}}{{\partial \theta }}$$, where P is PSA and $$\theta = \left( {p_s,\alpha ,\;\lambda ,\varphi ,\;\rho } \right)$$. Cell division rate λ was the least sensitive parameter, whereas ρ was the most sensitive (Supplementary Figure 3A–B). Correlation analysis45 was used to determine parameter identifiability. Using the covariance matrix defined as $$C = \left( {\chi ^T\chi } \right)^{ - 1}$$, the correlation coefficients were computed as $$c_{\mathrm{ij}} = \frac{{C_{{\mathrm{ij}}}}}{{\sqrt {C_{{\mathrm{ii}}}C_{{\mathrm{jj}}}} }}.$$ If a parameter pair shows a strong correlation, that is if $$\left| {c_{{\mathrm{ij}}}} \right| \, > \, \xi$$, then two parameters are correlated. In this case, we chose a correlation threshold of $$\xi = 0.95$$. Supplementary Figure 3C shows that λ is highly correlated with ρ and as the least sensitive parameter, we can confidently set λ to be patient uniform at its nominal value of ln(2) and optimize the four remaining parameters (Supplementary Figure 3D).

We used particle swarming optimization (PSO)46 to identify patient-specific model parameters that minimized the least squares error between model simulation and patient data. The parameter distributions obtained by calibrating the model to the data using these four parameters are shown in Supplementary Figure 4. Comparing the coefficients of variation between each parameter showed that φ may not be required to be patient-specific and can be assumed uniform for all patients. Allowing the optimizer to find the uniform value of φ and the remaining three patient-specific parameters produced accurate fits to the data (not shown). In addition, the coefficient of variation of ρ revealed that this parameter could also be uniform between patients. Therefore, φ and ρ were set as uniform, whereas ps and α were patient-specific. PSO was used to identify population uniform and patient-specific model parameters in the training cohort. The trained mathematical model was assessed for accuracy in the validation cohort. The learned population uniform parameters were kept constant for all patients, and PSO was performed to find appropriate values for ps and α to produce accurate data fits. Resulting PCaSC proportion dynamics are shown in Supplementary Figure 5.

In order to predict the evolution of resistance, we started by fitting the model to each cycle of the training cohort data individually. That is, finding the optimal values of ps and α, while allowing φ and ρ to remain fixed at the values previously found, to fit one cycle of data at a time (Fig. 2a). We then measured the relative change in ps between cycles and used this to generate cumulative probability distributions as shown in Fig. 2b. Sampling from the 95% confidence interval around the exponential curve relating ps to α in cycle i+1 (Fig. 2c), we found a corresponding $$\alpha _{i + 1}$$. This $$p_{s,i + 1}$$ and $$\alpha _{i + 1}$$ were used to predict the response in cycle i+1. In line with the trial by Bruchovsky et al.22, resistance was defined as PSA increasing during treatment and/or a PSA level above 4 µg/L at both 24 and 32 weeks after the start of a cycle of treatment. Simulations that satisfied either of these conditions were classified as resistant. Thus, for each cycle prediction we obtained a probability of resistance (P(Ω) = proportion of resistance predictions). If the resulting P(Ω) was greater than the given threshold (obtained from the training cohort, Fig. 2d), then the prediction was classified as resistant. Otherwise, it was classified as responsive. The accuracy was computed as the proportion of predictions that were correctly classified.

### Modeling concurrent docetaxel

Unlike ADT, docetaxel can induce cell death in both PCaSCs and non-stem cells, though to a lesser degree in PCaSCs compared with non-stem cells47. To model this, we extended the current model to include death of each cell type at rates βS (day−1) and βD (day−1). That is,

$$\frac{{dS}}{{dt}} = \left( {\frac{S}{{S + D}}} \right)p_s\lambda S - \beta _ST_{xD}S,$$
$$\frac{{dD}}{{dt}} = \left( {1 - \frac{S}{{S + D}}p_s} \right)\lambda S - \alpha T_xD - \beta _DT_{xD}S,$$
(2)

where TxD = 1 when docetaxel is on and TxD = 0 otherwise. Each cycle of docetaxel was simulated as a single dose on day n (TxD = 1) followed by 3 weeks without docetaxel (TxD = 0). The parameters βS = 0.0027 and βD = 0.008 were chosen such that approximately three times more non-stem cells died than PCaSCs47. The data that support the findings of this study are available from the corresponding author upon reasonable request.

### Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

## Data availability

The clinical data used to conduct this study are available in a public repository at http://www.nicholasbruchovsky.com/clinicalResearch.html. A reporting summary for this article is available as a Supplementary Information file.

## References

1. 1.

Siegel, R. L., Miller, K. D. & Jemal, A. Cancer statistics, 2019. CA Cancer J. Clin. 69, 7–34 (2019).

2. 2.

Fizazi K., et al. LATITUDE: A phase III, double-blind, randomized trial of androgen deprivation therapy with abiraterone acetate plus prednisone or placebos in newly diagnosed high-risk metastatic hormone-naive prostate cancer. J Clin Oncol 35, LBA3-LBA3 (2017).

3. 3.

Zhang J., Cunningham J. J., Brown J. S., Gatenby R. A. Integrating evolutionary dynamics into treatment of metastatic castrate-resistant prostate cancer. Nat. Commun. 8, 1816 (2017).

4. 4.

Wargo, A. R., Huijben, S., de Roode, J. C., Shepherd, J. & Read, A. F. Competitive release and facilitation of drug-resistant parasites after therapeutic chemotherapy in a rodent malaria model. Proc. Natl. Acad. Sci. USA 104, 19914–19919 (2007).

5. 5.

Crook, J. M. et al. Intermittent androgen suppression for rising PSA level after radiotherapy. N. Engl. J. Med 367, 895–903 (2012).

6. 6.

Baez, J. & Kuang, Y. Mathematical models of androgen resistance in prostate cancer patients under intermittent androgen suppression therapy. Appl. Sci. 6, 352 (2016).

7. 7.

Cunningham J. J., Brown J. S., Gatenby R. A., Stankova K. Optimal control to develop therapeutic strategies for metastatic castrate resistant prostate cancer. J. Theor. Biol. 459, 67–78 (2018).

8. 8.

Hirata Y., et al. Intermittent androgen suppression: estimating parameters for individual patients based on initial PSA data in response to androgen deprivation therapy. PLoS ONE 10, e0130372 (2015).

9. 9.

Hirata Y., Morino K., Akakura K., Higano C. S., Aihara K. Personalizing androgen suppression for prostate cancer using mathematical modeling. Sci. Rep. 8, 2673 (2018).

10. 10.

Hirata, Y., Bruchovsky, N. & Aihara, K. Development of a mathematical model that predicts the outcome of hormone therapy for prostate cancer. J. Theor. Biol. 264, 517–527 (2010).

11. 11.

Ideta, A. M., Tanaka, G., Takeuchi, T. & Aihara, Z. A mathematical model of intermittent androgen suppression for prostate cancer. J. Nonlinear Sci. 18, 593–614 (2008).

12. 12.

Morken, J. D., Packer, A., Everett, R. A., Nagy, J. D. & Kuang, Y. Mechanisms of resistance to intermittent androgen deprivation in patients with prostate cancer identified by a novel computational method. Cancer Res. 74, 3673–3683 (2014).

13. 13.

Portz T., Kuang, Y., Nagy, J. A clinical data validated mathematical model of prostate cancer growth under intermittent androgen suppression therapy AIP Advances 2, 0110022 (2012).

14. 14.

Elishmereni, M. et al. Predicting time to castration resistance in hormone sensitive prostate cancer by a personalization algorithm based on a mechanistic model integrating patient data. Prostate 76, 48–57 (2016).

15. 15.

Collins, A. T. & Maitland, N. J. Prostate cancer stem cells. Eur. J. Cancer 42, 1213–1218 (2006).

16. 16.

Enderling, H. Cancer stem cells: small subpopulation or evolving fraction? Integr. Biol. 7, 14–23 (2015).

17. 17.

Maitland, N. J. & Collins, A. T. Prostate cancer stem cells: a new target for therapy. J. Clin. Oncol. 26, 2862–2870 (2008).

18. 18.

Isaacs, J. T. & Coffey, D. S. Etiology and disease process of benign prostatic hyperplasia. Prostate Suppl. 2, 33–50 (1989).

19. 19.

Collins, A. T., Berry, P. A., Hyde, C., Stower, M. J. & Maitland, N. J. Prospective identification of tumorigenic prostate cancer stem cells. Cancer Res 65, 10946–10951 (2005).

20. 20.

Bruchovsky, N., Rennie, P. S., Coldman, A. J., Goldenberg, S. L., To M & Lawson, D. Effects of androgen withdrawal on the stem cell composition of the Shionogi carcinoma. Cancer Res. 50, 2275–2282 (1990).

21. 21.

Lee, S. O. et al. New therapy targeting differential androgen receptor signaling in prostate cancer stem/progenitor vs. non-stem/progenitor cells. J. Mol. Cell Biol. 5, 14–26 (2013).

22. 22.

Bruchovsky, N. et al. Final results of the Canadian prospective phase II trial of intermittent androgen suppression for men in biochemical recurrence after radiotherapy for locally advanced prostate cancer: clinical parameters. Cancer 107, 389–395 (2006).

23. 23.

Baxter W. C., Stanley J. S., Zhang Q., Smith W. D. Developing artificial neural network models of water treatment processes: a guide for utilities. J. Environ. Eng. Sci. 1, 201–211 (2002).

24. 24.

Johnson, C. H. Circadian clocks and cell division: what’s the pacemaker? Cell Cycle 9, 3864–3873 (2010).

25. 25.

Fawcett T. An introduction to ROC analysis. Pattern Recognit. Lett. 27, 861–874 (2005).

26. 26.

Hussain, M. et al. Intermittent versus continuous androgen deprivation in prostate cancer. N. Engl. J. Med. 368, 1314–1325 (2013).

27. 27.

Chi, K. N. et al. Apalutamide for metastatic, castration-sensitive prostate cancer. N. Engl. J. Med. 381, 13–24 (2019).

28. 28.

Hoyle, A. P. et al. Abiraterone in “high-“ and “low-risk” metastatic hormone-sensitive prostate cancer. Eur. Urol. 76, 719–728 (2019).

29. 29.

James, N. D. et al. Addition of docetaxel, zoledronic acid, or both to first-line long-term hormone therapy in prostate cancer (STAMPEDE): survival results from an adaptive, multiarm, multistage, platform randomised controlled trial. Lancet 387, 1163–1177 (2016).

30. 30.

Kyriakopoulos, C. E. et al. Chemohormonal therapy in metastatic hormone-sensitive prostate cancer: long-term survival analysis of the randomized phase III E3805 CHAARTED trial. J. Clin. Oncol. 36, 1080–1087 (2018).

31. 31.

Vale, C. L. et al. Addition of docetaxel or bisphosphonates to standard of care in men with localised or metastatic, hormone-sensitive prostate cancer: a systematic review and meta-analyses of aggregate data. Lancet Oncol. 17, 243–256 (2016).

32. 32.

Singh, S. K. et al. Identification of a cancer stem cell in human brain tumors. Cancer Res. 63, 5821–5828 (2003).

33. 33.

Boerrigter E., Groen L. N., Van Erp N. P., Verhaegh G. W., Schalken J. A. Clinical utility of emerging biomarkers in prostate cancer liquid biopsies. Expert. Rev. Mol. Diagn. 20, 219–230 (2020).

34. 34.

Stankova, K., Brown, J. S., Dalton, W. S. & Gatenby, R. A. Optimizing cancer treatment using game theory: a review. JAMA Oncol. 5, 96–103 (2019).

35. 35.

Gravis, G. et al. Androgen deprivation therapy (ADT) plus docetaxel versus ADT alone in metastatic non castrate prostate cancer: impact of metastatic burden and long-term survival analysis of the randomized phase 3 GETUG-AFU15 trial. Eur. Urol. 70, 256–262 (2016).

36. 36.

Brady R., Enderling H. Mathematical models of cancer: when to predict novel therapies, and when not to. Bull. Math Biol. 81, 3722–3731 (2019).

37. 37.

Lee, D. K. et al. Progression of prostate cancer despite an extremely low serum level of prostate-specific antigen. Korean J. Urol. 51, 358–361 (2010).

38. 38.

Thompson, I. M. et al. Prevalence of prostate cancer among men with a prostate-specific antigen level < or =4.0 ng per milliliter. N. Engl. J. Med. 350, 2239–2246 (2004).

39. 39.

Danila, D. C. et al. Circulating tumor cell number and prognosis in progressive castration-resistant prostate cancer. Clin. Cancer Res 13, 7053–7058 (2007).

40. 40.

Aragon-Ching, J. B. et al. Circulating tumor cells in biochemical recurrence of prostate cancer. Clin. Genitourin. Cancer 13, e341–e345 (2015).

41. 41.

Tu, S. M. & Lin, S. H. Prostate cancer stem cells. Clin. Genitourin. Cancer 10, 69–76 (2012).

42. 42.

Glass, L. Synchronization and rhythmic processes in physiology. Nature 410, 277–284 (2001).

43. 43.

Matsu-Ura, T. et al. Intercellular coupling of the cell cycle and circadian clock in adult stem cell culture. Mol. Cell 64, 900–912 (2016).

44. 44.

Banks H. T., Tran H. T. Mathematical and Experimental Modeling of Physical and Biological Processes. Chapman and Hall (2009).

45. 45.

Olufsen, M. S. & Ottesen, J. T. A practical approach to parameter estimation applied to model predicting heart rate regulation. J. Math. Biol. 67, 39–68 (2013).

46. 46.

Poli, R., Kennedy, J. & Blackwell, T. Particle swarm optimization. Swarm Intell. 1, 33 (2007).

47. 47.

Jaworska D., Szliszka E. Targeting apoptotic activity against prostate cancer stem cells. Int. J. Mol. Sci. 18, pii: E1648 (2017).

## Acknowledgements

We thank participants of the clinical trial and Dr. Bruchovsky for sharing the data. We also thank our patient advocate Mr. Robert Butler for fruitful discussions. This work was supported by NIH/NCI 1R21CA234787-01A1 “Predicting patient-specific responses to personalize ADT for prostate cancer”, and in part by NIH/NCI U54CA143970-05 (Physical Science Oncology Network) “Cancer as a complex adaptive system”, the Ocala Royal Dames for Cancer Research, Inc., and The JAYNE KOSKINAS TED GIOVANIS FOUNDATION FOR HEALTH AND POLICY, a Maryland private foundation dedicated to effecting change in health care for the public good. The opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and not necessarily those of the JAYNE KOSKINAS TED GIOVANIS FOUNDATION FOR HEALTH AND POLICY, its directors, officers, or staff.

## Author information

Authors

### Contributions

R.B., A.Z.W., T.Z., J.D.N., and H.E. conceptualized the study. R.B, J.D.N, T.A.G., and H.E. performed the modeling and statistical analyses. R.B., J.D.N., T.A.G., T.Z., A.Z.W., J.Z., R.A.G., and H.E. wrote the manuscript.

### Corresponding authors

Correspondence to Robert A. Gatenby or Heiko Enderling.

## Ethics declarations

### Competing interests

Provisional patent application entitled “Methods for PCa intermittent adaptive therapy”. Applicants/inventors: Renee Brady-Nicholls, Heiko Enderling; application number: 62/944.804; status: provisionally filed. The patent covers methods related to using the mathematical model to adjust an individual patient’s treatment administration (both timing and alternative treatment options), thereby increasing TTP. T.Z.: Research funding: Acerta, Novartis, Merrimack, Abbvie/StemCentrx, Merck, Regeneron, Mirati Therapeutics, Janssen, Astra Zeneca, Pfizer, OmniSeq, Personal Genome Diagnostics, Seattle Genetics; Speakers Bureau: Genentech/Roche, Exelixis, Sanofi-Aventis, Genomic Health; Advisory Board: Genentech/Roche, Merck, Exelixis, Sanofi-Aaventis, Janssen, Astra Zeneca, Pfizer, Amgen, BMS, Pharmacyclics, Seattle Genetics, Bayer; Consultant: Bayer, Astra Zeneca, Foundation Medicine; Employee: Capio Biosciences, Archimmune Therapeutics (spouse); Stockholder: Capio Biosciences, Archimmune Therapeutics (spouse); A.Z.W.: Cofounder, stockholder and consultant: Capio Biosciences, Archimmune Therapeutics; J.Z.: Consultant: Dendreon, Advisory Board: AstraZeneca, Bayer, Clovis Oncology, Seattle Genetics, Speaker Bureau: Merck, Sanofi. All other authors declare no competing interests.

Peer review information Nature Communications thanks Martin Nowak and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Brady-Nicholls, R., Nagy, J.D., Gerke, T.A. et al. Prostate-specific antigen dynamics predict individual responses to intermittent androgen deprivation. Nat Commun 11, 1750 (2020). https://doi.org/10.1038/s41467-020-15424-4

• Accepted:

• Published:

• ### Adaptive Therapy for Metastatic Melanoma: Predictions from Patient Calibrated Mathematical Models

• Eunjung Kim
• , Joel S. Brown
• , Zeynep Eroglu
•  & Alexander R.A. Anderson

Cancers (2021)

• ### A Quantitative Paradigm for Decision-Making in Precision Oncology

• Dalit Engelhardt
•  & Franziska Michor

Trends in Cancer (2021)

• ### Are all models wrong?

• Heiko Enderling
•  & Olaf Wolkenhauer

Computational and Systems Oncology (2020)

• ### Circulating cell-free nucleic acids as prognostic and therapy predictive tools for metastatic castrate-resistant prostate cancer

• Navid Sobhani
• , Marianna Sirico
• , Daniele Generali
• , Fabrizio Zanconati
•  & Bruna Scaggiante

World Journal of Clinical Oncology (2020)

• ### Interrogating the microenvironmental landscape of tumors with computational image analysis approaches

• Nektarios A. Valous
• , Rodrigo Rojas Moraleda
• , Dirk Jäger
• , Inka Zörnig
•  & Niels Halama

Seminars in Immunology (2020)