Noise-induced bistability in the fate of cancer phenotypic quasispecies: a bit-strings approach

Tumor cell populations are highly heterogeneous. Such heterogeneity, both at genotypic and phenotypic levels, is a key feature during tumorigenesis. How to investigate the impact of this heterogeneity in the dynamics of tumors cells becomes an important issue. Here we explore a stochastic model describing the competition dynamics between a pool of heterogeneous cancer cells with distinct phenotypes and healthy cells. This model is used to explore the role of demographic fluctuations on the transitions involving tumor clearance. Our results show that for large population sizes, when demographic fluctuations are negligible, there exists a sharp transition responsible for tumor cells extinction at increasing tumor cells’ mutation rates. This result is consistent with a mean field model developed for the same system. The mean field model reveals only monostability scenarios, in which either the dominance of the tumor cells or the dominance of the healthy cells is found. Interestingly, the stochastic model shows that for small population sizes the monostability behavior disappears, involving the presence of noise-induced bistability. The impact of the initial populations of cells in the fate of the cell populations is investigated, as well as the transient times towards the healthy and the cancer states.

anomalies in both tumor-suppressor genes and proto-oncogenes (1st bit or compartment), in genes responsible for genomic integrity (2nd bit) and in house-keeping (hk) genes (3rd bit). These three compartments were defined as bits in this minimal system giving place to different tumor cell phenotypes competing with healthy cells, given by strings 000. The full system, corresponding to the so-called quasispecies-like phenotypic models (see also [3]), describes the dynamics of different cancer cell phenotypes with a quasispecies populational structure (see Fig. S1). The characteristics of this population are the same as the ones displayed in Table I for the agent-based model investigated in this article.
As done for the stochastic bit-strings model, subinidices of the population variables correspond to the integer number of the binary sequences. The model parameters are given by the replication rate of cells (r > 0); the increase of proliferation of tumor cells (δ r > 0); the rate of mutation or accumulation of genomic anomalies of tumor cells 0 < µ < 1; and the increase of genome instability of tumor cells 0 < δ µ < 1 − µ. That is, tumor cells with bit 1 in the first position of the string present increased proliferation rates, r + δ r , due to mutations or anomalies in replicationrelated genes. Genome instability in introduced with µ + δ µ when the second bit of the strings is 1.
The cells with sequences ab1, a, b ∈ {0, 1} present anomalies or mutations in hk genes and thus are not able to proliferate. Finally, ν is the length of the sequences i.e., here with ν = 3.
The fixed points of Eqs. (1)-(8) and their stability was characterized in [1]. Among the fixed points identified, two of them were responsible for the two asymptotic states behind tumor persistence and extinction. These two fixed points, labeled P * 2 and P * 3 in [1] are the ones displayed in Fig. S2. Interestingly, the asymptotic states identified with the MonteCarlo simulation model developed in this article correspond to these two equilibria (compare the projections in Ω of Figs. 1 and 2 in the main manuscript with the simplexes of Fig. S2). According to the convention adopted in this manuscript, the fixed point P * 2 corresponds to the healthy absorbing state H as , while the fixed point P * 3 is the tumor absorbing state T as . According to Fig. S2, when µ < µ c the fixed point P * 3 is globally asymptotically stable while P * 2 is unstable. Recall that µ c = δ r /(r + δ r ). We note that these two fixed points present a heteroclinic connection (see below for the definition of heteroclinic connection). At the bifurcation value µ = µ c the heteroclinic connection is replaced by a line of fixed points. After the bifurcation, when µ > µ c , the nature of the stability of these two fixed points has been reversed, P * 2 being asymptotically globally stable and P * 3 being unstable. Hence, the dynamics of the system under study under its mean field limit involves monostability when µ = µ c . This means that the stable fixed points are globally stable and neither coexistence of solutions nor different basins of attraction can be found in the phase space of Eqs. (1)-(8) [1]. As mentioned, these two equilibrium points are connected heteroclinically. The so-called heteroclinic connection can be defined as follows: Definition I.1 (Heteroclinic connection). Let x * 1 and x * 2 be equilibria of a nonlinear function f : R → R n :ẋ being the value of a trajectory starting at the point x 0 at time t.
Remark I.1. From Definition II.1 it immediately follows that the heteroclinic connection is a part

B. Fixed points and absorbing states
As mentioned above, the stability of the fixed points P * 2 and P * 3 depends on the parameters µ, r, and δ r (see [1] for further details). It is important to note that these fixed points are absorbing states, regardless of their stability. It means that once one of these two fixed points is achieved by a trajectory, the system will remain trapped in that fixed point forever. For the deterministic model, the fixed points that the orbits will reach correspond to the stable ones whenever the initial condition does not coincide with the coordinates of the fixed point. However, in the stochastic model, noise can make trajectories to reach a fixed point that in the deterministic model is unstable.
Since these are absorbing states, the system will get trapped in one of these two fixed points reached by a stochastic trajectory, remaining there forever. The absorbing nature of these two fixed points becomes clear from the structure of the population and from the dynamical processes under consideration. Two important features determine the absorbing nature of these two equilibria: healthy cells do not mutate and backward mutations are not allowed. This means that once healthy cells become extinct, no possible production of S 0 is possible from the pool of mutants (since no backward mutations are allowed). Similarly, once the entire population is formed by healthy cells, no possible production of mutants is possible since S 0 cells cannot produce mutants.  The fixed points P * 3 and P * 2 are asymptotically globally stable below and above the bifurcation value, respectively. That is, before and after the trans-heteroclinic bifurcation the system is monostable.

Cells
Phenotype State (sequence) Subindex Replication Per-bit mutation