## Abstract

An evolutionary game emerges when a subset of individuals incur costs to provide benefits to all individuals. Public goods games (PGG) cover the essence of such dilemmas in which cooperators are prone to exploitation by defectors. We model the population dynamics of a non-linear PGG and consider density-dependence on the global level, while the game occurs within local neighborhoods. At low cooperation, increases in the public good provide increasing returns. At high cooperation, increases provide diminishing returns. This mechanism leads to diverse evolutionarily stable strategies, including monomorphic and polymorphic populations, and neighborhood-size-driven state changes, resulting in hysteresis between equilibria. Stochastic or strategy-dependent variations in neighborhood sizes favor coexistence by destabilizing monomorphic states. We integrate our model with experiments of cancer cell growth and confirm that our framework describes PGG dynamics observed in cellular populations. Our findings advance the understanding of how neighborhood-size effects in PGG shape the dynamics of growing populations.

## Introduction

Nature offers many examples where individuals provide public goods that benefit self and others, which can be maintained by population assortment^{1}. In banded mongooses, an individual in the foraging line flushes insects for others^{2}. Prairie dogs maintain sight lines around their burrows that become available to others in the colony^{3}. Predator inspection by guppies^{4}, or mobbing of hawks by crows^{5} provide safety to self and others regardless of participation. Via associational refuges, plants defended by spines or toxins may dissuade herbivores from feeding on any plants in a neighborhood^{6,7}. Microbes provide public goods by secreting defensive chemicals^{8,9}. Yeast synthesize and spill essential nutrients, such as key amino acids into their surroundings^{10}. Yeast consuming and synthesizing amino acids for themselves and others were also observed to coevolve in the form of a different social dilemma, the snowdrift game^{11}, in which stable coexistence can exist with or without assortment^{12}. Cancer cells, too, can engage in public goods games^{13,14,15}. As ecological engineers cancer cells can promote favorable environments^{16,17}, suggesting that they evolve cooperation^{18,19}, or form mutualistic relationships with other cells^{20}. It benefits a cancer cell to recruit blood vessels, signal normal cells and defend against the immune system, which benefit neighbors too. Such interactions can strongly influence the eco-evolutionary dynamics between cooperators (public good producers) and defectors (free-riders).

Benefits that are produced and shared are often only available within a finite neighborhood^{21,22}. These finite neighborhoods create a form of assortative interaction: an individual always experiences a slightly higher frequency of its own strategy within a neighborhood than present in the entire population. This tilt is because the individual contributes to the public good if it is a cooperator, and detracts from the collective public good if it is a defector. Furthermore, cooperators may interact more locally than defectors. We are interested in how strategy-specific neighborhood sizes influence the population dynamics and the evolutionarily stable strategy (ESS) of public goods games. In addition, public goods game studies often assume a linear relationship between benefits and the number of cooperators^{23,24}. Yet this collective benefit may be nonlinear^{25,26,27}. If one alarm call is sufficient to alert the colony of prairie dogs to a predator, then there are rapid diminishing returns to having several callers. If a collective defense by microbes against a predator, or by cancer cells against the immune system, requires a threshold effect, then there may be increasing returns where two producers more than double the collective benefit.

Here, we considered a nonlinear relationship between number of cooperators and the resulting collective benefit. Using this nonlinear relationship we integrated density-dependent population growth and frequency-dependent local interactions into a single modeling framework. We found that the resulting nonlinear intrinsic growth functions allow a diverse space of evolutionarily stable strategies. In addition to parameter regions in which cooperators can evade the tragedy of the commons, neighborhood-size-driven state switching occurred, which resulted in hysteresis between the monomorphic and polymorphic strategies if the neighborhood size was varied. We also found that stochastic fluctuations in the neighborhood size favored polymorphic strategies. To incorporate empirical evidence, we revisited recent experimental findings of non-linearities in public goods games played by IGF-II producer and non-producer pancreatic cancer cells in vitro^{14}, which suggests that our framework can be used to identify critical neighborhood sizes. We also tested the sensitivity of our model by considering its stochastic analog, in which one does not have to chose the neighborhood size exogenously, and we touched on the emerging question of how strategy-specific neighborhood sizes alter the available ESS. Whether higher multi-cellular organisms with complex social structure, yeast, or cancer cells, populations exhibit ecological dynamics (changes in population size), and evolutionary dynamics (changes in relative abundance). The framework we develop here sheds new light on the conditions needed to maintain cooperative traits that provide public good to the environment.

## Results

### Nonlinear growth rate and neighborhood size shape selection

To investigate the role of nonlinear growth and neighborhood size on the public good game, we considered a population that consists of two sub-populations. Producer cells, *C* (cooperators) produce a public good at a cost and consume it to gain a benefit. Free-rider cells, *D* (defectors), consume available public good, but neither produce nor incur a cost (Fig. 1a). We modeled proliferation and death events as an individual-based stochastic process (see Equation (1), Methods) and in its deterministic limit. The intrinsic growth rates of *C* and *D* are nonlinear functions of the public good shared among a neighborhood of size *n* and the frequency of *C*, which we implemented using two parameters, *β* (frequency-dependent effects) and *σ* (frequency-independent effects). These two parameters modulate the size of the nonlinearities in the growth rates (Fig. 1b) as a function of the frequency (fraction) of *C*. In our model, producers carry a cost *κ* that effectively reduces their intrinsic growth rate irrespective of the environment. Yet, at relatively low neighborhood sizes, producers take advantage of their own public good, as they experience a benefit-to-self. In addition, we considered ecological feed-back in the form of a carrying capacity parameter *K*. Using this model, we observed different ESSs emerge by varying parameters (Fig. 1c, d). Despite a low starting cooperator population, coexistence was still obtained in a few simulations via stochastic branching, which corresponded to passing the threshold needed to enter the long-term coexistence region (Fig. 1d). In the deterministic limit of the stochastic process, the system approaches a set of coupled logistic ordinary differential equations, which describes the dynamics of the two sub-populations, (see Equation (2), Methods).

The neighborhood size *n* also plays a substantial role in determining the number and stability of equilibria, as it shifts the phase boundaries that separate the stable strategies determined by *β* and *σ*. Increasing the neighborhood size favors defectors and decreasing it favors cooperation (Fig. 1e, f). Trajectories of the population dynamics in the deterministic limit unveil the existence of monomorphic and polymorphic strategies, and how these change by modulating the relative values of *β* and *σ* (Fig. 1g–j). These modulations give rise to different hallmark examples of frequency-dependent selection.

In Fig. 1e, f, we give explicit examples of the possible phase diagrams that indicate whether polymorphic equilibria exist and are stable. For fixed neighborhood size, baseline growth rates and cost, these phase diagrams are uniquely determined by the two payoff parameters *β* and *σ*. Increasing the neighborhood size reduces the chances of observing a stable all-*C*. On the other hand, increasing the neighborhood size can increase the parameter range for which we can expect bi-stable evolutionary dynamics (unstable polymorphism). Changes in neighborhood size can critically alter the range of benefit parameters that allow for a stable equilibrium where cooperators and defectors coexist. For example, a stable polymorphic equilibrium can be expected if *β* is an order of magnitude larger than the parameter *σ*, in which case the net growth rate advantage of cooperators is either non-monotonic, or decreases as the abundance of cooperators increases.

### Public goods form determines stability and number of equilibria

If the cost of cooperation is exorbitantly high, the all-*D* state is generally the only ESS. Since defectors do not carry a cost, they always fare better in the absence of any measurable public good effect. The stability of the all-*D* equilibrium is independent of the functional form of the growth rates and depends solely on whether defectors have a higher overall intrinsic growth rate than cooperators. In contrast, the stability of the all-*C* state depends critically on the functional form of the intrinsic growth rates. If *r*_{C} > *r*_{D} for all frequencies of cooperators in the population, then all-*C* is the ESS. A polymorphic ESS with a mix of *C* and *D* becomes possible when the ratio of respective growth rates is equal to the ratio of death rates *r*_{C}/*r*_{D} = *δ*_{C}/*δ*_{D} for some frequency of cooperators. We label the fraction of cooperators in an equilibrium polymorphic state *y*^{*}, and the total population size in this state *Y*^{*}. When all baseline growth and death rates are the same (*α*_{C} = *α*_{D} and *δ*_{C} = *δ*_{D}), *Y*^{*} is largest in the all-*C* state. A full linear stability analysis is presented in Supplementary Methods 4.

Next, let us analyze the number of possible polymorphic equilibria. We show that for the “almost identical” public good functions (*r*_{C} = *Ar*_{D}, for some constant *A* > 0) there can only be one internal equilibrium. The same holds for public good functions that are “always better” in terms of differential returns from the public good, e.g., a cooperator benefits more when another cooperator is added to the neighborhood, \(r_C^\prime (y) > r_D^\prime (y)\). Our choice of sigmoidal public good functions Equation (4), can have a maximum of two polymorphic equilibria. For details, see Supplementary Methods 4.2.

### Modulation of neighborhood-size drives state-switching

One internal equilibrium is typically unstable and the other is stable. The stability properties of internal equilibria critically depend on the neighborhood size *n*. This means that saddle-node and transcritical bifurcations^{28} are possible in which the neighborhood size *n* acts as the bifurcation parameter (Fig. 2). The parameter *β* can also control such bifurcations.

The growth rates can be non-monotonic functions of the fraction of *C* in the population. Then, an unstable and a stable polymorphic state can coalesce and annihilate each other at a critical value of neighborhood size. This is known as a saddle-node bifurcation and occurs without affecting the stability of the monomorphic states. Saddle-node bifurcations tend to co-occur with rather sharp transitions between no benefit (due to low numbers of cooperators) to a maximal benefit. Such sharp transitions can be observed for strong frequency-dependent selection (large values of *β*). In Fig. 2a, b we show an example of *β* > *σ* leading to a non-monotonic growth rate differential, which creates a saddle-node bifurcation controlled by the neighborhood size. The stochastic dynamics in this case exhibit a large region of long-term coexistence before large neighborhood sizes favor *D* (Fig. 2c).

In contrast, for *β* < *σ*, we observe monotonically increasing growth rate differences, leading to unstable coexistence. This pattern leads to a transcritical bifurcation (Fig. 2d, e) where the resulting ESSs are alternative stable states of all-*C* or all-*D*. Correspondingly, the stochastic dynamics (Fig. 2f) shows a sharp transition from all-*C* to all-*D* with increasing *n*, with little room for long-term coexistence. In this parameter regime, for *n* = 10, we see that about 10% of the stochastic simulations lead to coexistence, which is also corroborated by the example trajectories shown in Fig. 1d.

### Model integration with in vitro cancer cell growth kinetics

The case *β* > *σ* is of particular interest as it can lead to stable and unstable internal coexistence points due to non-monotonicity in growth rate differences. Public good functions that influence population growth rates in a non-monotonic fashion have been proposed in the context of bacterial, yeast and pancreatic cancer cell monolayers, and were subsequently measured empirically in an in vitro context^{14}. However, a concise numerical and statistical interpretation of this data was not included. With our model, we can in part close this gap and recapitulate the intriguing frequency-dependent cancer cell growth patterns measured by Archetti et al.^{14} They established that IGF-II (insulin-like growth factor 2) can act as a nonlinear public good that comes at a specific cost to cooperators (IGF-II producer cells). In the context of freely available nutrients (Fetal Bovine Serum; FBS), the benefit of producing the public good declines with increasing concentration of FBS (Fig. 3a, see also Fig. 3c in reference^{14}). To estimate realistic values of *β* and *σ*, we used data from cellular in vitro competition experiments using the population growth and expansion of IGF-II producer cells (cooperators) and non-producer cells (defectors). The actual neighborhood size was not measured or estimated for these experiments. Therefore, in fitting our model to this data, we considered neighborhood size to be independent and identically distributed between *n* = 4 (nearest neighbors in 2D) and *n* = 40 cells. For each value of *n* and for each experimental setting (percent FBS in the medium), we fit our mathematical modeling framework to the cellular growth data (see Supplementary Methods 5), to estimate values for *β* and *σ*. Our estimated values of *β* ranged from 1.5 to 6 (Fig. 3b), and values of *σ* from 0 to 3 (Fig. 3c). For example, in the context of 5% FBS in the background medium, we observed median values of *β* = 3.67 and *σ* = 1.87. In our model, these parameter values would predict an unstable polymorphic equilibrium near *y*^{*} = 0.235, and a stable polymorphic equilibrium near *y*^{*} = 0.784 (Fig. 3d). These estimates depend on the distribution of values of *n*, which highlights the need in future studies to devote more effort into determining the distance over which a public good spreads. This statistical analysis shows that our model framework is capable of explaining the evolutionary dynamics of cellular games with nonlinear public goods and confirms that cancer cell lines can exhibit non-monotonic growth rates which favor local coexistence if the background nutrients are sufficiently sparse (*β* > *σ*).

### Type-dependent neighborhood sizes

Cooperators may experience a different neighborhood size, *n*_{C}, than defectors, *n*_{D}, for example through aggregation effects^{29}, or by altering their local interaction topology^{30}. In such cases, defectors could fare better from unconditional interactions with as many cooperators as possible. Thus, with mechanisms that promote increases in neighborhood size at a cost smaller than the cost of cooperation, one could see adaptation toward strategy-dependent neighborhood sizes. Thus, we examined how strategy-specific neighborhood sizes influence coexistence, the ESSs, and the eco-evolutionary dynamics of Equation (2). We assumed that cooperators interact with other individuals within their specific neighborhood of size *n*_{C}, and that defectors interact within *n*_{D}. Under these additional assumptions, we examined under which conditions a small population of cooperators can invade a resident population of defectors. The relation *n*_{C} < *n*_{D} favors coexistence, which can be seen in the different bifurcation patterns in Fig. 4. When both *C* and *D* experience the same neighborhood size, then small sizes favor all-*C* and larger neighborhood sizes favor all-*D*. For a sufficiently small *n*_{C}, cooperators can invade. For a sufficiently high *n*_{D}, defectors can invade. Hence \(n_C \ll n_D\) reduces the likelihood that all-*D* and/or all-*C* will be an ESS, and increases the likelihood of ESS coexistence. It should be noted that it could be difficult to argue in favor of differences in *n*_{C} and *n*_{D} in isotropically diffusing growth factors. However, realistic spatial environments, such as a three-dimensional solid tumor with anisotropic collagen patterns could realistically present such a scenario^{31}.

### Fluctuations in the neighborhood size influence coexistence

Here, we are interested in how feedbacks between total population size and neighborhood size influence the ESS of *C* and *D*. To address these questions we devised a compartmentalized version of the eco-evolutionary growth model of Equation (2). Instead of exogenously fixing neighborhood sizes and performing averaging over neighborhood compositions, we let the total population size influence neighborhood sizes by dividing the population into *N* compartments. The total population *Y*, can then be randomly distributed across these compartments, resulting in a multinomial distribution of neighborhood sizes with a mean of *Y*/*N*. The public good is now produced and shared locally among a stochastically sampled number of neighbors. Here, the number of compartments acts as a surrogate for the inverse of an effective neighborhood size, which is an emergent quantity. This stochastic mixing of the population was performed numerically according to a multinomial sampling process (see Supplementary Methods 6). Hence the interaction neighborhood for each strategy type was a stochastic variable determined by the number of compartments and the total population size. A schematic of the compartment-based dynamics is given in Fig. 5a. Of critical importance to the dynamics of this model is the time between the mixing steps (see Supplementary Figures 1–3).

In the compartment approach, we observed that cooperators survived above a threshold number of compartments *N*, and that cooperators would become the ESS above a critical value *N*_{crit}. Depending on the values of *β* and *σ*, coexistence of cooperators and defectors was possible. We observed the equivalent of a transcritical bifurcation (switching of stable states), controlled by an increasing number of compartments (Fig. 5b), as well as the equivalent of a saddle-node bifurcation (Fig. 5c) with stable coexistence at intermediate number of compartments. We defined the critical number of compartments such that above *N*_{crit} all-*C* would be stable. Figure 5d shows how *N*_{crit} depends on the nonlinear public goods benefit parameters. Notably, if we did not re-initialize the population when changing the number of compartments, we could observe a strong hysteresis pattern (Fig. 5e) as predicted by the transcritical bifurcation (compare to Fig. 2e).

We used a mean-field approach to estimate *N*_{crit}. Instead of using a distribution for the individuals among the *N* compartments, we let each compartment have the expected value, *Y*/*N*. We then determined the number of compartments above which all-*D* becomes unstable and is no longer an ESS. Although this approach misses some important numerical and stochastic fluctuations, it recovers the general trend of the behavior of *N*_{crit} (Fig. 5f), which shows non-monotonic behavior as a function of *β*. For small but increasing *β*, *N*_{crit} decreases monotonically, but then assumes a minimal value at intermediate selection strength. For very strong frequency-dependent selection (large *β*), *N*_{crit} increases asymptotically toward *Y*^{*}. These findings highlight the dynamical feedbacks between population dynamics, neighborhood size, and frequency-dependent effects. These feedbacks also point to how measurements of population densities and interaction distances could determine an effective neighborhood size.

## Discussion

In summary, we modeled both density and frequency-dependent effects in the context of a non-linear public goods game that is effective within a neighborhood. There is a critical role for neighborhood size, which describes the group of individuals that are close enough together to benefit from the public good. Our results demonstrate the following. First, smaller neighborhood sizes favor cooperators and increase the likelihood that all-*C* is an ESS. Second, a linear relationship between the frequency of *C* in the neighborhood and the overall value of the public good permits two global ESSs, which are either globally all-*D* or globally all-*C*, without the possibility of coexistence or alternate stable states. Third, a nonlinear, sigmoidal relationship between the public good and the frequency of *C* allows for multiple stable ESSs. Such nonlinearity was previously measured in cancer cell dynamics^{14}, and we made use of their experimental findings to validate our modeling framework. Fourth, under alternate stable states we observe saddle-node and transcritical bifurcations and hysteresis. Last, feedback of population size on neighborhood size and/or strategy-dependent neighborhood sizes can increase the likelihood of coexistence between *C* and *D*. These results from the deterministic approach are corroborated by results obtained from simulations of a corresponding individual-based stochastic model.

Individual-based stochastic models have the benefit that they allow us to estimate extinction probabilities and distributions of the time to extinction. Notably, such systems have often been examined in the fixed population size scenario^{32,33,34}. Recent expansions have focused on emergent phenomena in populations of fluctuating size or coevolutionary feedback^{35,36}. These approaches also show that multi-level^{37} or multi-player^{38} population games deserve more attention in the context of fluctuation-driven phenomena. These problems are of interest since population size fluctuations are related to the maintenance of cooperation, extinction delay, or other counterintuitive selection effects^{24,39,40,41}, especially in the context of neighborhood-size dependent access to shared resources.

The number of individuals comprising a neighborhood may be influenced by the total size of the population^{42}. In this way, population dynamics can feedback in to the neighborhood size. If neighborhood size increases with population density, then density feedbacks can result in the coexistence of cooperators and defectors. As a consequence, an all-*C* world promotes a higher population size than an all-*D*, as it also allows a higher carrying capacity^{24}. With a feedback on neighborhood size, an all-*C* population promotes a higher neighborhood size which, in turn, favors the invasion of defectors. Similarly, an all-*D* population promotes smaller neighborhood sizes that facilitate the ability of cooperators to invade. Density feedbacks have been shown to influence the ESS in games of spite^{43} and producer-scrounger games^{44}. However, if the public good is an essential nutrient secreted by a protist or cancer cell, then it may only provide resources for a fixed number of others, regardless of distance from the producer. Hence, there may be little feedback at all. In many cellular interactions, we can expect that a fixed number of nearby neighbors become the beneficiaries of a producer.

Linear public good games do not allow for saddle-node bifurcations, and consequently all-*C* or all-*D* are globally stable ESSs, unless there are density-feedbacks as discussed above. We here show that a nonlinear relationship between the public good and the frequency of producers permits bifurcations and hysteresis between alternative equilibria. We considered a sigmoidal relationship between benefit and cooperator frequency, which can favor coexistence because increasing returns at low fractions of cooperators act in their favor, whereas diminishing returns toward high fractions of cooperators favors defectors. We have observed that there are often maximally two interior equilibria, one will always be unstable and the other representing the stable coexistence of cooperators and defectors. With a different nonlinear relationship, such as a Michaelis–Menten (Hill) function^{15} (see also Supplementary Methods 4.2), there could be up to five interior equilibria.

The multi-player snowdrift game provides another form of public goods game in which players receive a fixed benefit that is not influenced by the frequency of cooperators. Rather, it is the cost that is shared among the cooperators. Hence the cost of cooperating declines with the frequency of cooperators. In our game, it is the opposite. Cooperators bear a fixed cost and it is the benefit that is distributed based on the frequency of cooperators. A similar analysis could be applied to the snowdrift game as we have done here.

Whether it be cancer or other ecological public good games in nature, our results suggest new ways to manipulate the parameters of public good population games. Cancer cells may engage in public goods games in a manner that make the disease more devastating and harder to treat^{16,17}. Cells produce signals that promote vasculature, or signal other cells, which can produce vasculature (e.g., vascular endothelial growth factor, VEGF). Then, the recruitment of blood vessels provides a benefit to the producer but also to any cells within some neighborhood of these new vessels. In such systems, however, it is unclear whether there is a significant cost to cooperation. It may be possible to create therapies that incentivize the population game in a manner that shifts the equilibrium towards defectors, for instance by decreasing the benefit of the public good. This approach may have demonstrable value in bicarbonate therapy, which is a means of neutralizing the low pH created by glycolytic cancer cells^{45}. Increasing neighborhood size provides an intriguing, yet hitherto unappreciated therapeutic opportunity. Perhaps by increasing the diffusion rate of molecules (including VEGF) within the tumor, the provider cells would experience less of a self-benefit and defectors would perceive a higher benefit. Such manipulations could then achieve control, e.g. of pests or tumors, or promote polymorphism and diversity, e.g. of endangered or valuable species, or maintain cooperative sub-populations.

## Methods

We consider the dynamics of a population that consists of two sub-populations. Producer cells, *C* (cooperators) produce a public good at a cost and consume it to gain a benefit. Free-rider cells, *D* (defectors), consume available public good, but do not produce nor incur a cost. We assume that the total population can grow towards a maximal load, or carrying capacity, which introduces a global density-dependent effect. Proliferation and death events are described by the following process

where the birth rates *r*_{C,D} *E*_{K} incorporate global density dependence via the factor *E*_{K} = 1 − (*C* + *D*)/*K* that is multiplied to the growth rates *r*_{C,D} (for details, see Supplementary Equation (S2) in Supplementary Methods 1.1). The death rates *δ*_{C,D} are assumed to be constant. These reactions describe a stochastic individual-based process for which it is possible to derive a mean field dynamical system (see Supplementary Methods 1.2). In the deterministic limit, the densities of cooperators *C* and defectors *D* are given by *x*_{C} and *x*_{D}, and their dynamics are described by the system

We are interested in cases where the intrinsic growth rates depend on the frequency of cooperators *y* = *x*_{C}/(*x*_{C} + *x*_{D}). There could also be coevolution at the level of *K*^{35,46}, which in turn would influence the dynamics differently and is beyond the scope of this manuscript (see comment in the Supplementary Methods 3.2). Here we seek to quantify the impact of available public good on the growth rates, knowing that production can incur a cost on *C* (Fig. 1a).

To calculate how *C* and *D* benefit from public good consider that the public good is a local commodity consumed by all within a neighborhood of *n* cells, for example a growth factor produced by *C*. Thus, we have to account for all possibilities to select up to *n* cooperators. If the fraction of producer cells is *y*, we can obtain the following expected numbers of producer cells of any *C* or *D*

These values result from hypergeometric sampling *n* − 1 times with a probability *y* to pick *C*, assuming that the population is sufficiently large. The expected values *N*_{C,D} can then be used to calculate expected growth rates. As we discuss in the Supplementary Methods 1.4 and 2, the use of the hypergeometric distribution to generate a neighborhood and calculate growth rates leads to negligible changes in the dynamics for sufficiently large populations. In the difference between *N*_{C} and *N*_{D} we see that cooperators might experience an increased benefit, especially if *n* is small, as they are able to benefit from their own public good.

The benefit to each member of a neighborhood may increase nonlinearly with the frequency of cooperators^{14,25,47}. One could observe steep increases of fitness at low frequencies of cooperators, followed by a saturation effect in fitness once the fraction of cooperators crosses a threshold. Such a sigmoidal, or s-shaped fitness curve can be described using a parameter *β* that modulates frequency-dependence, and a parameter *σ* that modulates frequency-independence (background). The inflection point where benefit increase is maximal is then given by *σ*/*β*. For *y* < *σ*/*β*, additional cooperators create synergies. For *y* > *σ*/*β*, we see diminishing returns. Including the cost for cooperation, *κ*, we can now write the frequency-dependent intrinsic growth rates as

For a sketch of these sigmoidal growth rates, see Fig. 1b. Without any frequency-dependent benefit, *β* = 0, we obtain the intrinsic growth rates *r*_{C} = *α*_{C} − *κ* and *r*_{D} = *α*_{D}. For a very small *β* one can linearize the intrinsic growth rates *r*_{D} = *α*_{C}(1 + *N*_{C}(*β*)) − *κ*, and *r*_{D} = *α*_{D}(1 + *N*_{D}(*β*)). Linear relationships between benefits and the amount of the public good have frequently been used recently^{24,48,49,50}. For mathematical convenience we here chosen to implement intrinsic growth rates as a function of the expected number of cooperators. Instead, one can simulate the full stochastic dynamics in relation to the mean field model, or chose to average over group compositions at a different point. These changes are mostly quantitative, impacting only the location of bifurcations in the phase diagrams, the morphology remains the same.

Our mean field approach averages group compositions before calculating the non-linear benefits to the growth rates, which reflects the fully stochastic dynamics rather well (shown in Fig. 1c, d). In addition, one can ask how the complex structure of equilibria of the deterministic system, as modulated by the parameters *β* and *σ* (Fig. 1e–j), is altered by changing the assumptions that led to *N*_{C,D} and *r*_{C,D}. In Supplementary Figure 4, we show that these changes can be expected to be minimal.

We also explored fluctuations in neighborhood size. Implementing fluctuations in neighborhood size removes undesired effects when calculating expected growth rates: instead of considering *n* as an exogenous parameter, we defined a fixed number *N* of compartments among which all cells are randomly distributed to form neighborhoods. By creating compartments (patches), these local patches become the neighborhoods observed by the cells within, averaging is not applied. In this way, neighborhood sizes vary according to a multinomial distribution with a mean of (*x*_{C} + *x*_{D})/*N*. To simulate this, we ran a dynamic loop of two steps. First, we evaluated the outcome of a public goods game and logistic growth within each compartment for a fixed amount of time Δ*t*, according to Equation (2) with *n*_{i} and *y*_{i} in compartment *i*. Second, we modeled neighborhood assembly by pooling individuals from all compartments and re-distributing them randomly among compartments.

We used the approaches described here to examine how the nonlinear growth rates and neighborhood size changes determine the number and stability properties of equilibria (ESSs) of the deterministic system, corroborated by results of the individual-based approach and the compartment approach. All mathematical and computational methods and proofs are detailed further in the Supplementary Information.

### Reporting Summary

Further information on experimental design is available in the Nature Research Reporting Summary linked to this Article.

## Data availability

The simulation data and code used in this study are available in the nonlinearPGG repository at https://github.com/MathOnco/nonlinearPGG.

## Additional information

**Publisher’s note:** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## References

- 1.
Killingback, T., Bieri, J. & Flatt, T. Evolution in group-structured populations can resolve the tragedy of the commons.

*Proc. R. Soc. B.***273**, 1477–1481 (2006). - 2.
Rood, J. P. Population dynamics and food habits of the banded mongoose.

*Afr. J. Ecol.***13**, 89–111 (1975). - 3.
Davidson, A. D. & Lightfoot, D. C. Keystone rodent interactions: prairie dogs and kangaroo rats structure the biotic composition of a desertified grassland.

*Ecography***29**, 755–765 (2006). - 4.
Dugatkin, L. A. Tendency to inspect predators predicts mortality risk in the guppy (

*Poecilia reticulata*).*Behav. Ecol.***3**, 124–127 (1992). - 5.
Whitehouse, M. E. & Lubin, Y. The functions of societies and the evolution of group living: spider societies as a test case.

*Biol. Rev.***80**, 347–361 (2005). - 6.
Callaway, R. M., Kikodze, D., Chiboshvili, M. & Khetsuriani, L. Unpalatable plants protect neighbors from grazing and increase plant community diversity.

*Ecology***86**, 1856–1862 (2005). - 7.
Emerson, S. E., Brown, J. S., Whelan, C. J. & Schmidt, K. A. Scale-dependent neighborhood effects: shared doom and associational refuge.

*Oecologia***168**, 659–670 (2012). - 8.
Drescher, K., Nadell, C. D., Stone, H. A., Wingreen, N. S. & Bassler, B. L. Solutions to the public goods dilemma in bacterial biofilms.

*Curr. Biol.***24**, 50–55 (2014). - 9.
Nadell, C. D., Bassler, B. L. & Levin, S. A. Observing bacteria through the lens of social evolution.

*J. Biol.***7**, 27 (2008). - 10.
Zomorrodi, A. R. & Segrè, D. Genome-driven evolutionary game theory helps understand the rise of metabolic interdependencies in microbial communities.

*Nat. Commun.***8**, 1563 (2017). - 11.
Gore, J., Youk, H. & van Oudenaarden, A. Snowdrift game dynamics and facultative cheating in yeast.

*Nature***459**, 253–256 (2009). - 12.
Nowak, M. A.

*Evolutionary Dynamics*(Harvard University Press, Cambridge MA, 2006). - 13.
Pepper, J. W. Drugs that target pathogen public goods are robust against evolved drug resistance.

*Evolut. Appl.***5**, 757–761 (2012). - 14.
Archetti, M., Ferraro, D. A. & Christofori, G. Heterogeneity for IGF-II production maintained by public goods dynamics in neuroendocrine pancreatic cancer.

*Proc. Natl Acad. Sci. USA***112**, 1833–1838 (2015). - 15.
Li, X. & Thirumalai, D. Share, but unequally: a plausible mechanism for emergence and maintenance of intratumour heterogeneity.

*J. Royal Soc. Interface***16**, 20180820 (2019). - 16.
Marusyk, A. et al. Non-cell-autonomous driving of tumour growth supports sub-clonal heterogeneity.

*Nature***514**, 54–58 (2014). - 17.
Tabassum, D. P. & Polyak, K. Tumorigenesis: It takes a village.

*Nat. Rev. Cancer***15**, 473–483 (2015). - 18.
Tomlinson, I. & Bodmer, W. Modelling the consequences of interactions between tumour cells.

*Br. J. Cancer***75**, 157 (1997). - 19.
Axelrod, R., Axelrod, D. E. & Pienta, K. J. Evolution of cooperation among tumor cells.

*Proc. Natl Acad. Sci. USA***103**, 13474–13479 (2006). - 20.
Dingli, D., Chalub, F., Santos, F., Van Segbroeck, S. & Pacheco, J. Cancer phenotype as the outcome of an evolutionary game between normal and malignant cells.

*Br. J. Cancer***101**, 1130–1136 (2009). - 21.
Isaac, R. M., Walker, J. M. & Williams, A. W. Group size and the voluntary provision of public goods: Experimental evidence utilizing large groups.

*J. Public Econ.***54**, 1–36 (1994). - 22.
Driscoll, W. W. & Pepper, J. W. Theory for the evolution of diffusible external goods.

*Evolution***64**, 2682–2687 (2010). - 23.
Archetti, M. & Scheuring, I. Review: Evolution of cooperation in one-shot social dilemmas without assortment.

*J. Theor. Biol.***299**, 9–20 (2012). - 24.
Constable, G. W., Rogers, T., McKane, A. J. & Tarnita, C. E. Demographic noise can reverse the direction of deterministic selection.

*Proc. Natl Acad. Sci. USA***113**, E4745–E4754 (2016). - 25.
Hauert, C., Michor, F., Nowak, M. A. & Doebeli, M. Synergy and discounting of cooperation in social dilemmas.

*J. Theor. Biol.***239**, 195–202 (2006). - 26.
Brown, J. S. & Vincent, T. L. Evolution of cooperation with shared costs and benefits.

*Proc. R. Soc. Lond. B: Biol. Sci.***275**, 1985–1994 (2008). - 27.
Archetti, M. Dynamics of growth factor production in monolayers of cancer cells and evolution of resistance to anticancer therapies.

*Evolut. Appl.***6**, 1146–1159 (2013). - 28.
Strogatz, S.

*Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering (Studies in Nonlinearity)*(Westview Pr., 2000). - 29.
Brown, J. S., Cunningham, J. J. & Gatenby, R. A. Aggregation effects and population-based dynamics as a source of therapy resistance in cancer.

*IEEE Trans. Biomed. Eng.***64**, 512–518 (2017). - 30.
Ifti, M., Killingback, T. & Doebeli, M. Effects of neighbourhood size and connectivity on the spatial continuous prisoner’s dilemma.

*J. Theor. Biol.***231**, 97–106 (2004). - 31.
Stylianopoulos, T., Diop-Frimpong, B., Munn, L. L. & Jain, R. K. Diffusion anisotropy in collagen gels and tumors: the effect of fiber network orientation.

*Biophys. J.***99**, 3119–3128 (2010). - 32.
Nowak, M. A., Sasaki, A., Taylor, C. & Fudenberg, D. Emergence of cooperation and evolutionary stability in finite populations.

*Nature***428**, 646–650 (2004). - 33.
Altrock, P. M., Traulsen, A. & Galla, T. The mechanics of stochastic slowdown in evolutionary games.

*J. Theor. Biol.***311**, 94–106 (2012). - 34.
Ashcroft, P., Altrock, P. M. & Galla, T. Fixation in finite populations evolving in fluctuating environments.

*J. R. Soc. Interface***11**, 20140663 (2014). - 35.
Huang, W., Hauert, C. & Traulsen, A. Stochastic game dynamics under demographic fluctuations.

*Proc. Natl Acad. Sci. USA***112**, 9064–9069 (2015). - 36.
Chotibut, T. & Nelson, D. R. Population genetics with fluctuating population sizes.

*J. Stat. Phys.***167**, 777–791 (2017). - 37.
Traulsen, A. & Nowak, M. A. Evolution of cooperation by multi-level selection.

*Proc. Natl Acad. Sci. USA***103**, 10952–10955 (2006). - 38.
Gokhale, C. S. & Traulsen, A. Evolutionary games in the multiverse.

*Proc. Natl Acad. Sci. USA***107**, 5500–5504 (2010). - 39.
Melbinger, A., Cremer, J. & Frey, E. Evolutionary game theory in growing populations.

*Phys. Rev. Lett.***105**, 178101 (2010). - 40.
Houchmandzadeh, B. Fluctuation driven fixation of cooperative behavior.

*Biosystems***127**, 60–66 (2015). - 41.
McAvoy, A., Fraiman, N., Hauert, C., Wakeley, J. & Nowak, M. A. Public goods games in populations with fluctuating size.

*Theor. Popul. Biol.***121**, 72–84 (2018). - 42.
Morris, D. W., Lundberg, P. & Brown, J. S. On strategies of plant behaviour: evolutionary games of habitat selection, defence, and foraging.

*Evolut. Ecol. Res.***17**, 619–636 (2016). - 43.
Vickery, W. L. & Brown, J. S. Spite egotism, population stability, and resource conservation.

*Evolut. Ecol. Res.***11**, 253–263 (2009). - 44.
Coolen, I., Giraldeau, L. A. & Vickery, W. Scrounging behavior regulates population dynamics.

*Oikos***116**, 533–539 (2007). - 45.
Ibrahim-Hashim, A. et al. Defining cancer subpopulations by adaptive strategies rather than molecular properties provides novel insights into intratumoral evolution.

*Cancer Res.***77**, 2242–2254 (2017). - 46.
McKane, A. J. & Newman, T. J. Stochastic models in population biology and their deterministic analogs.

*Phys. Rev. E.***70**, 19 (2004). - 47.
Gerlee, P. & Altrock, P. M. Complexity and stability in growing cancer cell populations.

*Proc. Natl Acad. Sci. USA***112**, E2742–E2743 (2015). - 48.
Gerlee, P. & Altrock, P. M. Extinction rates in tumour public goods games.

*J. R. Soc. Interface***14**, 20170342 (2017). - 49.
Wakano, J. Y., Nowak, M. A. & Hauert, C. Spatial dynamics of ecological public goods.

*Proc. Natl Acad. Sci. USA***106**, 7910–7914 (2009). - 50.
Allen, B., Gore, J. & Nowak, M. A. Spatial dilemmas of diffusible public goods.

*eLife***2**, e01169 (2013). - 51.
Christofori, G., Naik, P. & Hanahan, D. A second signal supplied by insulin-like growth factor II in oncogene-induced tumorigenesis.

*Nature***369**, 414 (1994).

## Acknowledgements

We thank Marco Archetti (University of East Anglia, UK) for data sharing, Avinash Karamchandani (Northwestern University, US) for useful programming tips, and Arne Traulsen (Max Planck Institute for Evolutionary Biology, Germany) for insightful comments on an early draft of this manuscript. The work of P.G. was supported by the Swedish Research Council grant no. 2014-06095 and the Swedish Foundation for Strategic Research grant no. AM13-0046.

## Author information

### Affiliations

#### Department of Integrated Mathematical Oncology, H. Lee Moffitt Cancer Center and Research Institute, Tampa, FL 33629, USA

- Gregory J. Kimmel
- , Joel S. Brown
- & Philipp M. Altrock

#### Department of Mathematical Sciences, Chalmers University of Technology, Gothenburg, SE-412 96, Sweden

- Philip Gerlee

#### Department of Mathematical Sciences, University of Gothenburg, Gothenburg, SE-412 61, Sweden

- Philip Gerlee

### Authors

### Search for Gregory J. Kimmel in:

### Search for Philip Gerlee in:

### Search for Joel S. Brown in:

### Search for Philipp M. Altrock in:

### Contributions

P.G., J.S.B., and P.M.A conceived the project. G.J.K. and P.M.A. performed mathematical modeling. G.J.K and P.M.A. performed computational modeling and statistical analysis. All authors analyzed the data and results, and wrote the manuscript.

### Competing interests

The authors declare no competing interests.

### Corresponding author

Correspondence to Philipp M. Altrock.

## Supplementary Information

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

## Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.