Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Coevolutionary transitions from antagonism to mutualism explained by the Co-Opted Antagonist Hypothesis

## Abstract

There is now good evidence that many mutualisms evolved from antagonism; why or how, however, remains unclear. We advance the Co-Opted Antagonist (COA) Hypothesis as a general mechanism explaining evolutionary transitions from antagonism to mutualism. COA involves an eco-coevolutionary process whereby natural selection favors co-option of an antagonist to perform a beneficial function and the interacting species coevolve a suite of phenotypic traits that drive the interaction from antagonism to mutualism. To evaluate the COA hypothesis, we present a generalized eco-coevolutionary framework of evolutionary transitions from antagonism to mutualism and develop a data-based, fully ecologically-parameterized model of a small community in which a lepidopteran insect pollinates some of its larval host plant species. More generally, our theory helps to reconcile several major challenges concerning the mechanisms of mutualism evolution, such as how mutualisms evolve without extremely tight host fidelity (vertical transmission) and how ecological context influences evolutionary outcomes, and vice-versa.

## Introduction

Mutualisms play essential roles in the generation of biodiversity1; yet, the mechanisms driving mutualism evolution remain enigmatic2. While many mutualisms evolved de novo via the exchange of costless by-products, many other mutualisms are thought to have originated from antagonistic interactions3. Several studies have elucidated ecological or evolutionary conditions driving transitions from mutualism to antagonism (mutualism breakdown)4,5, yet limited theory has been developed to mechanistically explain the converse, evolutionary transitions from antagonism to mutualism (but see6,7). Models of virulence predict evolution of reduced antagonism, perhaps toward mutualism, in cases of parasitism where host fidelity is high and availability of alternate hosts is low8,9. In these models, host fidelity is achieved when lineages of parasites and their host are tightly linked through vertical transmission from parent to offspring. Compelling evidence for the ‘virulence theory’ comes from interactions involving microbial parasites10,11. The fact that horizontal transmission of mutualistic partners is common in nature, however, is a major challenge to virulence theory12, leaving the mechanisms of mutualism largely unresolved for some of the most conspicuous partnerships on Earth.

We develop a general eco-coevolutionary framework for investigating the evolutionary transition from antagonism to mutualism and use data-based modeling to advance and evaluate a new hypothesis: the Co-Opted Antagonist (COA) Hypothesis, whereby co-option of an antagonistic species to perform a beneficial function changes the ecology of the interaction and, simultaneously, the interaction evolves to net mutualism. We evaluate the COA by studying plant-insect interactions in which insects are pollinators and herbivores of the same plant species13,14,15. This type of interactions may be surprisingly common in nature; for example, over half of Lepidoptera in a central European community (54% of 995 species) nectar at (and thus potentially pollinate) at least one of their larval host plants16. These observations lead us to posit that interactions between some plants and insects may have transitioned from pure herbivory to pollination as plants evolved to co-opt insect search behavior for host plants to disperse their gametes.

To evaluate the COA hypothesis we develop data-based eco-coevolutionary models for a community in which a lepidopteran insect initially oviposits on, but does not pollinate, its larval host plants, which are pollinated by other species. Evolutionary co-option of the antagonist is achieved if plant traits evolve in a way that allows the antagonist to pollinate it (e.g., larger flowers shaped so as to allow the insect to nectar and transfer pollen). The evolutionary integration of two consumer-resource interactions, herbivory and nectar-feeding (with pollination as a by-product) may achieve the tight coupling required to induce strong selective pressures on plants (beyond those induced by other pollinators) and the insect. This coupling may in turn create the ecological conditions for plant-insect coevolution to drive pollination benefits beyond trophic costs, thus assimilating the herbivore as a pollinator (either alongside or replacing other pollinators). From the COA hypothesis, we predict that community context, in particular the degree of specialization of the herbivory and nectar-feeding interactions, should influence the transition: generalist larvae that can feed on multiple host plant species may facilitate the transition by ameliorating herbivory costs, whereas generalist adults feeding at multiple nectar sources may hinder the transition by amplifying oviposition costs.

## Results

We develop an eco-coevolutionary framework of evolutionary transitions from antagonism to mutualism (Methods), which we adapt to derive a fully-ecologically parameterized eco-coevolutionary model for a small community involving the hawkmoth Manduca sexta and its associated plant species in southern Arizona, USA (Box 1; Fig. 1). This community is an excellent system to evaluate the COA hypothesis because—while relatively specialized compared to most plant-pollinator or plant-herbivore communities in nature—it involves multiple plant species and is small enough for tractability. This allows us to test the effects of alternative trophic interactions (larval host plants and nectar sources) on the transition to mutualism and cast the COA hypothesis in slightly more general terms as a step towards studying full communities.

We link population dynamics and trait evolution via the adaptive dynamics framework17,18. We study four scenarios involving M. sexta and: (1) each Datura species separately, (2) both Datura species together, (3) each Datura species and an alternative larval host plant, and (4) each Datura species and an alternative nectar source. In each scenario, an ancestral insect oviposits on, but does not pollinate, ancestral Datura species i and nectar feeds at other plants. We model coevolution of plant traits, such as flower size (xiB), that allow the insect to pollinate it and subsequently increase pollination benefits (seed set of pollinated flowers, bi) as well as plant and insect traits driving attraction and defense. Attraction entails coevolution of plant traits such as the production of volatiles (xiV) that attract the insect and insect traits such as its sensitivity to olfactory cues (yiV) to locate larval host plants. In the model, coevolution of attraction (via xiV and yiV) changes the visitation rate, vi, which affects both nectaring and oviposition19. Defense entails coevolution of plant traits such as the production of chemical defenses (xiH) and insect traits such as increasing tolerance to plant secondary compounds (yiH). Coevolution of defense in the model (via xiH and yiH) affects the herbivory rate, hi. In the model, plant traits trade off with competitive ability (e.g., mutant plants with highly-attractive flowers or strong chemical defenses have low competitive ability) and insect traits trade off with oviposition (e.g., mutant insects with high sensitivity to olfactory cues or high tolerance to plant defenses produce fewer eggs).

### Ancestral antagonism

An ancestral insect persists as a pure antagonist of its exclusive larval host plant species i (bi = 0) when:

$${f}_{i}{s}_{i} \,> \, 1$$
(1)

where fi is insect lifetime fecundity and si is larval success (probability of maturing rather than dying) on plant species i (Methods). This defines the interaction breakdown boundary (dashed gray lines in Fig. 2) below which the ancestral insect is extinct. For an ancestral insect with similar trait values as M. sexta to persist with ancestral plant species with similar trait values as D. wrightii or D. discolor, the coevolved visitation and herbivory rates (white points in Fig. 2) must be above the interaction breakdown boundary.

### Ecological conditions for the transition from antagonism to mutualism

An interaction involving an insect that pollinates its exclusive larval host plant species i (bi >0) is mutualistic when:

$${f}_{i}{s}_{i} \,<\, 1$$
(2)

#### Methods

Equation (2) is the interaction transition boundary (dotted black lines in Fig. 2, which replace the dashed gray lines from the ancestral interaction) above which the interaction is net antagonistic (gray regions of Fig. 2) and below which the interaction is net mutualistic (green regions of Fig. 2). Equation (2) reveals an inherent conflict of interest between plant and insect: high insect life-time fecundity (fi) and larval success (si) benefit the insect but not the plant, and vice-versa. Equation (1) now gives the invasion criterion above which the insect can increase from low density and persist as a net antagonist of the plant despite being co-opted as a pollinator because its herbivory costs to the plant exceed its pollination benefits.

The new interaction breakdown boundary (solid black lines in Fig. 2) below which the insect always goes extinct (white regions of Fig. 2) is now approximated (Methods) by:

$${f}_{i}{s}_{i}\left(1+{b}_{i}\right) \,> \, 1$$
(3)

Equation (3) shows that pollination benefits, bi, effectively create a parameter region in which the insect persists as a net mutualist by buoying plant density and thus lowering the interaction breakdown boundary (Eq. 3 vs. Eq. 1). Within this mutualistic region, however, the insect cannot invade from very low density because, when the insect is rare, it cannot buoy plant density sufficiently to maintain a positive per capita growth rate (mathematically, the insect’s invasion criterion—Eq. 1—cannot hold when Eq. 2 is satisfied). The mutualistic region is therefore characterized by bistability (see Supplementary Figure 1) and an Allee threshold above which the insect persists as a net mutualist of the plant and below which it goes extinct.

The interaction transition boundary (Eq. 2) is the exact reverse condition of the interaction breakdown boundary in the ancestral interaction (Eq. 1). Thus, co-option of the antagonist as a pollinator (i.e., bi > 0) does not itself create mutualism because the phenotype of any viable ancestral interaction remains in the antagonistic regions of Fig. 2 following co-option. Integration of the antagonist as a pollinator, however, fundamentally changes the ecology of the interaction by making mutualism possible (within the green regions of Fig. 2 defined by Eqs. 2 and 3) and by reshaping the evolution of plant and insect traits governing the outcome of the interaction by changing their underlying selection gradients (Methods).

Given a herbivorous insect that can pollinate, the evolutionary transition to mutualism requires that selection yields a coevolutionary stable state (coESS) at which pollination benefits exceed herbivory costs for the plant (within the green regions of Fig. 2). We model simultaneous coevolution of three responses: pollination (via xiB, affecting pollination benefits from the insect, bi), attraction (via xiV and yiV, affecting visitation rate, vi) and defense (via xiH and yiH, affecting herbivory rate, hi). The question becomes: can coevolution drive the system across the interaction transition boundary from antagonism to mutualism?

### Coevolutionary transitions from antagonism to mutualism

Plant evolution of pollination benefits from the antagonist (bi) creates and expands the green mutualistic regions of Fig. 2, as depicted by the green arrows (Supplementary Movies 1 and 2). Simultaneous to the evolution of pollination benefits, coevolution of attraction and defense leads to a coESS at which the interaction has transitioned to net mutualism, as depicted by the orange arrows in Fig. 2 pointing from the ancestral coESS (white points) to the new coESS (black points). This is true irrespective of ancestral trait values: for any initial trait values in the ancestral interaction (above the dashed gray lines in Fig. 2), coevolution of pollination benefits, attraction, and defense following the co-option of the antagonist as a pollinator leads the interaction to mutualism. Coevolution, in turn, affects the ecology of the interactions. As pollination benefits, attraction and defense coevolve (Fig. 2e, f; Supplementary Fig. 2), plant and insect densities increase such that both species attain greater equilibrium density following the evolutionary transition to mutualism (Fig. 2c, d).

The same results apply to a model in which both plant species co-occur (Fig. 3) and thus coevolve with each other and the antagonist. We model this case based on data from our experiments involving both D. wrightii and D. discolor19. Comparing the predicted coESS with independent data provides a strong validation of the model—at least of the selection gradients on attraction and defense around the contemporary state. The coESS for attraction and defense predicted by the model (black points in Fig. 3) are well within the standard errors our empirical estimates (blue crossbars in Fig. 3). Evolution leads to a coESS that would have driven evolutionary purging of the insect in either one-plant species community (i.e., the coESS in Fig. 3 would be in the extinction regions of Fig. 2), underscoring the importance of multiple larval host plants for the system’s eco-evolutionary stability. Coevolutionary dynamics cycle transiently around the coESS (Fig. 3e, f) due to varying selective pressures on each plant species and the antagonist, driving slight oscillations in the ecological equilibria (Fig. 3c, d).

The predicted coESS for pollination benefits from the antagonist (seed set of flowers pollinated by M. sexta) and oviposition efficiency (average number of eggs laid by a female M. sexta after a floral visit) are also within the standard errors of our empirical estimates (Supplementary Table 3). The presence of multiple Datura species has other notable qualitative and quantitative effects on evolutionary outcomes: both species evolve lower attraction (lower vi) and greater defense (lower hi) than they do in a one-plant species community (Fig. 2). Intriguingly, the Datura species evolve different strategies in response to M. sexta: D. wrightii evolves high attraction and defense (high vw, low hw), while D. discolor evolves low attraction and defense (low vd, high hd). This difference arises because D. wrightii receives greater pollination benefits, but is also preferentially visited—and oviposited on—by M. sexta (both in the model and field; Supplementary Data 1). D. wrightii therefore differentially benefits from M. sexta pollination, but must evolve relatively high defense, while D. discolor mitigates herbivory costs by evolving low attraction, and therefore requires relatively low defense.

### Coevolutionary transitions in more generalized trophic interactions

We now assess how alternative plant species that either partition the costs or share the benefits of the insect influence evolutionary transitions from antagonism to mutualism. Our study system (Box 1) is well suited because it is small enough to enable model parameterization and involves both an alternative larval host plant and an alternative nectar source. Recent M. sexta trophic generalization to P. parviflora and A. palmeri are well documented in nature20,21.

When each Datura species and the alternative larval host plant coevolve with M. sexta, mutualism can evolve over a large parameter region (Fig. 4a, b) because the alternative larval host plant effectively subsidizes some of the costs to the Datura species associated with oviposition. Conversely, the presence of the alternative nectar source narrows the parameter region in which mutualism can arise (Fig. 4c, d) by increasing oviposition and amplifying the antagonistic component of the interactions. In both scenarios, coevolution (Fig. 4i–l) increases the equilibrium densities of both Datura species and M. sexta (Fig. 4e–h).

## Discussion

We advance and evaluate the Co-Opted Antagonist (COA) Hypothesis as a mechanism for explaining the coevolutionary transition from antagonism to mutualism. The COA mechanism is an eco-coevolutionary process whereby natural selection favors the co-option of an antagonist to perform a beneficial function and coevolution of the interacting species drives the interaction from antagonistic to net mutualistic. The transition involves a counterintuitive evolutionary route through increased interactions with an antagonist. This is an example of a dangerous liaison22 in which there are potential benefits of an interaction, but also substantial costs. Evolutionary co-option of the antagonist requires the evolution of traits, such as changes in floral structure or phenology in plant species, that allow an antagonist to function as a mutualist and subsequently increase mutualistic benefits obtained from the co-opted antagonist. We model coevolution of mutualistic benefits and two other responses that drive a transition from antagonism to net mutualism: attraction, which determines the strength of the interaction; and defense, which reduces the costs incurred by the species due to interacting with the antagonist. These traits may have evolved initially in response to other species, but are driven by selective pressures imposed predominantly by the antagonist.

To evaluate the COA hypothesis, we develop a general eco-coevolutionary framework that we adapt to derive a data-based model of a small community in which a herbivorous insect pollinates its larval host plants (Box 1). Very few models focus on interactions in which an insect is both a pollinator and a herbivore of the same plant species (e.g. 23,24), despite evidence that these interactions may be surprisingly common in nature16. Our model is based on the well-studied Manduca-Datura system and is designed to be parameterized with data from empirical studies. Our eco-coevolutionary model reveals how evolutionary transitions from antagonism to net mutualism can occur via the COA hypothesis for systems embodying this biology.

For both Datura species, evolution drove a transition from antagonism to net mutualism in both the one- and two-plant species communities despite tightly-coupled interactions with a voracious herbivore. Importantly, the adapted trait values predicted by the model are statistically indistinguishable from our empirical estimates (Supplementary Table 3). The model is further supported by other predictions that align with empirical findings. The model predicts that D. wrightii evolves greater attraction and defense than does D. discolor. M. sexta has a strong preference for nectar-feeding at D. wrightii over D. discolor19, perhaps reflecting selection on D. wrightii for floral traits that attract M. sexta. Indeed, D. wrightii has flowers that are highly attractive to moths20, whereas D. discolor has far less conspicuous flowers25,26. As nectar-feeding and oviposition behaviors are tightly linked in M. sexta19, however, the coevolution of attraction necessarily changes pollination and oviposition simultaneously. For interactions lacking this behavioral linkage, coevolution might favor high floral visitation rates without an associated increase in oviposition risk, which would further facilitate transitions to mutualism. D. wrightii is also heavily defended against herbivory, producing proteinase inhibitors and other defensive compounds27. The seemingly paradoxical combination of D. wrightii being heavily defended and strongly preferred over D. discolor19 is consistent with our model predictions. The strong oviposition preference may be a by-product of a greater visitation rate, as assumed in the model. It could also be that D. wrightii is a higher-quality host plant for M. sexta than is D. discolor19, and M. sexta has evolved to tolerate its defenses. This difference is incorporated to a degree into the model via greater maturation and lower larval mortality on D. wrightii than on D. discolor.

The COA hypothesis resolves some of the key difficulties of the virulence theory of antagonism-to-mutualism evolution8,9, while also predicting how mutualism can evolve in many of the most conspicuous partnerships on Earth. In virulence theory, mutualism evolution requires high host fidelity through vertical transmission and an evolutionary decrease in virulence. In the COA, evolutionary co-option also increases coupling with an antagonist, potentially increasing the costs of antagonism. Rather than a partner evolving lower virulence, however, it is the host plant evolving defense that reduces the antagonistic component of the interaction (against selection on the co-opted antagonist for greater antagonism), together with a level of attraction that increases the by-product benefit of the interaction (pollination, in our model system).

The COA hypothesis also makes novel predictions about the influence of the ecological context of the interaction on evolutionary outcomes, showing that even slight trophic generalization by the insect (from one larval host plant species to two larval host plants and/or nectar sources) can profoundly affect the transition. In the M. sexta system, an alternative larval host plant increases the parameter region in which mutualism occurs relative to the one-plant species communities (Fig. 4a, b vs. 2a, b) by effectively subsidizing the costs of herbivory experienced by the Datura species. Conversely, an alternative nectar source greatly reduces the parameter region in which mutualism occurs relative to the one-plant species communities (Fig. 4c, d vs. 2a, b) by increasing oviposition and its consequent herbivory costs. The important insight is that lower partner fidelity due to even slight trophic generalization can either facilitate or hinder the transition from antagonism to mutualism depending on the specific resource axis along which the partner’s niche broadens.

We have focused on evolutionary transitions from antagonism to mutualism, but there are also well established transitions from mutualism to antagonism or from mutualism to interaction breakdown (i.e., no interaction at all)4,5. Our results suggest that evolution can buffer interactions from transitioning to antagonism (as indicated by the distance from the coESSs to the interaction transition boundary in Figs. 24), but may predispose interactions to breakdown via partner extinction (as indicated by the proximity of the coESSs to the interaction breakdown boundary in Figs. 24). Thus, mutualism breakdown might more often result in partner loss28,29 rather than a transition back to antagonism. Varying the evolutionary coefficients in the model reveals cases in which coevolution drives evolutionary purging of the antagonist or, more rarely, in which the net antagonism persists despite co-option of the antagonist (Fig. 5). Our model results also suggest that environmental change, for example, could result in interaction breakdown or even transitions back to antagonism if it were to drastically alter evolutionary constraints (Fig. 5).

The COA framework reveals several key factors underlying evolutionary transitions from antagonism to mutualism. First, species (plants in our system) must possess traits that evolution can modify to co-opt the antagonist. In our system, for example, plants must have previously evolved flowers to benefit from other pollinators, which evolution could modify to co-opt the antagonist as a pollinator. Otherwise, the interaction would always be antagonistic. Second, species (plants) must be able to mount responses, such as attraction and defense, that could coevolve toward greater mutualistic benefits at lower trophic costs. Otherwise, the interaction would remain net antagonistic despite co-option of the antagonist. Third, if fitness costs are too extreme, the antagonism would either persist (gray regions of Fig. 5) or would disappear via evolutionary purging of the antagonist (white regions of Fig. 5). Finally, species (plants) must experience stronger selective pressures from the antagonist than from other species in the community.

For tractability, our model focuses on a system that is relatively specialized compared to most plant-pollinator and plant-herbivore communities. A critical question for future studies is how other interactions influence evolutionary trajectories from antagonism to mutualism. We briefly highlight two examples of how other interactions might modify the transition to mutualism. First, evolution of traits that attract the antagonist might come at the expense of attracting other pollinators or deterring other herbivores. If evolutionary constraints are too extreme, for example, the antagonism might persist or evolution might purge the interaction in favor of other pollinators. Second, competition among plant species might modify coevolution, for example, by increasing interactions between the insect and competitively-dominant and abundant plant species. More broadly, extending the COA framework to larger communities is a key step.

The COA hypothesis generates several key predictions that could be tested empirically. We highlight two here. First, the COA predicts that the ecological context of an interaction strongly influences its evolutionary outcome. For example, our models suggest that an interaction is more likely to be mutualistic for species interacting with a focal partner that is more generalized in its antagonistic interactions than its mutualistic interactions (e.g., an insect that has multiple larval host plant species and few nectar sources). Second, the COA predicts that evolution of traits that allow an antagonist to function as a mutualist are not sufficient for an interaction to be mutualistic: coevolution of the interacting species is also necessary. For example, evidence that a herbivorous insect nectars at its larval host plant species does not inherently indicate that it is a mutualist: its benefits as a mutualist must be assessed relative to its costs as an antagonist30.

The COA hypothesis helps to resolve the evolution of mutualisms that did not evolve de novo from costless interactions. For example, molecular evidence suggests that mutualistic bacteria evolved more frequently from parasitic than from free-living ancestors11. While virulence theory posits that it was the parasites that evolved lower virulence, it is also possible that the hosts evolved increased defense that reduced the virulence of co-opted partners in line with the COA hypothesis. Similarly, some insects have co-opted their parasitic endosymbionts for defense against parasitoids31. More generally, mycorrhizal mutualisms are thought to have evolved from parasitism3, perhaps as plants co-opted the fungi to acquire their nutrients; early Angiosperms may have co-opted the foraging behaviors of pollen-feeders to disperse their pollen3; and some ant-defense mutualisms may have evolved from antagonism as ants were co-opted for defense32,33. Our eco-coevolutionary framework could help to unify ideas from the COA hypothesis, virulence theory8,9, and other frameworks6,7 to explain mutualism evolution across these natural systems.

We generated the COA hypothesis from the study of a pollination-herbivory system. However, the COA mechanism—evolutionary co-option and coevolutionary mitigation of antagonism (through defense) and amplification of mutualistic benefits (through attraction)—is not specific to this system and could apply much more broadly, including to microbial, mycorrhizal, and ant-defense mutualisms. Our results call for the integration of two lines of research that have proceeded largely independently: evolution of different interaction types (e.g., mutualism, antagonism) and interaction structures (e.g., specialized, generalized). This integration requires an eco-evolutionary perspective2 in which the greatest progress will come by combining theoretical and empirical approaches. This study provides a critical step towards integrating theory and data to study the ecological and evolutionary dynamics of biological communities.

## Methods

### General framework for eco-coevolutionary transitions from antagonism to mutualism

We develop a general framework in which we model interactions between host species i (density Hi) and its partner species k (density Fk), which are initially purely antagonistic. The model is general, but could be applied broadly to bacterial hosts and parasitic phages or plant hosts and animal or fungal partners, for example. The ecological dynamics of this community (without evolution) are given by:

$$\frac{d{H}_{i}}{{dt}}={g}_{i}{H}_{i}\left(1-\mathop{\sum}\limits_{j}{q}_{{ij}}{H}_{j}\right)+{\sum }_{k}{f}\!_{ik}\left[\beta \left({H}_{i},{F}_{k}\right),\alpha \left({{H}_{i},F}_{k}\right)\right]$$
(4a)
$$\frac{d{F}_{k}}{{dt}}=\mathop{\sum}\limits_{i}{f}_{ki}\left[\beta \left({H}_{i},{F}_{k}\right),\alpha \left({{H}_{i},F}_{k}\right)\right]-{\delta }_{k}{F}_{k}$$
(4b)

The first term of Eq. (4a) describes host population growth in the absence of partner species, where gi is its intrinsic per capita growth rate and qij is the competitive effect of host j on host i for other limiting factors. The general function fik describes the effects of interactions with partner k on host i: β(Hi, Fk) gives the potential mutualism and α(Hi, Fk) describes the antagonism. In Eq. (4b), the general function fki gives the effects of interactions with host i on partner k and δk is the partner’s per capita mortality rate.

To derive an explicit eco-coevolutionary model, we apply Equation (4) to model interactions between a single host species and its exclusive partner species (for the sake of simplicity) in terms of host traits xi and partner traits yi (involved in interactions with host i); the ecological dynamics of which are given by:

$$\frac{1}{{H}_{i}}\frac{d{H}_{i}}{{dt}}={g}_{i}\left(1-{q}_{i}{H}_{i}\right)+\frac{b\left[{x}_{i}^{B}\right]v\left[{x}_{i}^{V},{y}_{i}^{V}\right]{F}_{k}}{{S}_{i}+v\left[{x}_{i}^{V},{y}_{i}^{V}\right]{F}_{k}}-h\left[{x}_{i}^{H},{y}_{i}^{H}\right]v\left[{x}_{i}^{V},{y}_{i}^{V}\right]{F}_{k}$$
(5a)
$$\frac{1}{{F}_{k}}\frac{d{F}_{k}}{{dt}}=e\left[{y}_{i}^{V},{y}_{i}^{H}\right]v\left[{x}_{i}^{V},{y}_{i}^{V}\right]h\left[{x}_{i}^{H},{y}_{i}^{H}\right]{H}_{i}-{\delta }_{k}$$
(5b)

where b is the mutualistic benefits to the host, v is the visitation rate, Si is a saturation constant, h is the costs of antagonism to the host and its benefits to the partner, and e is the partner’s conversion efficiency. The mutualistic and antagonistic interactions are assumed to contribute additively to host population growth and multiplicatively to partner population growth, assumptions that may be valid for many types of interactions, but will not apply universally. To prevent unbounded population growth in the model, the effects of mutualism on host population growth are assumed to saturates with increasing partner density.

The function b[xiB] gives the mutualistic benefits of the partner as a function of host trait xiB:

$$b\left[{x}_{i}^{B}\right]={b}_{{\max },i}\left(\frac{2}{1+{e}^{-{B}_{i}^{{\prime} }{x}_{i}^{B}}}-1\right)$$
(6a)

where bmax,i gives the maximum mutualistic benefits and $${B}_{i}^{{\prime} }$$ is a saturation constant. The interaction is purely antagonistic when xiB = 0. As xiB increases, the mutualistic benefits b[xiB] increase towards bmax,i.

The function v[xiV, yiV] gives visitation rate as a sigmoid function of host trait xiV and partner trait yiV:

$$v\left[{x}_{i}^{V},{y}_{i}^{V}\right]=\frac{{v}_{{\max },i}}{1+{e}^{-{V}_{i}^{{\prime} }\left({x}_{i}^{V}+{y}_{i}^{V}\right)}}$$
(6b)

where vmax,i is the maximum visitation rate and $${V}_{i}^{{\prime} }$$ determines how rapidly visitation rate changes as host and partner traits change. As xiV or yiV increase, the visitation rate increases and approaches vmax,i when xiV + yiV → ∞. As xiV or yiV decrease, the visitation rate decreases and approaches zero when xiV + yiV → −∞. Negative values of xiV indicate that the host species is reducing its attraction of the partner species.

The function h[xiH, yiH] gives the costs of antagonism to the host and its benefits to the partner, which is described via a sigmoid function of the difference between host trait xiH and partner trait yiH:

$$h\left[{x}_{i}^{H},{y}_{i}^{H}\right]=\frac{{h}_{{\max },i}}{1+{e}^{{H}_{i}^{{\prime} }\left({x}_{i}^{H}-{y}_{i}^{H}\right)}}$$
(6c)

where hmax,i gives the maximum antagonism and $${H}_{i}^{{\prime} }$$ determines how antagonism changes as the difference between host and partner traits increases. When xiH > yiH, antagonism declines and approaches zero when xiHyiH → ∞, while when xiH < yiH, antagonism increases and approaches hmax,i when xiHyiH → -∞ (unlike xiV, xiH cannot be negative).

Partner traits yiV and yiH trade off with conversion efficiency via the function e[yiV, yiH] as defined by:

$$e\left[{y}_{i}^{V},{y}_{i}^{H}\right]={e}_{{\max },i}{e}^{-\left({c}_{I,i}^{V}{\left({y}_{i}^{V}\right)}^{2}+{c}_{I,i}^{H}{\left({y}_{i}^{H}\right)}^{2}\right)}$$
(6d)

where emax,i is the maximum conversion efficiency when interacting with host i (when yiV = yiH = 0), and cI,iV and cI,iH determine how rapidly conversion efficiency declines as yiV or yiH increase, thus quantifying the costliness of traits yiV and yiH, respectively. This trade-off shape was chosen because it is unimodal and constrains conversion efficiency to always be positive. Host trade-offs are defined below (Eq. 8c).

### Host-partner coevolutionary dynamics

We model coevolution via the adaptive dynamics framework17,18. Coevolution of a mutant host trait ximut and partner trait yimut (for any general traits xi and yi) is given by:

$$\frac{d{{x}_{i}}^{{mut}}}{d\tau }={\mu }_{x}{\left.\frac{\partial {W}_{H}\left({{x}_{i}}^{{mut}},{x}_{i},{y}_{i}\right)}{\partial {{x}_{i}}^{{mut}}}\right|}_{{{x}_{i}}^{{mut}}={x}_{i}}$$
(7a)
$$\frac{d{{y}_{i}}^{{mut}}}{d\tau }={\mu }_{y}{\left.\frac{\partial {W}_{F}\left({{y}_{i}}^{{mut}},{y}_{i},{x}_{i}\right)}{\partial {{y}_{i}}^{{mut}}}\right|}_{{{y}_{i}}^{{mut}}={y}_{i}}$$
(7b)

where τ is the evolutionary timescale, μx and μy give, respectively, the rates of host and partner evolution, and WH(ximut,xi,yi) and WF(yimut,yi,xi) are the invasion fitness (per capita growth rate when rare) of a mutant host and partner species with trait ximut and yimut in a resident community with trait xi and yi, respectively. The partial derivatives $${\left.\partial {W}_{H}/\partial {{x}_{i}}^{{mut}}\right|}_{{{x}_{i}}^{{mut}}={x}_{i}}$$ and $${\left.\partial {W}_{F}/\partial {{y}_{i}}^{{mut}}\right|}_{{{y}_{i}}^{{mut}}={y}_{i}}$$ are the selection gradients.

We model coevolution of mutualistic benefits from the focal partner species (via b), attraction (via v), and defense (via h). The invasion fitness of the mutant host and a mutant partner are given by:

$${W}_{H}={g}_{i}\left(1-q\left[{{x}_{i}}^{{mut}},{x}_{i}\right]{{H}_{i}}^{\ast }\right)+\frac{b\left[{x}_{i}^{B,{mut}}\right]v\left[{x}_{i}^{V,{mut}},{y}_{i}^{V}\right]{{F}_{k}}^{\ast }}{{S}_{i}+v\left[{x}_{i}^{V,{mut}},{y}_{i}^{V}\right]{{F}_{k}}^{\ast }}-h\left[{x}_{i}^{H,{mut}},{y}_{i}^{H}\right]{v\left[{x}_{i}^{V,{mut}},{y}_{i}^{V}\right]{F}_{k}}^{\ast }$$
(8a)
$${W}_{F}=e\left[{y}_{i}^{V,{mut}},{y}_{i}^{H,{mut}}\right]v\left[{x}_{i}^{V},{y}_{i}^{V,{mut}}\right]h\left[{x}_{i}^{H},{y}_{i}^{H,{mut}}\right]{{H}_{i}}^{\ast }-{\delta }_{k}$$
(8b)

where Hi* and Fk* are species’ densities at the ecological equilibrium (of Eq. 5). The functions b, v, h, and e are given by Eq. (6a–d), respectively, where xi and yi are replaced with ximut in Eq. (8a) and yimut in Eq. (8b). The function q[ximut,xi] describes trade-offs between mutant host traits and mutant host competitive ability as defined by:

$$q\left[{{x}_{i}}^{{mut}},{x}_{i}\right]=1+{c}_{H,i}^{B}\left({\left({{x}_{i}}^{B,{mut}}\right)}^{{s}_{i}^{B}}-{\left({{x}_{i}}^{B}\right)}^{{s}_{i}^{B}}\right)+{c}_{H,i}^{V}\left({\left({{x}_{i}}^{V,{mut}}\right)}^{{s}_{i}^{V}}-{\left({{x}_{i}}^{V}\right)}^{{s}_{i}^{V}}\right)+{c}_{H,i}^{H}\left({\left({{x}_{i}}^{H,{mut}}\right)}^{{s}_{i}^{H}}-{\left({{x}_{i}}^{H}\right)}^{{s}_{i}^{H}}\right)$$
(8c)

If ximut > xi for any trait, the competitive effect experienced by the mutant host is increased by an amount taken to be proportional (for simplicity) to the difference between the trait values, ximutxi, whereas if ximut < xi, the competitive effect experienced by the mutant host is decreased by that amount. The coefficients cH,iB, cH,iV, and cH,iH measure the costs associated with the trade-off for each trait, while the shape parameters siB, siV, and siH define whether the trade-offs are linear (si = 1), concave (si < 1), or convex (si > 1).

Mutualism can evolve via the COA for all trade-off shapes (Supplementary Fig. 3). Parameter space plots show that the interaction transitions from antagonism to net mutualism when the costs associated with host traits underlying attraction (cH,iV) and defense (cH,iH) are within a range beyond which there is evolutionary purging of the partner (Supplementary Fig. 3a–c). Only with convex trade-offs can the net antagonism persist. The coevolution of mutualism also requires that the costs associated with partner traits underlying visitation (cFV) and antagonism (cFH) exceed a threshold (Supplementary Fig. 3d–f) below which there is evolutionary purging of the partner (linear or convex trade-offs) or the net antagonism persists (linear or concave trade-offs). Coevolution of mutualism occurs across greater parameter ranges when the trade-offs are linear or slightly concave because costs increase less rapidly than with convex trade-offs.

### Ecological model of plant-insect interactions

We tailor the general model (Eq. 4) to model populations of D. wrightii (density Pw) and D. discolor (density Pd) interacting with M. sexta. We scale the model so that Pi = 1 in the absence of M. sexta: thus, Pi >1 indicates that pollination benefits exceed herbivory costs, and Pi < 1 indicates that herbivory costs exceed pollination benefits. The Datura species do not rely obligately on M. sexta and, consistent with ecology of the natural community (Box 1), the model incorporates the alternative host plant, Proboscidea parviflora (density Pp), and the alternative nectar source, Agave palmeri. The ecological dynamics of this community (without evolution) are given by:

$$\frac{1}{{P}_{i}}\frac{d{P}_{i}}{{dt}}=\left(1-{P}_{i}\right)+\frac{{b}_{i}{v}_{i}A}{H+{v}_{i}A}-{h}_{i}{L}_{i}$$
(9a)
$$\frac{d{L}_{i}}{{dt}}=\varepsilon {e}_{i}{v}_{i}{P}_{i}A-{m}_{i}{h}_{i}{L}_{i}-{d}_{i}{L}_{i}$$
(9b)
$$\frac{{dA}}{{dt}}=\mathop{\sum}\limits_{i}{\rho }_{i}{m}_{i}{h}_{i}{L}_{i}-{d}_{A}A$$
(9c)

Equation (9a) describes the population dynamics of plant species i (D. wrightii, D. discolor, or P. parviflora). Equation (9b,c) give the dynamics of M. sexta: Li gives the larvae density on plant species i, which recruit into the adult population, A. Pollination is described by the term biviA/(H+viA), where bi is the per capita growth of plant species i due to pollination by the antagonist, vi is the visitation rate to plant species i per antagonist adult, and H is the saturation constant for pollination. Oviposition is given by εeiviPiA, where ei is the oviposition efficiency (number of eggs laid per floral visit) and ε is the fractional increase in egg production due to nectar-feeding at A. palmeri. Floral visits lead to both pollination and oviposition because these behaviors have been shown to be tightly linked in M. sexta19. Pollination and oviposition are given by saturating and linear functions, respectively, based on our data (Supplementary Data 1). Herbivory damage is given by the term hiLi, where hi is the herbivory rate per larvae on plant species i. Larvae mature at rate mihiLi, where mi is the maturation efficiency (fraction of larvae maturing on plant species i). Larval mortality on plant species i is di, adult mortality is dA, and ρi is pupae survival (due to data constraints, we include pupae survival in our estimates of maturation mi, set ρi = 1, and drop ρi from equations hereafter). Equation (9a) gives the dynamics of the alternative larval host plant, P. parviflora (bp = 0 and cannot evolve), which can coevolve attraction and defense. The alternative nectar source, A. palmeri, is incorporated within the model via the parameter ε.

### Model scaling

Without the antagonist, plant population growth is given by gi (1 – qiPi), where gi is the per capita growth rate of plant species i due to autonomous self-pollination or pollination by other species and qi is plant self-limitation. As qi is very difficult to quantify in nature, we scale the model so that Pi = 1 without the antagonist. We scale plant density ($$\hat{{P}_{i}}={q}_{i}{P}_{i}$$), larvae density ($$\hat{{L}_{i}}={q}_{i}{L}_{i}$$), herbivory rate ($$\hat{{h}_{i}}={h}_{i}/{q}_{i}$$), maturation efficiency ($$\hat{{m}_{i}}={q}_{i}{m}_{i}$$), and survival of pupae ($$\hat{{\rho }_{i}}={\rho }_{i}/{q}_{i}$$); where the hats denote scaled quantities and are dropped elsewhere for clarity. Thus, the model is scaled for parameterization, but is not non-dimensionalized. We then scale gi to 1 such that pollination benefits, bi, are estimated by the ratio of the seed set of moth-pollinated flowers to autonomously self-pollinated flowers. Parameter estimates are for scaled quantities.

### Interaction breakdown boundary for ancestral interaction in a one-plant species community

For the ancestral insect to persist, its per capita growth rate must be positive when it is rare (i.e., at Pi* = 1, Li* = 0, A* = 0). In stage-structured models, the per capita growth rate is given by the dominant eigenvalue (λD) of the matrix:

$$\left[\begin{array}{cc}-{m}_{i}{h}_{i}-{d}_{i} & \varepsilon {e}_{i}{v}_{i}{{P}_{i}}^*\\ {m}_{i}{h}_{i} & -{d}_{A}\end{array}\right]$$

which is given by:

$${\lambda }_{D}=\frac{1}{2}\left(-{d}_{A}-{d}_{i}-{m}_{i}{h}_{i}+\sqrt{{({d}_{A}+{d}_{i}+{m}_{i}{h}_{i})}^{2}-4({d}_{A}\left({d}_{i}+{m}_{i}{h}_{i}\right)-\varepsilon {e}_{i}{v}_{i}{m}_{i}{h}_{i})}\right).$$

For the insect to persist, λD must have a positive real part, which occurs only when the second term in the square root of λD is negative; i.e., $${d}_{A}\left({d}_{i}+{m}_{i}{h}_{i}\right)-\varepsilon {e}_{i}{v}_{i}{m}_{i}{h}_{i} \,<\, 0$$. Rearranging this condition yields: $$\left(\frac{\varepsilon {e}_{i}{v}_{i}}{{d}_{A}}\right)\left(\frac{{m}_{i}{h}_{i}}{{m}_{i}{h}_{i}+{d}_{i}}\right) \,> \, 1$$. Applying $${f}_{i}=\frac{\varepsilon {e}_{i}{v}_{i}}{{d}_{A}}$$ and $${s}_{i}=\frac{{m}_{i}{h}_{i}}{{{m}_{i}{h}_{i}+d}_{i}}$$, where fi is insect lifetime fecundity and si is the larval success (probability of larvae maturing rather than dying), yields Eq. (1).

### Interaction transition boundary in a one-plant species community

For the interaction to transition from antagonism to mutualism, equilibrium plant density, Pi* must exceed one (see “Model scaling”). Setting Eq. (9b) to zero and solving for Pi* yields:$${{P}_{i}}^{\ast }=\frac{{{m}_{i}{h}_{i}+d}_{i}}{\varepsilon {e}_{i}{v}_{i}}\left(\frac{{{L}_{i}}^{\ast }}{{A}^{\ast }}\right)$$. Setting Eq. (9c) to zero and rearranging terms then yields: $$\frac{{{L}_{i}}^{\ast }}{{A}^{\ast }}=\frac{{d}_{A}}{{m}_{i}{h}_{i}}$$. Thus, $${{P}_{i}}^{\ast }=\frac{{{m}_{i}{h}_{i}+d}_{i}}{\varepsilon {e}_{i}{v}_{i}}\left(\frac{{d}_{A}}{{m}_{i}{h}_{i}}\right)$$ and (rearranging slightly) the condition for mutualism to arise is: $${{P}_{i}}^{\ast }=\left(\frac{{d}_{A}}{\varepsilon {e}_{i}{v}_{i}}\right)\left(\frac{{{m}_{i}{h}_{i}+d}_{i}}{{m}_{i}{h}_{i}}\right) \,> \, 1$$. Rearranging and applying $${f}_{i}=\frac{\varepsilon {e}_{i}{v}_{i}}{{d}_{A}}$$ and $${s}_{i}=\frac{{m}_{i}{h}_{i}}{{{m}_{i}{h}_{i}+d}_{i}}$$ yields Eq. (2).

### Interaction breakdown boundary in a one-plant species community

In the ancestral interaction, insect persistence is evaluated by whether or not it can increase from low density, which yields Eq. (1). Within the net mutualistic region, however, the insect cannot increase from very low density because it cannot buoy plant density sufficiently to maintain a positive per capita growth rate (mathematically, Eq. 1 cannot hold when Eq. 2 is satisfied). The mutualistic region is thus characterized by bistability (see Supplementary Figure 1), and the interaction breakdown boundary is determined by the conditions for the coexistence equilibrium to exist. At the coexistence equilibrium, the larval and adult densities are: $${{L}_{i}}^{\ast }=\frac{-B+\sqrt{{B}^{2}-4{A}_{L}{C}_{L}}}{2{A}_{L}}$$ and $${A}^{\ast }=\frac{-B+\sqrt{{B}^{2}-4{A}_{A}{C}_{A}}}{{2A}_{A}}$$, where $${A}_{L}=\varepsilon {e}_{i}{{v}_{i}}^{2}{h}_{i}{d}_{A}$$, $${A}_{A}=\varepsilon {e}_{i}{{v}_{i}}^{2}{m}_{i}{{h}_{i}}^{2}$$, $$B=\varepsilon {e}_{i}{{v}_{i}}^{2}{m}_{i}{h}_{i}\left(\frac{1}{{f}_{i}{s}_{i}}+\frac{H}{{v}_{i}{m}_{i}}-\left(1+{b}_{i}\right)\right)$$, $${C}_{L}=\varepsilon {e}_{i}{v}_{i}H{d}_{A}\left(\frac{1}{{f}_{i}{s}_{i}}-1\right)$$, and $${C}_{A}=\varepsilon {e}_{i}{v}_{i}{m}_{i}{h}_{i}H\left(\frac{1}{{f}_{i}{s}_{i}}-1\right)$$. For the coexistence equilibrium to exist, either CL and CA must be negative or B must be negative and Li* and A* must be real. CL and CA are negative when fi si > 1, which is Eq. (1) and cannot hold within the mutualistic region because Eq. (2) must be satisfied. However, B is negative when $${f}_{i}{s}_{i}\left(\left(1+{b}_{i}\right)-\frac{H}{{v}_{i}{m}_{i}}\right) \,> \, 1$$, which is approximated by Eq. (3) when the last term is assumed to be small. For Li* and A* to be real, B2 – 4ALCL > 0 and B2 – 4AACA > 0. Assuming that the pollination saturation constant is small (i.e., H ≈ 0) yields CL ≈ CA ≈ 0 such that $${{L}_{i}}^{\ast }\approx \frac{-B}{{A}_{L}}\approx \frac{{m}_{i}}{{d}_{A}{f}_{i}{s}_{i}}\left({f}_{i}{s}_{i}\left(1+{b}_{i}\right)-1\right)$$ and $${A}^{\ast }\approx \frac{-B}{{A}_{A}}\approx \frac{1}{{h}_{i}}\left({f}_{i}{s}_{i}\left(1+{b}_{i}\right)-1\right)$$, which are both positive when fi si (1 + bi) > 1 as approximated by Eq. (3).

### Interaction transition and breakdown boundaries in a two-plant species community

These boundaries are analytically intractable and are estimated by simulation (see codes provided online).

### Coevolutionary dynamics of plants and insect

The effects of plant traits xi and insect traits yi on the ecological dynamics of the interactions are given by:

$$\frac{1}{{P}_{i}}\frac{d{P}_{i}}{{dt}}=\left(1-{P}_{i}\right)+\frac{b\left[{x}_{i}^{B}\right]v\left[{x}_{i}^{V},{y}_{i}^{V}\right]A}{H+v\left[{x}_{i}^{V},{y}_{i}^{V}\right]A}-h\left[{x}_{i}^{H},{y}_{i}^{H}\right]{L}_{i}$$
(10a)
$$\frac{d{L}_{i}}{{dt}}=\varepsilon e\left[{y}_{i}^{V},{y}_{i}^{H}\right]v\left[{x}_{i}^{V},{y}_{i}^{V}\right]{P}_{i}A-{m}_{i}h\left[{x}_{i}^{H},{y}_{i}^{H}\right]{L}_{i}-{d}_{i}{L}_{i}$$
(10b)
$$\frac{{dA}}{{dt}}=\mathop{\sum}\limits_{i}{m}_{i}h\left[{x}_{i}^{H},{y}_{i}^{H}\right]{L}_{i}-{d}_{A}A$$
(10c)

We model coevolution of plant-insect interactions using the adaptive dynamics framework17,18 to link population dynamics and trait coevolution. The coevolution of mutant plant trait xmut and insect trait ymut (for general traits x and y) is given by Equation (7). We model the coevolution of pollination benefits from the antagonist, bi (via mutant plant trait xiB,mut), attraction (via mutant plant trait xiV,mut and mutant insect trait yiV,mut), and defense (via mutant plant trait xiH,mut and mutant insect trait yiH,mut). The invasion fitness of a mutant plant is given by:

$${W}_{P,i}\left({x}_{i}^{{mut}},{x}_{i},{y}_{i}\right)=\left(1-q\left[{x}_{i}^{{mut}},{x}_{i}\right]{{P}_{i}}^{\ast }\right)+\frac{b\left[{x}_{i}^{B,{mut}}\right]v\left[{x}_{i}^{V,{mut}},{y}_{i}^{V}\right]{A}^{\ast }}{H+v\left[{x}_{i}^{V,{mut}},{y}_{i}^{V}\right]{A}^{\ast }}-h\left[{x}_{i}^{H,{mut}},{y}_{i}^{H}\right]{{L}_{i}}^{\ast }$$
(11a)

where Pi*, Li*, and A* are the densities of the plant, insect larvae per plant, and insect adults, respectively, at the ecological equilibrium (of Eq. 10). The functions $$b[x_{i}^{B,mut}], v[x_{i}^{V,mut} , y_{i}^{V}],$$ and $$h[x_{i}^{H,mut}, y_{i}^{H}]$$, describe the effects of mutant plant traits $$x_{i}^{B,mut}$$, $$x_{i}^{V,mut}$$, $$x_{i}^{H,mut}$$, and $$x_{i}^{H,mut}$$ on pollination benefits, attraction, and defense, respectively, which are defined by Eq. (6a–c), where xi is replaced with ximut (where the plant is the host species and the insect is the partner species). The function q[x,mut, xi] defines the trade-offs between mutant plant traits and the competitive ability of mutant plants, which is given by Eq. (8c) (with si = 1). At a coESS, ximut = xi for all traits such that q[ximut, xi] = 1 and the original definition of Pi >1 indicating that pollination benefits exceed herbivory costs is retained when pollination benefits evolve.

Invasion fitness of a mutant insect is given by the dominant eigenvalue of its system of equations evaluated at the resident equilibrium. In a one-plant species community, the insect invasion fitness is:

$${W}_{I,i}=\frac{1}{2}\left(-{d}_{A}-{d}_{i}-{m}_{i}{h}_{i}^{{mut}}+\sqrt{{\left({d}_{A}+{d}_{i}+{m}_{i}{h}_{i}^{{mut}}\right)}^{2}-4\left({d}_{A}\left({d}_{i}+{m}_{i}{h}_{i}^{{mut}}\right)-\frac{\varepsilon {e}_{i}^{{mut}}{v}_{i}^{{mut}}{h}_{i}^{{mut}}{d}_{A}\left({d}_{i}+{m}_{i}{h}_{i}\right)}{{e}_{i}{v}_{i}{h}_{i}}\right)}\right)$$
(11b)

where vimut, himut, and eimut are functions describing the effects of mutant insect traits on attraction, defense, and mutant oviposition efficiency, respectively, which are given by Eq. (6b–d), where yi is replaced with yimut. Invasion fitness of a mutant insect in a two-plant species community is given by the dominant eigenvalue of its system of equations evaluated at the resident equilibrium, which is analytically tractable, but sufficiently complicated that we do not include it here (see codes provided online).

The curves where the selection gradients (see Eqs. 7) become zero give the evolutionary isoclines for the coevolutionary system. The points where the isoclines intersect give the coevolutionary singularities, which are coevolutionary stable states (coESSs) when they are stable for both plants and the insect. For tractability, the local stability of the coevolutionary singularities was assessed by carefully inspecting the selection gradient of each trait in the neighborhood of its coESS with all other traits held at their coESS as well as by simulating coevolutionary dynamics. Importantly, all three plant traits (xiB, xiV, and xiH) and both insect traits (yiV and yiH) all coevolve simultaneously in the model.

### Coevolution of the ancestral antagonistic interaction

In the ancestral interaction, pollination by the antagonist is impossible (bi = 0) and thus visitation only contributes to oviposition. From the plant perspective, the selection gradients for attraction and defense in the ancestral interaction are given by:

$${\left.\frac{\partial {W}_{P,i}}{\partial {x}_{i}^{V,{mut}}}\right|}_{{x}_{i}^{{mut}}={x}_{i}}=-{c}_{P,i}^{V}{{P}_{i}}^{\ast }$$
(12a)
$${\left.\frac{\partial {W}_{P,i}}{\partial {x}_{i}^{H,{mut}}}\right|}_{{x}_{i}^{{mut}}={x}_{i}}=\frac{{h}_{{\max },i}{H}_{i}^{{\prime} }{e}^{{H}_{i}^{{\prime} }\left({x}_{i}^{H}-{y}_{i}^{H}\right)}}{{\left(1+{e}^{{H}_{i}^{{\prime} }\left({x}_{i}^{H}-{y}_{i}^{H}\right)}\right)}^{2}}{{L}_{i}}^{\ast }-{c}_{P,i}^{H}{{P}_{i}}^{\ast }$$
(12b)

Equation (12a) predicts that selection favors plant traits that reduce attracting the antagonist (e.g., reduced production of volatiles) and lower costs associated with competitive ability. We constrain xiV to be non-negative in the ancestral interaction so that xiV = 0 at the coESS; otherwise, xiV → –∞ and the plant always purges the insect given this model parameterization. Selection balances reduced herbivory damage (first term of Eq. 12b) with costs of reduced competitive ability (second term of Eq. 12b). Selection gradients for insect traits are sufficiently complicated that we do not include them here (see codes provided online); however, selection balances traits that increase visitation and overcome plant defenses with the costs associated with reduced oviposition. The ancestral coESSs are given in Supplementary Table 3.

### Coevolution of pollination benefits, attraction, and defense

The evolution of mutant plant traits that allow the antagonist to pollinate it (bimut > 0) initiates the evolution of pollination benefits from the antagonist. The selection gradient for pollination benefits from the antagonist is given by:

$${\left.\frac{\partial {W}_{P,i}}{\partial {x}_{i}^{B,{mut}}}\right|}_{{x}_{i}^{{mut}}={x}_{i}}=\frac{2{b}_{{\max },i}{B}_{i}^{{\prime} }{e}^{-{B}_{i}^{{\prime} }{x}_{i}^{B}}v\left[{x}_{i}^{V},{y}_{i}^{V}\right]{A}^{\ast }}{{\left(1+{e}^{-{B}_{i}^{{\prime} }{x}_{i}^{B}}\right)}^{2}\left(H+v\left[{x}_{i}^{V},{y}_{i}^{V}\right]{A}^{\ast }\right)}-{c}_{P,i}^{B}{{P}_{i}}^{\ast }$$
(13a)

Equation (13a) shows that plants evolve traits to benefit from floral visits by the antagonist when selection for increased pollination benefits (first term of Eq. 13a) exceeds the costs associated with reduced competitive ability (second term of Eq. 13a).

In the model, pollination benefits from the antagonist evolve via Eq. (13a) simultaneously with plant and insect traits affecting attraction and defense. The plant selection gradient for attraction is now:

$${\left.\frac{\partial {W}_{P,i}}{\partial {x}_{i}^{V,{mut}}}\right|}_{{x}_{i}^{{mut}}={x}_{i}}=\frac{b\left[{x}_{i}^{B}\right]{v}_{{\max },i}{V}_{i}^{{\prime} }{e}^{-{V}_{i}^{{\prime} }\left({x}_{i}^{V}+{y}_{i}^{V}\right)}H{A}^{\ast }}{{\left(H\left(1+{e}^{-{V}_{i}^{{\prime} }\left({x}_{i}^{V}+{y}_{i}^{V}\right)}\right)+{v}_{{\max },i}{A}^{\ast }\right)}^{2}}-{c}_{P,i}^{V}{{P}_{i}}^{\ast }$$
(13b)

The co-option of the antagonist has fundamentally changed selection on attraction (Eq. 13b vs. Equation 12a), which now balances traits affecting attraction (first term of Eq. 13b) with the costs of reduced competitive ability (second term of Eq. 13b). Co-option of the antagonist also modifies selection on defense (which is still given by Eq. 12b) by changing both trait values and equilibrium densities.

### Model parameterization

All ecological parameters are estimated from empirical data. Here we parameterize the saturation constant H, maturation efficiency mi, larval mortality di, and adult mortality dA as well as the parameters for the alternative larval host plant and the alternative nectar source (see “Model validation” for other parameters).

We cannot fit the saturation constant H to data because seed set saturates with even a single floral visit. We therefore estimate H as follows: D. wrightii flowers have a 91% chance of setting fruit30; thus, $${v}_{w}A/(H+{v}_{w}A)$$= 0.91 for a single visit ($${v}_{w}$$A = 1). Solving $$1/(H+1)$$= 0.91 for H yields: H = 0.1. H is assumed to be the same for D. discolor as pollination benefits saturate with a single visit for D. discolor. For maturation efficiency mi, only 0.5% of M. sexta larvae on D. wrightii survive through the final larval instar in nature34; thus, mw = 0.005. As M. sexta suffers 40% lower larval survival on D. discolor (5/8 larvae surviving to pupation) than on D. wrightii (10/10 larvae surviving to pupation) in our experiment19, we estimate that maturation efficiency is ~40% lower on D. discolor than on D. wrightii; i.e., md = (1 – 0.4)mw = 0.003. To estimate larval mortality, we note that larval survival is given by: $${m}_{i}={e}^{-{d}_{i}{D}_{i}}$$, where Di is development time. M. sexta has a larval stage of ~20 days on D. wrightii35 and there is no difference in development on D. wrightii and D. discolor, at least to the 5th instar19. Solving for di yields: dw ≈ 0.25 and dd ≈ 0.3. Finally, adults live ~5 days in the wild36. Assuming adult mortality is roughly the inverse of the lifespan: dA ≈ 0.2.

For the alternative larval host plant, females lay similar numbers of eggs on D. wrightii and P. parviflora34; thus, visitation rate and oviposition efficiency are assumed to be the same as with D. wrightii; i.e., vp = vw and ep = ew. Because P. parviflora plants are of similar size and architecture as D. wrightii34, we assume that herbivory rate on P. parviflora is the same as on D. wrightii; i.e., hp = hw. (see “Model validation” for estimates of vw, ew, and hw). Only 1% of M. sexta larvae on P. parviflora survive through the final larval stage34; thus, mp = 0.01. As larvae have roughly the same development time on P. parviflora as on D. wrightii (~20 days37), solving $${m}_{p}={e}^{-{d}_{p}{D}_{p}}$$ yields an estimate of larval mortality on P. parviflora of: dp ≈ 0.25.

For the alternative nectar source, A. palmeri provides M. sexta with copious amounts of nectar that females likely utilize for egg production38. M. sexta females lay 100–300 eggs/night39. If females foraging exclusively on D. wrightii lay the minimum 100 eggs/night and females that also forage at A. palmeri lay the maximum 300 eggs/night, then A. palmerii is estimated to increase oviposition by a factor of: ε = 3.

### Model validation

Pollination benefits (bi), visitation rate (vi), herbivory rate (hi), and oviposition efficiency (ei) all evolve simultaneously in the model. We independently validate the coESSs predicted by the models whenever possible by estimating these parameters using data that were not used to parameterize the models. We estimate bi via the ratio of the seed set of moth-pollinated flowers to autonomously self-pollinated flowers (autonomously self-pollinated seeds germinate as readily as do outcrossed seeds;30). Pollinated D. wrightii and D. discolor flowers set bw = 4.6 ± 0.2 and bd = 3.6 ± 0.1 times more seeds, respectively, than do autonomously self-pollinated flowers (D. wrightii: n = 21 fruit; D. discolor: n = 85 fruit). Moths averaged vw = 4.3 ± 0.6 floral visits to D. wrightii (n = 89 plants) and vd = 2.4 ± 0.4 floral visits to D. discolor (n = 33 plants) in our experiment19. Estimating the herbivory rate is very difficult in nature; however, we can make cursory estimates based on our data. A single M. sexta larvae can consume 1400–1900 cm2 of leaves, which is more than many D. wrightii plants in nature30. Assuming that an average D. wrightii plant supplies larvae with 1400 cm2 of leaves, the variation in leaf consumption (500 cm2) represents ~0.4 plants (=500/1400). Thus, M. sexta larvae are estimated to consume: hw ≈ 1 ± 0.4 D. wrightii plants. M. sexta larvae consumed roughly two times more D. discolor leaf biomass than D. wrightii leaf biomass based on our cursory estimates from our experiments; thus, hd = 2hw ≈ 2 ± 0.8. We estimate oviposition efficiency by the slope of a linear regression of the number of eggs versus the number of floral visits that each plant received from each female moth in our experiments19, which yields: ew = 0.6 ± 0.1 (n = 34 plants) and ed = 0.6 ± 0.2 (n = 24 plants) (Supplementary Data 1).

### Estimating evolutionary model parameters

Directly estimating evolutionary parameters with data is not possible. We therefore use theory to predict how key parameters affect eco-coevolutionary outcomes and to select reasonable parameter estimates. Our approach is as follows. We set the rates of plant and insect evolution to one (μx = μy = 1); these rates affect the speed of evolution, but not the coESSs. For each trait, we need to estimate the maximum value (bmax,i, vmax,i, hmax,i, and emax,i), the coefficient ($${R}_{i}^{{\prime} }$$, $${V}_{i}^{{\prime} }$$, and $${H}_{i}^{{\prime} }$$), and the associated costs (cP,iB, cP,iV, and cP,iH for plant i and cI,iV and cI,iH for the insect). Maximum trait values were chosen to constrain coevolution to a realistic range. We set the coefficients $${R}_{i}^{{\prime} }$$, $${V}_{i}^{{\prime} }$$, and $${H}_{i}^{{\prime} }$$ to one for simplicity because the exact value of any trait x and y are themselves somewhat arbitrary. The costs associated with the traits therefore largely determine the coevolutionary outcomes in the model.

We estimate the costs of each trait by systematically varying the costs of plant traits in the one-plant species community given reasonable values for the insect costs and then systematically varying the costs of insect traits while holding plant costs constant at their chosen values (Fig. 5). Parameter space plots show that the interactions transition from antagonism to net mutualism provided that the costs associated with insect traits underlying visitation (cI,iV) exceed a threshold below which the plant and insect engage in an evolutionary arms-race that results in the evolutionary purging of the antagonist (Fig. 5a, b). Only very rarely does the net antagonism persist. We assigned all insect traits a cost of 0.5 (black points in Fig. 5a, b) and then systematically vary the costs of plant traits associated with attraction and defense.

Parameter space plots show that interactions transition from antagonism to net mutualism when the costs associated with defense are high relative to the costs associated with attraction (cP,iH > cP,iV); otherwise, coevolution drives evolutionary purging of the antagonist (Fig. 5c, d). When the costs associated with attraction and defense are both fairly high, the net antagonism persists. We assigned values of cP,iH and cP,iV to D. wrightii and D. discolor such that the parameters for D. discolor are closer to the threshold at which evolutionary purging occurs than are those of D. wrightii (Fig. 5d vs. 5c), reflecting the smaller range of ecological parameters over which M. sexta can persist with D. discolor versus with D. wrightii (Fig. 2b vs. 2a). Finally, the costs associated with pollination benefits from the antagonist (cP,iB) must be very high for the net antagonism to persist and we never observed evolutionary purging of the insect within the range of values used (see codes provided online). We assigned values of cP,iB so that pollination benefits to D. wrightii and D. discolor are well below their maximum values. Our estimates of evolutionary parameters are reported in Supplementary Table 2. Evolutionary parameters for P. parviflora are set equal to D. discolor because, in the absence of more information, both species are annual plants that may face broadly similar evolutionary constraints, at least relative to the perennial D. wrightii.

### Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

## Data availability

Data used to parameterize and validate the models are available within the paper and its Supplementary Information as well as on Zenodo: https://doi.org/10.5281/zenodo.4628187. Our data was recorded in Microsoft Excel (v. 16.48) and was analyzed in R (v. 3.1.0).

## Code availability

Codes are available on Zenodo: https://doi.org/10.5281/zenodo.4628187. The codes were developed in Mathematica (v. 12.0).

## References

1. Bronstein, J. L. Mutualism. (Oxford University Press, UK, 2015).

2. Bronstein, J. L., Alarcón, R. & Geber, M. The evolution of plant–insect mutualisms. N. Phytol. 172, 412–428 (2006).

3. Weiblen, G. & Treiber, E. Evolutionary origins and diversification of mutualism. in Mutualism (ed. Bronstein, J) 37–56 (Oxford University Press UK, 2015).

4. Kiers, T. E., Palmer, T. M., Ives, A. R., Bruno, J. F. & Bronstein, J. L. Mutualisms in a changing world: an evolutionary perspective. Ecol. Lett. 13, 1459–1474 (2010).

5. Sachs, J. & Simms, E. Pathways to mutualism breakdown. Trends Ecol. Evol. 21, 585–592 (2006).

6. de Mazancourt, C., Loreau, M. & Dieckmann, U. Can the evolution of plant defense lead to plant‐herbivore mutualism? Am. Nat. 158, 109–123 (2001).

7. de Mazancourt, C., Loreau, M. & Dieckmann, U. Understanding mutualism when there is adaptation to the partner. J. Ecol. 93, 305–314 (2005).

8. Yamamura, N. Vertical transmission and evolution of mutualism from parasitism. Theor. Popul. Biol. 44, 95–109 (1993).

9. Yamamura, N. Evolution of mutualistic symbiosis: a differential equation model. Res. Popul. Ecol. 38, 211–218 (1996).

10. Weeks, A., Turelli, M., Harcombe, W., Reynolds, K. & Hoffmann, A. From parasite to mutualist: rapid evolution of Wolbachia in natural populations of Drosophila. PLoS Biol. 5, e114 (2007).

11. Sachs, J., Skophammer, R., Bansal, N. & Stajich, J. Evolutionary origins and diversification of proteobacterial mutualists. Proc. R. Soc. B Biol. Sci. Sci. 281, 20132146 (2014).

12. Sachs, J. & Wilcox, T. A shift to parasitism in the jellyfish symbiont Symbiodinium microadriaticum. Proc. R. Soc. B Biol. Sci. 273, 425–429 (2006).

13. Adler, L. S. & Bronstein, J. L. Attracting antagonists: does floral nectar increase leaf herbivory? Ecology 85, 1519–1526 (2004).

14. Bronstein, J. L., Huxman, T. E. & Davidowitz, G. Plant-mediated effects linking herbivory and pollination. in Ecological Communities (eds. Ohgushi, T., Craig, T. P. & Price, P. W.) 75–103 (Cambridge University Press, 2007).

15. Kessler, D., Diezel, C. & Baldwin, I. Changing pollinators as a means of escaping herbivores. Curr. Biol. 20, 237–242 (2010).

16. Altermatt, F. & Pearse, I. S. Similarity and specialization of the larval versus adult diet of European butterflies and moths. Am. Nat. 178, 372–382 (2011).

17. Metz, J., Nisbet, R. & Geritz, S. How should we define ‘fitness’ for general ecological scenarios? Trends Ecol. Evol. 7, 198–202 (1992).

18. Dieckmann, U. & Law, R. The dynamical theory of coevolution: a derivation from stochastic ecological processes. J. Math. Biol. 34, 579–612 (1996).

19. Smith, G. P., Johnson, C. A., Davidowitz, G. & Bronstein, J. L. Linkages between nectaring and oviposition preferences of Manduca sexta on two co-blooming Datura species in the Sonoran Desert. Ecol. Entomol. 43, 85–92 (2018).

20. Riffell, J. A. et al. Behavioral consequences of innate preferences and olfactory learning in hawkmoth-flower interactions. Proc. Natl Acad. Sci. USA 105, 3404–3409 (2008).

21. Mechaber, W. L. & Hildebrand, J. G. Novel, non-Solanaceous host plant record for Manduca sexta (Lepidoptera: Sphingidae) in the Southwestern United States. Ann. Entomol. Soc. Am. 93, 447–451 (2000).

22. van Baalen, M. & Jansen, V. A. A. Dangerous liaisons: the ecology of private interest and common good. Oikos 95, 211–224 (2001).

23. Revilla, T. A. Numerical responses in resource-based mutualisms: a time scale approach. J. Theor. Biol. 378, 39–46 (2015).

24. Picot, A., Georgelin, E. & Loeuille, N. From antagonistic larvae to mutualistic adults: coevolution of diet niches within life cycles. Oikos 128, 392–404 (2019).

25. Raguso, E. A., Henzel, C., Buchmann, S. L. & Nabhan, G. P. Trumpet flowers of the Sonoran desert: floral biology of Peniocereus Cacti and Sacred Datura. Int. J. Plant Sci. 164, 877–892 (2003).

26. Reisenman, C., Riffell, J., Duffy, K. & Pesque, A. Species-specific effects of herbivory on the oviposition behavior of the moth Manduca sexta. J. Chem. 39, 76–89 (2013).

27. Hare, J. D. & Walling, L. L. Constitutive and jasmonate-inducible traits of Datura wrightii. J. Chem. Ecol. 32, 29–47 (2006).

28. Werner, G. D. et al. Symbiont switching and alternative resource acquisition strategies drive mutualism breakdown. Proc. Natl Acad. Sci. USA 15, 5229–5234 (2018).

29. Chomicki, G. & Renner, S. Partner abundance controls mutualism stability and the pace of morphological change over geologic time. Proc. Natl Acad. Sci. USA 114, 3951–3956 (2017).

30. Bronstein, J. L., Huxman, T., Horvath, B., Farabee, M. & Davidowitz, G. Reproductive biology of Datura wrightii: the benefits of a herbivorous pollinator. Ann. Bot. 103, 1435–1443 (2009).

31. Douglas, A. E. How multi-partner endosymbioses function. Nat. Rev. Microbiol. 14, 731–743 (2016).

32. Nelsen, M., Ree, R. & Moreau, C. Ant–plant interactions evolved through increasing interdependence. Proc. Natl Acad. Sci. USA 115, 12253–12258 (2018).

33. Stadler, B. & Dixon, A. F. G. Ant attendance in aphids: why different degrees of myrmecophily? Ecol. Entomol. 24, 363–369 (1999).

34. Mira, A. & Bernays, E. A. Trade-offs in host use by Manduca sexta: plant characters vs natural enemies. Oikos 97, 387–397 (2002).

35. Potter, K., Davidowitz, G. & Woods, H. A. Cross‐stage consequences of egg temperature in the insect Manduca sexta. Funct. Ecol. 25, 548–556 (2011).

36. Levin, E., Mitra, C. & Davidowitz, G. Fed males increase oviposition in female hawkmoths via non-nutritive direct benefits. Anim. Behav. 12, 111–118 (2016).

37. Diamond, S. E. & Kingsolver, J. G. Fitness consequences of host plant choice: a field experiment. Oikos 119, 542–550 (2010).

38. Alarcón, R., Riffell, J. A., Davidowitz, G., Hildebrand, J. G. & Bronstein, J. L. Sex-dependent variation in the floral preferences of the hawkmoth Manduca sexta. Anim. Behav. 80, 289–296 (2010).

39. Mechaber, W. L., Capaldo, C. T. & Hildebrand, J. G. Behavioral responses of adult female tobacco hornworms, Manduca sexta, to hostplant volatiles change with age and mating status. Journal of Insect Science 2, 1e8 (2002).

40. Alarcón, R., Davidowitz, G. & Bronstein, J. L. Nectar usage in a southern Arizona hawkmoth community. Ecol. Entomol. 33, 503–509 (2008).

41. McFadden, M. W. Observations on feeding and movement of tobacco hornworm larvae. J. Econ. Entomol. 61, 352 (1968).

## Acknowledgements

We thank Julie Johnson (Life Science Studios) for the beautiful illustrations in Fig. 1 and for help designing all of the Figures. We thank Olivia Brinkerhoff, Kimberly Kopplin, Nico Lorenzen, Keane Sullivan, and James Wadsworth for their help in growing plants and rearing larvae as well as Heather Costa for colony maintenance. This work was supported by Science Foundation Arizona grant BSP 0528-13 to C.A.J, U. S. National Science Foundation grant IOS-1053318 to G.D., U. S. National Science Foundation, Dimensions of Biodiversity program (DEB-1831493) to R.F., and the Keep Engaging Youth in Science (KEYS) program of the University of Arizona BIO5 Institute.

## Author information

Authors

### Contributions

C.A.J., G.P.S., G.D. and J.B. designed the experiments; C.A.J. and G.P.S. performed the experiments and analyzed the data; C.A.J. developed the models; C.A.J., K.Y. and R.F. developed methods to analyze the models; and C.A.J. wrote the manuscript with input from all authors. J.B. and R.F. co-advised the project.

### Corresponding author

Correspondence to Christopher A. Johnson.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

Peer review information Nature Communications thanks Ellen Simms, Istvan Scheuring and the other, anonymous, reviewers for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Johnson, C.A., Smith, G.P., Yule, K. et al. Coevolutionary transitions from antagonism to mutualism explained by the Co-Opted Antagonist Hypothesis. Nat Commun 12, 2867 (2021). https://doi.org/10.1038/s41467-021-23177-x

• Accepted:

• Published:

• DOI: https://doi.org/10.1038/s41467-021-23177-x