Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The likelihood of homoploid hybrid speciation


New species may be formed through hybridization and without an increase in ploidy. The challenge is for hybrid derivatives to escape the homogenizing effects of gene flow from parental species. The mechanisms hypothesized to underlie this process were modelled using a computer simulation. The model is of recombinational speciation, in which chromosomal rearrangements between parental species result in poor fertility of F1 hybrids, but through recombination, novel homozygous types are formed that have restored fertility. In simulations, stable populations bearing the recombinant karyotypes originated frequently and were maintained when the fertility of F1 hybrids was high. However, this high rate of origination was offset by low genetic isolation, and lower F1 hybrid fertility increased the evolutionary independence of derived populations. In addition, simulations showed that ecological and spatial isolation were required to achieve substantial reproductive isolation of incipient species. In the model, the opportunity for ecological isolation arose as a result of adaptation to extreme habitats not occupied by parental species, and any form of spatial isolation (e.g. founder events) contributed to genetic isolation. Our results confirmed the importance of the combination of factors that had been emphasized in verbal models and illustrate the trade-off between the frequency at which hybrid species arise and the genetic integrity of incipient species.


Hybridization between taxa can act as a creative force in evolution and lead to cladogenesis and speciation. In addition to the well-known cases of polyploid speciation, hybridization has given rise to new species without an increase in ploidy (homoploid speciation or hybrid speciation, hereafter; examples reviewed in Rieseberg, 1997). Empirical studies of homoploid speciation in plants have described several aspects of the process (Stebbins, 1957; Grant, 1966; Abbott, 1986; Cruzan et al., 1994; Rieseberg et al., 1995b, 1996, 1998; Hodges et al., 1996; Abbott et al., 1998; Ungerer et al., 1998), but the factors that influence its likelihood and dynamics are only beginning to be understood (McCarthy et al., 1995). The purpose of this study was to use computer simulations to investigate further some of the ecological and genetic factors that affect the likelihood of homoploid hybrid speciation. We used an individual-based model in which we tracked the dynamics of multilocus genotypes in spatially separated habitats.

By necessity, hybrid speciation must be initiated while the parental species are in sympatry or parapatry, and in most instances in close proximity. It is this limitation and the resulting population dynamics that constitute the principal challenges to understanding the process of hybrid speciation. Although parental genotypes necessarily serve as the source of hybrid genotypes, they also limit the increase of hybrid genotypes and the development of reproductive isolation if interbreeding among lineages is common (e.g. hybrids backcrossing to parental lineages).

The verbal model for homoploid hybrid speciation hypothesizes two principal mechanisms by which hybrid forms become stabilized (Grant, 1958). The first mechanism is referred to as recombinational speciation and involves hybridization between two parental species that differ by two or more chromosomal rearrangements (Fig. 1; Stebbins, 1957; Grant, 1958). The chromosomal differences reduce fertility in F1 hybrids and in later generation hybrids that are heterozygous for the rearrangements. The reduction in fertility can result from impaired chromosomal pairing in meiosis, or from crossing-over between heterologous chromosomal blocks and a reduction in viable gametes. In either case, the reduction of fertility in F1 hybrids constitutes a barrier to introgression between parental species. However, backcrossing or interbreeding among F1 individuals may give rise to some novel, chromosomally balanced genotypes that have restored fertility. Because crosses with the parental species will produce heterozygous progeny, these novel genotypes are at least partially reproductively isolated from the parental species, and if they can persist as a lineage, may constitute a novel hybrid species.

Figure 1

Schematic representation of the model of recombinational speciation. In this example, the parental species differ by two chromosomal inversions (indicated by shaded chromosomal blocks containing upright or inverted ‘v’). Hybridization produces F1 progeny that are heterozygous for the rearrangements and as a result have low fertility. Mating among the F1 progeny or backcrosses can give rise to partially balanced genotypes (two examples illustrated), as well as novel homozygous genotypes. The novel homozygotes are fertile but at least partially intersterile with the parental species.

Empirical support exists for the applicability of the recombinational model to at least one group containing species of homoploid hybrid origin. Within Helianthus sunflowers, comparative linkage maps of two parental species and a derived hybrid species indicate that chromosomal rearrangements between parental species are significant barriers to introgression, and that the genome of the hybrid species is based on a combination of structural features of parental chromosomes (Rieseberg et al., 1995a,b).

The second mechanism that may lead to the stabilization of hybrid lineages is ‘the segregation of a new type isolated by external barriers’ (Grant, 1981; p. 243). The external barriers are thought to result from ecological selection and niche differentiation among lineages. Several authors have emphasized the critical role ecological barriers are likely to play in hybrid speciation (Kerner 1894–1895; Anderson, 1948; Grant, 1981; Templeton, 1981; Rieseberg, 1997). In particular, it is thought that hybrid species are able to form in sympatry with parental species in part because of the ecological opportunity created by available habitat that can not be occupied by either of the parental species. The occupancy of divergent niches by hybrid genotypes may give rise to ecological isolation, the efficacy of which is a function of the strength of natural selection against parental genotypes in the divergent habitat. Support for the importance of this mechanism comes from the observation that all of the known examples of homoploid hybrid species occur in habitats that differ from those occupied by their parental species (Rieseberg, 1997). Often these are extreme habitats relative to the requirements of parents, rather than habitats with characteristics intermediate between the parental niches. A good example of this is Helianthus anomalus, a hybrid species that occupies sand dunes and swales, and that exists in conditions that parental species can not tolerate (Heiser et al., 1969).

Grant’s hypothesized mechanisms for hybrid speciation are not mutually exclusive and both are likely to have contributed to the origin of most of the documented homoploid hybrid species (Rieseberg, 1997). In the model we describe below, we examine both mechanisms, along with the effect of geographical isolation, an additional factor that will affect the rate of gene flow between parents and derived hybrid lineages (Charlesworth, 1995; Rieseberg, 1997). The possibility for recombinational speciation arises in our model by assuming that parental species differ for two or more chromosomal rearrangements. We have incorporated habitat-specific ecological performance and consider cases in which hybrid species occupy a habitat in which parents perform poorly. Finally, although hybrid speciation must be initiated in sympatry, the model includes spatial separation of habitats as a variable, because it may confer critical geographical isolation between the hybrid population and the parental taxa.

An earlier simulation study of recombinational speciation by McCarthy et al. (1995) developed a framework for modelling homoploid speciation that we utilize extensively. Their work not only led to important conclusions about the evolutionary conditions that might favour this mode of speciation, but also predicted that new ‘recombinational species’ are likely to arise rapidly in a punctuated manner. However, important aspects of homoploid hybrid speciation remain unexplored, and the earlier results (McCarthy et al., 1995) suggested that our understanding of the dynamics of hybrid speciation would benefit from further study. More specifically, in the McCarthy et al. (1995) simulations two outcomes occurred: a favoured hybrid genotype eventually spread throughout the spatial grid and supplanted parental species (a type of recombinational speciation), or alternatively, the favoured hybrid genotype never increased to constitute a significant proportion (≈10%) of the total number of individuals. Consequently, the McCarthy et al. (1995) model described conditions that favour a special case of recombinational speciation, one in which a novel species is generated but both parental species become locally extinct. This type of speciation is known to have occurred through the complete genetic merger of two Raphanus species (cultivated radish, R. sativus, and jointed charlock, R. raphanistrum) in California (N. C. Ellstrand, pers. comm.). In the other documented cases of homoploid hybrid speciation, parental species continue to coexist with the derived hybrid species (Rieseberg, 1997). The conditions that favour hybrid speciation and the maintenance of parental species are likely to differ from those that lead to hybrid speciation through the merger of species’ genomes.

A number of the differences between the current model and that of McCarthy et al. (1995) were motivated by the goal of describing the conditions under which novel hybrid species can arise and coexist with parental species. For example, we employed a spatial model with an initially unoccupied and ecologically divergent habitat and that allowed for ecological differentiation and isolation of species, whereas the previous model was of a uniform ecological environment. Similarly, in the present model we have added spatial isolation between species. In addition, our model included independent genetic factors for viability and fertility, whereas both were based on a single set in the previous model. A final refinement was that we monitored a set of marker loci to quantify the amount of genetic isolation between parental and hybrid species.

Details of computer simulation

To investigate the effects of postulated intrinsic and extrinsic factors on the likelihood of hybrid speciation, we focused on the contribution of hybrid fertility barriers, ecological selection, and geographical isolation to the speciation process. We used a computer simulation to quantify the frequency of hybrid speciation, as well as the genetic integrity of derived lineages, across a range of parameter values for the strength of fertility and ecological selection, and the degree of geographical isolation between populations. The model is based on hybridization between hermaphroditic, annual plants, but its results and conclusions should apply to other systems as well.

Spatial grid

The frequencies of multilocus genotypes were tracked on a spatial grid that simulated the interface between hybridizing parental species. The spatial grid was 20 columns by 80 rows in size and each cell on the grid had a carrying capacity of five individuals. The grid was divided into three habitat types. Rows at the ends of the grid represented parental habitats (i.e. areas in which parental genotypes have high survivorship). Between parental habitats was a zone in which intermediate hybrid genotypes survived at a higher rate than parental types; this zone is referred to as hybrid habitat. In addition, these habitats were geographically isolated by ‘gaps’ (2–8 rows) in which plants could not survive, regardless of genotype. Parental habitats (29 rows each) and the distance between them (22 rows) were fixed in size, whereas the size of the hybrid habitat could be varied (6–18 rows) to accommodate different sized gaps separating the habitats. To minimize artifacts with respect to dispersal (e.g. edge effects), the grid was assumed to be an elongated cylinder, such that the long edges of the grid were connected and ends of the grid were distant from the hybrid zone. As a result, the grid simulated a transect through a more extensive hybrid zone.

Genetics and selection

We employed a simple genetic model for fertility in which we assumed that the two parental species differed by two chromosomal inversions (inversion haplotypes) that reduced fertility in heterozygous individuals. Four haplotypes would need to be tracked to simulate two chromosomal translocations as the basis for hybrid speciation. The number of progeny produced by individuals varied according to fertility genotype (i.e. karyotype) and was distributed as a Poisson variable with parameter λ, where λ=4rn/2, r is the fertility of the F1 hybrid relative to the parental species, n is the number of heterozygous inversion haplotypes within the genotype, and 2 is the number of chromosomal rearrangements by which the initial species differed. Thus, individuals with parental (a/a a/a, b/b b/b) and novel homozygous fertility genotypes (a/a b/b and b/b a/a) had full fertility, and completely heterozygous individuals produced a fraction (r) of the offspring produced by homozygous types. This is equivalent to a two-locus model of heterozygote inferiority with multiplicative fitness effects across loci. Similarly, pollen viability was a function of the pollen donor’s fertility genotype with p(viable pollen)=rn/2.

In our model ecological performance is a function of two unlinked genetic loci that segregate independently of the inversion haplotypes, and of the habitat occupied by a plant. We assumed that the loci underlying ecological performance interacted additively and that the parental species were initially fixed for alternative alleles (e and f, with effects −1 and 1, respectively). This resulted in a matrix of genotypic values that included two novel homozygous genotypes (e/e f/f and f/f e/e) that can be formed by hybridization: (See the Table 1 below)

Table 1

This two-locus, two-allele model is the simplest we could conceive; a single locus, with parents fixed for alternative alleles, would be insufficient because hybrid lineages would be heterozygous and could not breed true.

Selection on ecological performance occurred at the seedling stage and was habitat-specific. Parents were adapted to ecological conditions present in their respective habitats. Ecological selection in the hybrid habitat favoured the intermediate genotypic value that is found in the two novel homozygotes and the double heterozygote at the two ecological loci. A genotype’s probability of survival in a given habitat was a function of the strength of selection (s, the probability of survival for the least fit genotype) and the genotype’s deviation (d ) from the optimum genotypic value in the habitat relative to the largest possible deviation in that habitat (max d ): probability of survival=1 − s (d/max d ).

Thus, members of the parental species had a 1 − s probability of survival in the habitat of the other species, whereas potential hybrid species survived at a 1 − s(0.5) rate outside the hybrid habitat.

The intermediate genotypic value favoured in the hybrid habitat does not necessarily imply intermediate phenotypic expression. Instead, intermediate genotypic values might be permissive of extreme phenotypes, such as tolerance of or compensation for extreme or stressful environmental conditions.

Outline of simulation

At the beginning of the simulation there were no plants in the hybrid zone and the parental habitats were filled to carrying capacity. In each generation the following cycle was repeated. The number of progeny produced by each plant as a seed parent was determined. For each ovule, a pollen donor was chosen from plants that lay a random number of cells from the seed parent, where the distance was given by a geometric probability distribution with parameter 0.4. If the pollen donor produced viable pollen, a single progeny was produced and placed on the grid after seed dispersal (successive donors were chosen until viable pollen was produced); the distance of seed dispersal was given by a geometric distribution with parameter 0.6. These geometric distributions match the leptokurtic patterns of pollen and seed dispersal observed in nature (Loveless & Hamrick, 1984). To avoid pollen limitation, the progeny allowance of only the maternal parent was decremented. Ecological selection occurred at this stage and the probability that the seedling survived varied according to the ecological selection function described above. Mating continued until the progeny allowance of each plant was exhausted. Finally, at the end of each generation the number of individuals per cell was reduced randomly to a maximum of five (population regulation) and the cycle was repeated.

The simulation proceeded until (i) plants bearing one of the homozygous fertility types occupied the hybrid zone to at least 50% capacity, or (ii) 4000 generations passed without any genotype becoming numerically dominant in the hybrid zone. In replicates in which a fertility type successfully colonized the hybrid habitat, we continued the simulation to monitor its persistence. When a population of individuals with a parental fertility genotype occupied the hybrid zone, those individuals always contained a recombinant, adapted ecological genotype. We categorized this outcome as ecological introgression if the population persisted for an additional 100 generations. When one of the possible hybrid fertility genotypes increased to 50% of the capacity of the hybrid zone, this population was recognized as a potential hybrid species and its fate was tracked for an additional 1000 generations. If the potential hybrid species decreased to be less common than another fertility type in the hybrid zone, we concluded that the hybrid species failed to persist and terminated that replicate of the simulation.

For replicates in which recombinational species arose, we quantified genetic isolation of the species by calculating pairwise GST values (Nei, 1973) for three different portions of the genome. Separate GST values were based on marker loci completely linked to the two ecological loci and to the two inversion haplotypes, and on two unlinked markers that segregated independently of all loci under selection. In the generation in which a potential hybrid species became numerically dominant in the hybrid zone, all individuals bearing its fertility type and both parental species were marked with diagnostic alleles at the marker loci. Thus, GST between the species was initially maximal and we could monitor its decay.

The program for this simulation was written in the C computer language. Source code is available from the first author upon request.


Effects of hybrid fertility and ecological selection

The frequency of simulation replicates ending in hybrid speciation increased with greater hybrid fertility (r) and stronger ecological selection (s; Fig. 2). For hybrid speciation to occur at an appreciable frequency in our model, relatively intense ecological selection was required. In the majority of cases where hybrid speciation did not occur, the simulation ended in ecological introgression instead, and this type of introgression was by far the most frequent outcome of hybridization. In simulations with the most extreme ecological selection (s=0.9), some replicates ended after 4000 generations without ecological introgression or hybrid speciation.

Figure 2

Proportion of simulation replicates ending in ecological introgression (light lines) or hybrid speciation (heavy lines). Fifty replicates were run at each combination of hybrid fertility (r) and ecological selection (s) parameters. Geographical isolation between habitats was held constant at eight grid rows on each side of the hybrid habitat. At s=0.9 some replicates ended after 4000 generations without ecological introgression or hybrid speciation.

Over the range of hybrid fertility and ecological selection parameters examined (0.1 ≤ r, s ≤ 0.9), we saw no instances of hybrid species declining significantly after becoming numerically dominant; coefficients of variation (SD/mean × 100%) in population size over a period of 1000 generations were on average below 8%. However, although hybrid species lineages arose and persisted for 1000 generations, observations of pairwise GST values suggested significant exchange of alleles between these populations at some parameter values (Fig. 3). With strong ecological selection and low hybrid fertility, GST values decayed at a slower rate and were higher after 1000 generations than in replicates with weaker selection. In other words, the genomes of hybrid species remained more isolated from parental species when protected by stronger ecological and fertility barriers. The decay in GST was similar for markers linked to ecological loci and inversion haplotypes, whereas values for markers that were not linked to loci under selection declined more rapidly and to lower values after 1000 generations.

Figure 3

Decay of GST between the derived hybrid species and one of the parental species over 1000 generations. Separate GST values were calculated for markers linked to each of the inversion haplotypes and ecological loci, and for markers unlinked to loci under selection (neutral loci). At each combination of ecological selection (s) and hybrid fertility (r) parameters shown, 50 replicate simulations were run (size of gap between parental and hybrid habitats was eight rows). No hybrid speciation occurred at s=0.1 and 0.3, and at parameter combinations with empty panels. Mean GST after 1000 generations is printed in the lower left corner of each panel.

Effect of geographical isolation

Because hybrid speciation occurred at an appreciable frequency only when ecological selection was relatively strong (s ≥ 0.7), we examined the interaction of ecological and fertility selection with geographical isolation for a subset of parameter combinations (see Figs 4 and 5). Geographical isolation did not have a uniform effect on the number of simulation replicates ending in hybrid speciation and ecological introgression (Fig. 4). High F1 hybrid fertility (r=0.9, s=0.9) led to a high likelihood of hybrid species formation that declined with increasing geographical isolation. The frequency of hybrid species formation was less affected by geographical isolation at lower values for hybrid fertility (Fig. 4).

Figure 4

Proportion of simulation replicates ending in ecological introgression (thin lines) or hybrid speciation (heavy lines). Fifty replicates were run at each combination of geographical isolation (number of rows between parental and hybrid habitats) with four pairs of fertility (r) and ecological (s) parameters.

Figure 5

Decay of GST between the derived hybrid species and one of the parental species over 1000 generations. Separate GST values were calculated for markers linked to each of the inversion haplotypes and ecological loci, and for markers unlinked to loci under selection (neutral loci). At each of eight combinations of geographical isolation and hybrid fertility (r) parameters, 50 replicate simulations were run (ecological selection parameter fixed at s=0.9). Mean GST after 1000 generations is printed in the lower left corner of each panel.

The effect of geographical isolation on gene flow between species and on the decay of GST was more drastic and significant (Fig. 5). Little geographical isolation (gap size = 2) in combination with strong fertility selection (r = 0.1, s = 0.9) could not prevent significant decay of GST, and coupled with weak fertility selection (r = 0.9, s = 0.9) resulted in a surprising amount of gene flow between genotypically differentiated, stable populations.


By simulating a realistic model for hybridization between differentiated taxa, we have shown that previously hypothesized processes may lead to homoploid hybrid speciation without the local extinction of parental species. Our results indicate that the recombinational speciation model, and the resulting fertility selection, are very unlikely to lead to hybrid speciation without additional isolating mechanisms. In combination with strong ecological and geographical barriers to gene flow between species, hybrid speciation may be a relatively frequent outcome of hybridization, equally likely as the alternative, ecological introgression (Figs 2 and 4). However, with such strong barriers to dispersal, successful hybridization is itself rare.

Under the conditions that most frequently lead to the formation of a population bearing a novel homozygous combination of inversion haplotypes (i.e. a potential hybrid species), the derived populations experience extensive gene flow with parental species and are unlikely to be lineages that may continue to diverge over evolutionary time. In such cases, the decay of GST values for initially diagnostic markers is striking evidence of the homogenizing effects of hybridization, whereas the maintenance of karyotypically differentiated populations gives the superficial and misleading appearance of genetic isolation. Variability in the permeability of portions of the genome subject to different levels of selection highlights the difficulty of delineating species when hybridization is common (Harrison, 1990).

The underlying processes

As in McCarthy et al. (1995), increasing F1 fertility increases the frequency at which hybrid species originate (Figs 2 and 4). This is reasonable because a decrease in the fertility barrier between parents makes it more likely that parental fertility types will introgress and form novel fertility types. Because of differences in the models, the effect of F1 fertility is the primary comparison that can be made between the results of McCarthy et al. (1995) and those described here. In McCarthy et al. (1995), a weak fertility barrier rapidly led to the merger and extinction of parental species, and persistent genetic isolation from parental species was not an issue. In contrast, if parental species persist locally, hybrid species should not persist if they are not well isolated. Under the conditions that led to hybrid speciation in the present model, populations of individuals with the novel hybrid genotypes always remained in the hybrid zone for at least 1000 generations after their appearance. However, under conditions most favourable to the origin of hybrid species, our observations of GST revealed a considerable exchange of alleles at all marker loci as a result of interspecific gene flow (Fig. 3). Results under the most favourable conditions indicate that although stable populations of derived hybrid genotypes might persist in the hybrid zone, these can not be viewed as independent lineages, or species. Instead these stable populations are a component of a step cline in allele or haplotype frequencies at loci under selection, and individuals bearing the dominant genotype are regularly the result of recurrent hybridization rather than being identical by descent. They are not genetically isolated from parental species and only evolve to alternative allele frequencies at loci that are subject to selection that varies among habitats.

Strong ecological selection also increases the frequency at which hybrid species form. When ecological selection is weak, individuals with the parental karyotypes are able to survive in the hybrid zone at a relatively high rate, and as a result, introgression at ecological loci is the common outcome under these conditions. It is only when ecological selection is relatively strong that ecological introgression is less likely and hybrid speciation increases in frequency (Fig. 2). In addition, the genetic integrity of incipient hybrid species is increased by stronger ecological selection (Fig. 3). As would be expected, divergent ecological selection among habitats creates a neighbourhood of genetically similar individuals and acts as an isolating barrier.

The geographical context of hybrid speciation typically has been assumed to be sympatric, because of the necessity of hybridization between progenitor species. However, as was observed in this study, any increase in the rate of hybridization in geographical proximity should be offset by the diluting effects of gene flow with parental species (Fig. 5). As a result, it has been noted that hybridization is more likely to lead to speciation when hybrids are geographically isolated from parents, and, in the extreme, when hybrid founder events occur. Greater distance between habitats did have the anticipated effect of genetically isolating hybrid populations from the parental species, whereas less geographical isolation resulted in a stable step cline and with negligible opportunity for divergence, given the low identity by descent of alleles (Fig. 5). In our model, 1.01% of pollen and 0.03% of seeds were dispersed a great enough distance to cross a gap of eight rows on the grid. This amount of geographical isolation already had a significant effect on the genetic integrity of hybrid species, and with sufficiently strong fertility and ecological selection, more extreme isolation (e.g. founder events) would not be a necessary precondition for strong genetic isolation.

Because we defined the origin of new hybrid species in terms of the fertility genotype (i.e. karyotype) there was the potential for hybrid species to differ from parental species only at inversion haplotypes. However, hybrid speciation via recombination at both inversion haplotypes and ecological loci was far more common than via recombination at inversion haplotypes alone. This is probably because substantial ecological selection was necessary for hybrid speciation to occur and as a result multilocus associations usually built up between inversion haplotypes and ecological loci.

Assumptions and alternatives

Several components of our model require discussion of the underlying assumptions. The first of these is our treatment of ecological performance, in which hybrids outperform parental species in the previously unoccupied habitat between parental habitats. Empirical evidence exists to support the assumption of habitat-specific performance (Lewontin & Birch, 1966; Abbott & Horrill, 1991; Grant & Grant, 1994; Graham et al., 1995; Arnold, 1997; Wang et al., 1997; Burke et al., 1998; Schluter, 1998), and the possibility of hybrids outperforming parents (review in Arnold & Hodges, 1995). Also, verbal models of hybrid speciation have typically considered hybrid habitats that are extreme relative to those occupied by the progenitor species (Grant, 1981; Templeton, 1981; Rieseberg, 1997). Our model is consistent with this if one assumes that the intermediate ecological genotypes of hybrid species make possible the development of extreme phenotypes, whereas parental genotypes are relatively costly (e.g. developmentally and physiologically) and constrain ecological phenotypes. In addition, using a more complex additive genotypic model, we have considered transgressive segregation, in which some hybrid genotypes have genotypic values that lie outside the range exhibited by parental genotypes (deVicente & Tanksley, 1993). This is perhaps a more realistic genetic model, because the production of extreme phenotypes through transgressive segregation is a common outcome of hybridization between species (Rieseberg & Ellstrand, 1993). This alternative model yields results equivalent to those presented here (data not shown).

With respect to ecological selection, we chose a simple additive genetic model for fitness, yet it is very possible that ecological performance and fitness do not decline linearly as a function of deviation from the optimum. Alternative models for the decline in fitness would change the steepness of adaptive peaks in the different habitats and directly affect the amount of isolation resulting from ecological selection. Similarly, plants experienced ecological selection only in terms of variation in survivorship at the seedling stage, whereas ecological factors presumably could also affect fecundity. Additional episodes of similar selection would only increase divergence among habitats and should thus make hybrid speciation more likely.

In our model, individuals that are heterozygous for chromosomal inversion haplotypes have reduced fertility, and homozygotes for the inversions have restored fertility. This assumption is well justified because chromosomal inversions are known to have contributed to isolation in at least one well documented example of hybrid speciation (Rieseberg et al., 1995a,b). Moreover, variation in F1 hybrid fertility among species pairs is expected to be based upon variation in the size of chromosomal rearrangements and recombination rates within rearrangements.

Nonetheless, the question of whether a genic sterility barrier (i.e. one not involving chromosomal rearrangements) might make hybrid speciation more or less likely is of interest given the importance of genic sterility to isolation between many plant and animal species. One applicable model of hybrid sterility is the Muller–Dobzhansky model of complementary genes (Muller, 1942; Dobzhansky, 1951). In this model, isolated populations of an ancestral species (a/a b/b) diverge to alternative alleles (species 1: A/A b/b; species 2: a/a B/B). The derived species are reproductively isolated because their novel alleles (A and B) are incompatible and interact negatively in interspecific hybrids (e.g. F1 hybrid: A/a B/b). As with chromosomal rearrangements, the recovery of fully fertile, recombinant genotypes is straightforward if the number of genic sterility factors is small. For example, with a two-locus model, the recovery of the ancestral genotype (a/a b/b) results in full fertility. However, this genotype will be completely interfertile with both parental genotypes. This is one of several difficulties with a hybrid speciation model that is mediated by the sorting of genic sterility factors. In comparison to the chromosomal model: (i) more genic sterility factors are required to generate a recombinant genotype that is intersterile with both parents (four or more loci); and (ii) a much smaller number of recombinant genotypes will be both fertile and genetically well isolated from the parental species. Consequently, purely genic models for hybrid speciation will necessarily be more complex than chromosomal models, and hybridization under a purely genic model will more often lead to introgression rather than well isolated hybrid species. Although genic factors probably contribute to sterility barriers among species involved in hybrid speciation, the requirements for a purely genic model are thus relatively stringent.

Grant (1958) noted that recombinational speciation should be more likely under conditions of inbreeding, and this was confirmed by both McCarthy et al. (1995) and the present model (data not shown). However, the species that are thought to have originated through hybrid speciation in nature all have an outcrossing mating system (Rieseberg, 1997). Thus, it was valuable to demonstrate that hybrid speciation may occur under previously hypothesized conditions and without the aid of a selfing mating system.

Three species of sunflowers (Helianthus spp.) are of homoploid hybrid origin and have been studied in detail (e.g. Rieseberg et al., 1995b, 1996; Ungerer et al., 1998). Overall, our model matches observations of hybridization among sunflowers, where F1 hybrid progeny have fertilities less than 10% (Ungerer et al., 1998) and there are significant differences in the habitats occupied by parental and homoploid derivative species (Rieseberg, 1991). The observed result that hybridization more often results in introgression rather than hybrid speciation is consistent with the pattern among sunflowers, where there are only three known hybrid species, but innumerable hybrid swarms where the parental species meet (Heiser, 1947).


The recombinational model can lead to homoploid hybrid speciation under conditions that promote isolation of derived lineages. Simulations showed that hybrid speciation can occur in completely outcrossing species via previously hypothesized mechanisms, and without the local extinction of parental species. In particular, the results confirm the importance of ecological selection and geographical isolation in hybrid speciation that had been suggested in verbal models. Finally, modelling the process demonstrated that there is a trade-off between the rate at which hybrid species originate and the persistence of the genetically isolated, derived species.


  1. Abbott, R. J. (1986). Life history variation associated with polymorphisms for capitulum type and outcrossing rate in Senecio vulgaris L. Heredity. 56: 381–391.

    Article  Google Scholar 

  2. Abbott, R. J. and Horrill, J. C. (1991). Survivorship and fecundity of the radiate and non-radiate morphs of groundsel, Senecio vulgaris L., raised in pure stand and mixture. J Evol Biol. 4: 241–257.

    Article  Google Scholar 

  3. Abbott, R. J., Bretagnolle, F. C. and Thebaud, C. (1998). Evolution of a polymorphism for outcrossing rates in Senecio vulgaris: influence of germination behavior. Evolution. 52: 1593–1601.

    PubMed  Google Scholar 

  4. Anderson, E. (1948). Hybridization of the habitat. Evolution. 2: 1–9.

    Article  Google Scholar 

  5. Arnold, M. L. (1997) Natural Hybridization and Evolution. Oxford University Press, New York.

    Google Scholar 

  6. Arnold, M. L. and Hodges, S. A. (1995). Are natural hybrids fit or unfit relative to their parents? Trends Ecol Evol. 10: 67–71.

    CAS  Article  Google Scholar 

  7. Burke, J. M., Carney, S. E. and Arnold, M. L. (1998). Hybrid fitness in Louisiana irises: analysis of parental and F1performance. Evolution. 52: 37–43.

    PubMed  Google Scholar 

  8. Charlesworth, D. (1995). Evolution under the microscope. Curr Biol. 5: 835–836.

    CAS  Article  Google Scholar 

  9. Cruzan, M. B., Hamrick, J. L., Arnold, M. L. and Bennett, B. D. (1994). Mating system variation in hybridizing irises: effects of phenology and floral densities on family outcrossing rates. Heredity. 72: 95–105.

    Article  Google Scholar 

  10. Devicente, M. C. and Tanksley, S. D. (1993). QTL analysis of transgressive segregation in an interspecific tomato cross. Genetics. 134: 585–596.

    CAS  PubMed  PubMed Central  Google Scholar 

  11. Dobzhansky, T. (1951) Genetics and the Origin of Species. Columbia University Press, New York.

    Google Scholar 

  12. Graham, J. H., Freeman, D. C. and Mcarthur, E. D. (1995). Narrow hybrid zone between two subspecies of big sagebrush (Artemisia tridentata: Asteraceae). II. Selection gradients and hybrid fitness. Am J Bot. 82: 709–716.

    Article  Google Scholar 

  13. Grant, V. (1958). The regulation of recombination in plants. In: Exchange of Genetic Material: Mechanisms and Consequences, Cold Spring Harbor Symposium on Quantitative Biology, vol. 23, pp. 337–363. Cold Spring Harbor, NY.

  14. Grant, V. (1966). The origin of a new species of Gilia in a hybridization experiment. Genetics. 54: 1189–1199.

    CAS  PubMed  PubMed Central  Google Scholar 

  15. Grant, V. (1981) Plant Speciation. Columbia University Press, New York.

    Google Scholar 

  16. Grant, P. R. and Grant, B. R. (1994). Phenotypic and genetic effects of hybridization in Darwin’s finches. Evolution. 48: 297–316.

    Article  Google Scholar 

  17. Harrison, R. G. (1990). Hybrid zones: windows on evolutionary process. Oxford Surveys Evolutionary Biology. 7: 69–128.

    Google Scholar 

  18. Heiser, C. B. JR (1947). Hybridization between the sunflower species Helianthus annuus and H. petiolaris. Evolution. 1: 249–262.

    Article  Google Scholar 

  19. Heiser, C. B. JR, Smith, D. M., Clevenger, S. and Martin, W. C. (1969). The North American sunflowers Helianthus. Mem Torrey Bot Club. 22: 1–218.

    Google Scholar 

  20. Hodges, S. A., Burke, J. M. and Arnold, M. L. (1996). Natural formation of iris hybrids: experimental evidence on the establishment of hybrid zones. Evolution. 50: 2504–2509.

    Article  Google Scholar 

  21. Kerner, A. (1894–1895) The Natural History of Plants, vols 1 and 2. Blackie and Son, London.

    Google Scholar 

  22. Lewontin, R. C. and Birch, L. C. (1966). Hybridization as a source of variation for adaptation to new environments. Evolution. 20: 315–336.

    CAS  Article  Google Scholar 

  23. Loveless, M. and Hamrick, J. L. (1984). Ecological determinants of genetic structure in plant populations. Ann Rev Ecol Syst. 15: 65–95.

    Article  Google Scholar 

  24. Mccarthy, E. M., Asmussen, M. A. and Anderson, W. W. (1995). A theoretical assessment of recombinational speciation. Heredity. 74: 502–509.

    Article  Google Scholar 

  25. Muller, H. J. (1942). Isolating mechanisms, evolution, and temperature. Biol Symposium. 6: 71–125.

    Google Scholar 

  26. Nei, M. (1973). Analysis of gene diversity in subdivided populations. Proc Natl Acad Sci USA. 70: 3321–3323.

    CAS  Article  Google Scholar 

  27. Rieseberg, L. H. (1991). Homoploid reticulate evolution in Helianthus (Asteraceae): evidence from ribosomal genes. Am J Bot. 78: 1218–1237.

    Article  Google Scholar 

  28. Rieseberg, L. H. (1997). Hybrid origins of plant species. Ann Rev Ecol Syst. 28: 359–389.

    Article  Google Scholar 

  29. Rieseberg, L. H. and Ellstrand, N. C. (1993). What can morphological and molecular markers tell us about plant hybridization? Crit Rev Plant Sci. 12: 213–241.

    CAS  Google Scholar 

  30. Rieseberg, L. H., Linder, C. R. and Seiler, G. J. (1995a). Chromosomal and genic barriers to introgression in Helianthus. Genetics. 141: 1163–1171.

    CAS  PubMed  PubMed Central  Google Scholar 

  31. Rieseberg, L. H., van Fossen, C. and Desrochers, A. (1995b). Hybrid speciation accompanied by genomic reorganization in wild sunflowers. Nature. 375: 313–316.

    CAS  Article  Google Scholar 

  32. Rieseberg, L. H., Sinervo, B., Linder, C. R., Ungerer, M. and Arias, D. M. (1996). Role of gene interactions in hybrid speciation: evidence from ancient and experimental hybrids. Science. 272: 741–745.

    CAS  Article  Google Scholar 

  33. Rieseberg, L. H., Baird, S. and Desrochers, A. (1998). Patterns of mating in wild sunflower hybrid zones. Evolution. 52: 713–726.

    Article  Google Scholar 

  34. Schluter, D. (1998). Ecological causes of speciation. In: Howard, D. J. and Berlocher, S. H. (eds) Endless Forms: Species and Speciation, pp. 114–129. Oxford University Press, New York.

  35. Stebbins, G. L. (1957). The hybrid origin of microspecies in the Elymus glaucus complex. Cytologia. (Suppl. 36),336–340.

  36. Templeton, A. R. (1981). Mechanisms of speciation — a population genetic approach. Ann Rev Ecol Syst. 12: 23–48.

    Article  Google Scholar 

  37. Ungerer, M. C., Baird, S. J. E., Pan, J. and Rieseberg, L. (1998). Rapid hybrid speciation in wild sunflowers. Proc Natl Acad Sci USA. 95: 11757–11762.

    CAS  Article  Google Scholar 

  38. Wang, H., Mcarthur, E. D., Sanderson, S. C., Graham, J. H. and Freeman, D. C. (1997). Narrow hybrid zone between two subspecies of big sagebrush (Artemisia tridentata: Asteraceae). IV. Reciprocal transplant experiments. Ecology. 51: 95–102.

    Google Scholar 

Download references


Many thanks to D. Dudle, K. Gardner, A. Peters, D. Repasky, N. Takebayashi, M. Ungerer and D. Wolf for their advice and encouragement in the early stages of this project. We are also grateful to members of the Rieseberg and Gastony labs and two anonymous reviewers for their careful reading of the manuscript. This research was supported by National Science Foundation Grant DEB-9806290 to L.H.R. and was facilitated by the extensive UNIX computing facilities at Indiana University.

Author information



Corresponding author

Correspondence to C Alex Buerkle.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Buerkle, C., Morris, R., Asmussen, M. et al. The likelihood of homoploid hybrid speciation. Heredity 84, 441–451 (2000).

Download citation


  • ecological selection
  • fertility selection
  • hybridization
  • introgression
  • recombinational speciation
  • spatial isolation

Further reading


Quick links