Introduction

The strength of pre- and postzygotic isolation mechanisms varies among taxa from barriers that completely prevent interspecific mating to weaker impediments that allow varying degrees of hybridization (Stebbins, 1950; Howard, 1993; Arnold, 1997). Postzygotic mating barriers can result in the elimination of virtually all first-generation hybrid genotypes (Stebbins, 1950; Dobzhansky, 1951; Christie & MacNair, 1987) or may act to reduce the numbers of other specific hybrid genotypes (Grun, 1976; Lachaise et al., 1986; Cruzan & Arnold, 1994; Johannes et al., 1995; Ulloa et al., 1995). The observation that reductions in viability and fertility vary in a nonadditive manner with the proportions of the parental genomes in hybrids suggests the action of interspecific epistatic effects (i.e. caused by differences in coadapted gene complexes), and such interactions often occur between the cytoplasmic and the nuclear genomes (Grun, 1976; Cruzan & Arnold, 1994; Johannes et al., 1995; Liu et al., 1995; Ulloa et al., 1995). Moreover, characterization of patterns of epistasis in interspecific hybrids may provide an assessment of alternative models of speciation (Whitlock et al., 1995; Gavrilets, 1997). Although nuclear and cytonuclear epistasis are apparently common and may contribute to the genetic barriers between taxa, their influence on patterns of introgression and the genetic structure of hybrid populations has received little attention.

Population-level studies of hybrid zones have contributed to our understanding of the effects of dispersal, selection and assortative mating upon patterns of hybridization and introgression. The development of methods for the analysis of cytonuclear disequilibria in hybrid populations (as measured by the statistics D1, D2 and D3; Asmussen et al., 1987; Arnold, 1993) has provided insights into processes contributing to the genetic structure of hybrid populations (e.g. Asmussen et al., 1989; Forbes & Allendorf, 1991; Paige et al., 1991; Cruzan & Arnold, 1993; Patton & Smith, 1993; Scribner & Avise, 1993). Departures from random associations between nuclear and cytoplasmic genetic markers can be useful as indicators of dispersal of parental genotypes into the hybrid populations (Paige et al., 1991; Arnold, 1993), assortative mating among similar genotypes (Asmussen et al., 1987; Patton & Smith, 1993), selection against specific hybrid genotypes (Arnold et al., 1988; Arnold, 1993; Cruzan & Arnold, 1994; Babcock & Asmussen, 1996) or cytoplasmic male sterility (Cellino & Arnold, 1993). Previous analytical treatments of sex-specific differences in viability and fertility (Cellino & Arnold, 1993; Babcock & Asmussen, 1996) indicate that cytonuclear interactions can affect intergenomic associations. In particular, cytonuclear epistasis (cytonuclear incompatibility or positive cytonuclear interactions) could contribute to the maintenance of disequilibria in populations and influence the extent and pattern of introgression.

Here, the effects of cytonuclear incompatibility and nuclear-based assortative mating on levels of cytonuclear disequilibria are analysed. This study was prompted by the observation of strong intraspecific genetic associations in hybrid populations of Louisiana irises (Cruzan & Arnold, 1993). The aim was to determine specifically whether the observed levels of selection and assortative mating could maintain the high levels of cytonuclear disequilibria found in these hybrid populations.

Materials and methods

Simulations

The effects of assortative mating and cytonuclear epistatic selection on measures of homozygote cytonuclear disequilibria (measured as the deviation from the expected frequency for combinations of homozygote nuclear genotypes and the cytoplasmic haplotype, D1=u1−ux and D1=−D3; Table 1; Asmussen et al., 1987) are examined for a single-locus diploid nuclear marker and a maternally inherited cytoplasmic marker. This approach differs from the cytoplasmic male sterility models of Cellino & Arnold (1993), because the effects of cytonuclear interactions on the viability of offspring rather than on adult fertility are examined. Viability differences among hybrid genotypes are relatively common (e.g. Christie & MacNair, 1987; Cruzan & Arnold, 1994; Johannes et al., 1995; Liu et al., 1995) and would be expected to have a stronger impact than fertility differences on the genetic composition of adult populations. We assume a large population not influenced by genetic drift, mutation or gene flow. The two parental species are characterized by diagnostic nuclear genotypes (AA or aa) and cytoplasmic haplotypes (C or c; Table 1). At the beginning of each generation cycle, a population of newly formed seeds is subjected to viability selection based on their cytonuclear genotypes to produce individuals at the adult stage. Adults mate randomly or according to the model specified below to produce the subsequent generation of seeds. All analyses were initiated with the nuclear alleles and cytoplasmic haplotypes from the two parental species at equal frequencies. The initial population consisted of parental (AA and aa) and hybrid (Aa) genotypes at Hardy–Weinberg frequencies, with no association between nuclear genotypes and cytotypes (D1=D2=D3=0). The four homozygote cytonuclear combinations in Table 1 were thus initially at a frequency of 0.125, and both heterozygote genotypes were at a frequency of 0.25.

Table 1 Genotypic frequencies for two cytoplasmic markers and a diallelic nuclear locus (adapted from Asmussen et al., 1987)
Full size table

Selection was imposed on the population by reducing the frequency of individuals possessing interspecific cytonuclear combinations between the seed and adult stages (Table 2) and then normalizing each cytonuclear class with respect to the total of all classes (1−su2−sw1−sh v; Table 2). Homozygous interspecific genotypes were removed at a rate determined by s (0.0<s≤1.0), and heterozygote frequencies in the adult class were controlled by a combination of selection and the dominance of nuclear alleles (sh; Table 2). The alleles at the nuclear locus were assumed either to interact in an additive fashion (h=0.5) or to exhibit varying levels of dominance with respect to the intraspecific (0≤h<0.5) or interspecific allele (0.5<h≤1.0).

Table 2 Weighted contributions of each cytonuclear genotype to the adult generation after the effects of selection (s) and the level of dominance at the nuclear locus (h) that results from cytonuclear incompatibility
Full size table

Genotype frequencies in each new seed stage were determined by reciprocal matings between pairs of adult genotypes acting equally as pollen and seed parents. We assume that mating patterns are controlled strictly by a nuclear locus. Our approach differs from that of the refined mating model of Arnold et al. (1988) and Asmussen et al. (1989) because it does not assume any direct influence of the cytoplasmic haplotype on mating patterns. The mating model analysed here also differs from nuclear-based assortative mating models (Asmussen et al., 1987) because it defines a separate mating frequency for heterozygote genotypes. Mating frequencies in this model were determined by weighting combinations of nuclear genotypes (Table 3). Weights for matings between alternative nuclear homozygotes ranged from fi=1.0 (no assortative mating) to fi=0 (complete assortative mating for parental genotypes). For the models analysed here, it was assumed that mating between heterozygotes and each homozygous genotype occurred according to a weight intermediate between the interspecific rate and random mating [fh=(1+fi)/2]. Summing the mating frequencies in Table 3 and the equivalent frequencies for c haplotypes yields the frequency of each seed genotype in the next generation as:

Table 3 Frequencies for different mating events and for the progeny cytonuclear genotypes produced by assortative mating
Full size table

where

is a normalization factor based on the sum of all classes, and primes denote values in the subsequent seed generation. Nuclear genotypical frequencies expected under random mating [i.e. (u+½v)2, 2(u+½v)(w+½v) and (w+½v)2 for AA, Aa and aa, respectively; Hedrick, 1983, p. 42] can be obtained as a special case of this model by assuming no assortative mating (fi=fh=1) and by summing across cytoplasmic classes (i.e. 1a with 1d, 1b with 1e, and 1c with 1f).

Numerical simulations were conducted to examine the effects of (i) selection as a result of cytonuclear incompatibility; (ii) nuclear-based assortative mating; (iii) selection and different levels of dominance for the nuclear alleles; and (iv) assortative mating combined with selection on levels of homozygous cytonuclear disequilibria. All simulations were run for 50 generations or until changes in disequilibria measures were less than 0.001.

Experiments

The viability of genotypes with different cytonuclear combinations was inferred from patterns of seed abortion observed in a population of hybrid Louisiana irises using nuclear (RAPD) and cytoplasmic (cpDNA) markers. Data on abortion patterns were obtained for open-pollinated fruits collected in a population located north of St Martinville, St Martin Parish, LA (Cruzan & Arnold, 1993, 1994). Previous analyses indicated that individuals in this population possess various mixtures of genetic markers diagnostic for Iris fulva and I. brevicaulis (Cruzan & Arnold, 1993). Seeds from flowers that had been open during the period when both parental genotypes and hybrids were in bloom were collected as fruits matured (Cruzan & Arnold, 1994). Seeds were dissected and DNA isolated from their embryos, which tended to remain healthy even in aborted seeds. Abortion of seeds was apparent from degeneration of the endosperm that ranged from darkening and liquefaction of the tissue to seeds that only contained desiccated fragments as the only remaining evidence of endosperm (Cruzan & Arnold, 1994). Embryos were easily identified in all seeds from their distinct torpedo-like morphology.

Embryos were assayed for six RAPD markers with dominant expression (IF154A, IB156A, IB165A, IF165A, IF165B and IB169B) and a codominant RAPD marker (L180; Cruzan & Arnold, 1994). Chloroplast DNA is nearly always maternally inherited in these species (Cruzan et al., 1993), so the cpDNA haplotypes of embryos were assumed to be the same as that of the seed parent. In all cases, the markers used were confirmed to be diagnostic for these species in allopatric populations. Amplifications and scoring of DNA markers were conducted using previously described methods (Cruzan & Arnold, 1993). The effect on viability of the number of nuclear genetic markers that were interspecific relative to the cpDNA haplotype was assessed using linear regression and graphical analyses.

Data were analysed by examining the relationship between the number of foreign nuclear markers (with respect to the cpDNA) and the level of embryo viability. The effects of specific combinations of nuclear markers and cytoplasmic haplotypes on embryo viability were analysed with G-tests of heterogeneity (Sokal & Rohlf, 1981). The strong association among nuclear and cytoplasmic genotypes made it impossible to analyse a full model that included all possible combinations of nuclear and cytoplasmic markers, so the effects of combinations of nuclear markers on viability were examined using tests of heterogeneity.

Results

Simulations

We report only the homozygote cytonuclear disequilibrium (D1), as the heterozygote measure (D2) remained at zero because of a lack of asymmetrical mating in these simulations, and D3 directly reflects values of D1 (i.e. D1=−D3 when D2=0; Asmussen et al., 1987).

The level of cytonuclear disequilibrium (D1) measured at the adult stage increased proportionally with the intensity of selection (s) caused by cytonuclear incompatibility (Fig. 1). The effectiveness of selection was increased when the expression of the interspecific allele was raised from zero (completely recessive) to one (completely dominant; Fig. 1). The increase in cytonuclear disequilibria with strong selection and high levels of dominance apparent in Fig. 1 was caused by the preservation of disequilibria from the adult stage to the subsequent seed stage. Disequilibria values were reduced by half in each generation of seeds, but a combination of strong selection and dominance maintained relatively high levels of disequilibria. Similar effects of reduced levels of cytonuclear disequilibria after mating were observed in the models examined by Asmussen et al. (1989).

Fig. 1
figure 1

The effects of selection on interspecific cytonuclear genotypes and the level of dominance of the interspecific nuclear allele on the level of cytonuclear disequilibrium in adults. Selection is imposed between the adult and seed stages and mating is random.

Full size image

The influence of cytonuclear disequilibria at the seed stage on the level of disequilibrium at the adult stage can also be seen in the interaction between assortative mating and selection (Fig. 2). It is clear that, by itself, assortative mating based only on nuclear genotypes cannot generate cytonuclear disequilibria. However, in combination with moderate to strong selection, assortative mating increased the levels of cytonuclear disequilibria at both the seed and the adult stages (Fig. 2). Other models have indicated that assortative mating influenced by cytoplasmic haplotypes could produce relatively high levels of cytonuclear disequilibria (Arnold et al., 1988; Asmussen et al., 1989). In those analyses of epistatic mating models, the mating behaviour was determined by a combination of the cytoplasmic haplotypes and the nuclear genotype, which promoted the generation of intraspecific cytoplasmic associations. In the model being analysed here, mating behaviour is determined by the nuclear genotype alone. The increase in the level of disequilibria apparently results from the tendency for nuclear-determined assortative mating to preserve cytonuclear associations in the subsequent seed generation, especially at relatively high levels of assortative mating at which the drop in disequilibria between the adult and seed stage is not as severe (Fig. 2). Assortative mating, in combination with moderate to strong selection, produced a synergistic effect that allowed the maintenance of relatively high levels of disequilibria (Fig. 2).

Fig. 2
figure 2

The influence of assortative mating on the level of homozygous cytonuclear disequilibria (D1) in adults and their progeny maintained in a population for different levels of selection on interspecific cytonuclear genotypes. Nuclear alleles are assumed to be codominant (h=0.5).

Full size image

Experiments

A total of 172 embryos from 16 fruits was assayed for all seven nuclear markers. Of the 11 different maternal genotypes sampled (not more than two fruits from any single genotype), eight were carrying I. brevicaulis cpDNA (n=127 seeds), and three possessed I. fulva cpDNA haplotypes (n=45 seeds). For I. brevicaulis haplotypes, eight out of 11 fruits had at least some aborted seeds (mean=39.2% aborted, range 0.0–83.3%), whereas only one out of five fruits contained aborted seeds for the I. fulva haplotypes (mean=4.6% aborted, range 0.0–23.1%).

Embryo viability for I. brevicaulis haplotypes was greatest for genotypes that had only intraspecific nuclear markers relative to the cpDNA haplotype and decreased with the number of interspecific nuclear markers (Fig. 3). Embryo viability for I. fulva haplotypes remained relatively high for all genotypical classes and did not display a decrease when more interspecific markers were present. For individuals carrying I. brevicaulis cpDNA, the decrease in viability occurred at a rate of nearly 10% per interspecific nuclear marker (regression weighted by sample size per class: slope=−9.6%, r2=0.987, P<0.001).

Fig. 3
figure 3

Viability of embryos from open-pollinated plants with cpDNA from Iris fulva and I. brevicaulis in response to different numbers of foreign nuclear markers. Numbers in parentheses above the axis indicate the total number of seeds in each genotypic class, and numbers in parentheses below the axis represent the percentage representation by foreign diagnostic genetic markers for each marker class.

Full size image

In addition to the overall effects of foreign nuclear markers on embryo viability, individual markers appear to vary in the strength of their interaction with the cpDNA haplotype (Table 4) and Table 5). For embryos carrying I. brevicaulis cpDNA, foreign RAPD genotypes for two of the markers were associated with higher frequencies of abortion (IF165A and IB169B), three of the markers displayed weaker associations with increased frequencies of abortion (L180, IF165B and IB165A) and two markers had no apparent association (IB156A and IF154A). For I. fulva cpDNA haplotypes, none of the nuclear markers had strong associations with embryo viability, but abortion frequencies were slightly lower for interspecific RAPD phenotypes for the majority of the markers (Tables 4 and 5).

Table 4 Levels of embryo viability (percentage) in a population of Louisiana irises for one codominant and six dominant nuclear marker genotypes (RAPDs) in combination with two cpDNA haplotypes
Full size table
Table 5 Associations between nuclear markers and levels of embryo viability in a hybrid population of Louisiana irises
Full size table

Discussion

Cytonuclear incompatibility and nuclear-determined assortative mating can contribute to homozygous cytonuclear disequilibria (i.e. D1; Asmussen et al., 1987), and together are capable of maintaining relatively strong associations between cytoplasmic genomes and nuclear genotypes at the adult stage. Incompatibility results in the removal of seeds carrying interspecific combinations of cytoplasmic and nuclear genotypes and has a direct impact on the level of disequilibria in the adult generation. In contrast, fertility selection resulting from cytoplasmic male sterility appears to have little impact on levels of disequilibria that can be maintained in hybrid populations (Cellino & Arnold, 1993). Viability selection on seed cytonuclear genotypes can have rather large effects on disequilibria, and the consequences of cytonuclear incompatibility are greater when interspecific nuclear alleles display some level of dominance (i.e. they are associated with a reduction in viability when in the heterozygous state). Although assortative mating can contribute to increases in the level of homozygous disequilibria at the seed and adult stages, mating behaviour that is based on the nuclear genotype alone is not capable of generating a cytonuclear disequilibrium de novo (see also Asmussen et al., 1987, 1989). However, a combination of assortative mating and cytonuclear incompatibility can generate relatively high levels of disequilibria. Such a synergistic interaction between cytonuclear incompatibility and assortative mating may be contributing to the high levels of disequilibria observed in hybrid iris populations.

Levels of embryo abortion in open-pollinated hybrid irises appear to be associated with the number of nuclear markers that are interspecific with respect to the cpDNA marker for one haplotype but not the other. Our ability to distinguish nuclear interactions from nuclear–cytoplasmic epistatic effects was hampered by strong associations among intraspecific markers present in this population (Cruzan & Arnold, 1994). Genetic associations also limited the number of genotypical classes that could be observed (four and six interspecific nuclear marker classes out of eight possible for I. fulva and I. brevicaulis haplotypes, respectively). However, if the observed patterns of embryo abortion were strictly caused by interactions among nuclear loci, then levels of embryo abortion would be expected to be highest when the nuclear genome consisted of an equal mixture from the two parental species (i.e. at a 50% mixture, which occurred between classes 3 and 4 in Fig. 3), and to be lower when the nuclear genome was dominated by markers from one species. The observed pattern of embryo viability for I. brevicaulis cpDNA haplotypes contradicts this expectation, with abortion rates being highest when the nuclear genotype was dominated by markers that were interspecific with respect to the cpDNA haplotype (pooled classes 4 and 5 compared with classes 0–3: G=2.75, P<0.05, one-tailed test of heterogeneity; Fig. 3). Although additional experiments would be required to analyse fully the interactions between nuclear and cytoplasmic genomes, the available data suggest that cytonuclear epistatic effects may be contributing to the genetic structure of this population of hybrid irises.

The apparent asymmetry of cytonuclear interactions in this hybrid population of irises also implies that cytonuclear incompatibility is affecting the distribution of adult genotypes. Whereas the effects of interspecific cytonuclear interactions appear to be severe for seeds carrying I. brevicaulis cpDNA, there is no apparent reduction in viability for seeds from I. fulva maternal parents. Similar asymmetrical effects of cytoplasmic factors in hybrids have been observed in plant (e.g. Oenothera and Epilobium: reviewed in Grun, 1976; Allium: Ulloa et al., 1995; Brassica: Song et al., 1995) and animal (e.g. Drosophila: Lachaise et al., 1986; Gambusia: Scribner & Avise, 1993; Nasonia: Johannes et al., 1995) hybrids. The effects of nuclear–cytoplasmic interactions range from slight reductions in male or female fertility to a complete loss of viability for offspring resulting from crosses in one direction that are not apparent in reciprocal crosses (Grun, 1976). These fitness reductions for certain nuclear–cytoplasmic combinations would be expected to have consequences for the maintenance of cytonuclear disequilibria and for patterns of introgression in hybrid populations.

The consequences of cytonuclear incompatibility for the maintenance of disequilibria between cytoplasmic and nuclear loci depends largely on dominance relationships among alleles at the nuclear locus. The observation that F1 hybrids from I. fulva and I. hexagona appear to be relatively vigorous and even to display some heterosis under greenhouse conditions (M. L. Arnold, unpubl. data) suggests that the majority of deleterious cytonuclear interactions in Louisiana iris hybrids are probably recessive. Deleterious recessive interactions appear to be typical of hybrids between closely related taxa, whereas evidence of deleterious dominant cytonuclear interactions in first-generation hybrids are more commonly seen for combinations of more distantly related species (Grun, 1976). For advanced generation hybrids, however, the behaviour of genetic markers will be governed by the cumulative effects of a number of associated loci, which may be carrying parental alleles in all combinations. With recombinant hybrid genotypes, the influences of associated epistatic loci would be expected to produce apparent codominance at marker loci.

Using observed changes in the level of disequilibria for a codominant nuclear marker between the adult and subsequent seed stages (table 2 in Cruzan & Arnold, 1994), the strength of selection acting on different cytonuclear genotypes can be estimated. The majority of seeds produced (52%) had heterozygous nuclear genotypes, whereas only 5% were homozygous for alleles that were interspecific relative to the cpDNA haplotype. In comparison, in the previous adult stage, there were no interspecific homozygotes, and the frequency of heterozygotes was 51% lower. Assuming that the cumulative effect of associated loci would produce codominance (i.e. h=0.5), the change in heterozygote frequencies between generations can be used to infer that s must be near 1.0 (from Table 2, s=[v2−v′2]/hv2; with v2=0.52 and v′2=0.27 from Table 2 in Cruzan & Arnold, 1994). Hence, if the observed differences in genotypical frequencies between these two generations are typical of the past, then the level of disequilibria that could be maintained by cytonuclear incompatibility alone would be moderate (D1=0.15: Fig. 1). This is somewhat lower than levels of disequilibria observed in hybrid iris populations (mean D3=−0.20, D3≈−D1 and D1 ranges from 0.00 to 0.25: Cruzan & Arnold, 1993, 1994), suggesting that additional factors may be promoting the maintenance of strong intraspecific associations in Louisiana iris hybrids.

Assortative mating could contribute to the high levels of cytonuclear disequilibria observed in Iris populations by preserving genetic associations developed through selection on seed genotypes. Previous results suggest that assortative mating in hybrid irises can be relatively strong (Cruzan & Arnold, 1994), with more than 80% (145 out of a total of 175) of the plants that differ in their cpDNA haplotype not overlapping in flowering time. This difference in flowering time combined with a moderate amount of interhaplotype mating during flowering overlap (Cruzan & Arnold, 1994) would be expected to translate into a relatively strong assortative mating coefficient (i.e. fi<0.20). When combined with strong selection on specific cytonuclear combinations, this level of assortative mating could produce disequilibrium values that are in the range of those found in this population of hybrid irises (for s=1.0 and fi=0.20, D1≲0.20 at equilibrium; Fig. 2). It should also be noted that previous results suggest that environmental selection contributes to the distribution of parental and hybrid genotypes in this population (Cruzan & Arnold, 1993).

Assortative mating could be contributing to the maintenance of associations among intraspecific alleles at nuclear loci. Several nuclear loci with negative epistatic effects on embryo viability that were acting in concert could have a more profound effect on levels of cytonuclear disequilibria than loci acting individually. In Louisiana irises, two of the genetic markers (IF165A and IB169B; Tables 4 and 5) displayed relatively strong effects on abortion frequencies when in combination with other nuclear and cytoplasmic markers, suggesting that at least two linkage groups included epistatic loci. If the strength of epistatic interactions for loci cosegregating with these markers and elsewhere in the genome were strong enough, then their cumulative effect might maintain relatively high levels of cytonuclear disequilibria even if all of the deleterious alleles involved were completely recessive. Assortative mating could maintain genetic associations across the genome by promoting intraspecific genetic combinations at the seed stage. The combined effects of assortative mating and epistatic selection on a number of loci would reduce opportunities for introgression and for the production of recombinant backcross genotypes.

The results from this and a number of other studies suggest that cytoplasmic incompatibilities and interactions among nuclear loci in hybrids are common. These epistatic effects may make specific genetic combinations from related taxa difficult or impossible to achieve. The combination of positive and negative epistasis could lead to the repeated formation of certain hybrid genotypes (Rieseberg et al., 1996; Arnold, 1997). Such interactions expressed as ‘coadapted gene complexes’ have been emphasized as an important component of integration within, and separation between, species (Dobzhansky, 1951; Templeton, 1989). Intergenomic epistasis will have consequences for the genetic structure of hybrid populations, patterns of introgression and hybrid speciation (e.g. Rieseberg et al., 1996; Arnold, 1997). The examination of fitnesses of different genetic combinations in hybrid populations will enhance our understanding of genetic barriers between taxa and may provide insights into adaptive landscape topographies and the alternative modes of speciation that have been proposed (Whitlock et al., 1995; Gavrilets, 1997).