Introduction

Researchers have repeatedly shown that sympatric species pairs tend to exhibit stronger species mating discrimination (sexual isolation) than allopatric species pairs of the same genetic divergence (e.g. Coyne & Orr, 1989; Howard, 1993; Butlin, 1995). The same observation is also noted among populations within a species: populations sympatric with a sibling species show greater discrimination against mating with that species than populations allopatric with it. These observations do not stem exclusively from a publication bias, where researchers are quick to publish observations that appear to demonstrate results from natural selection rather than those that do not (e.g. such a bias would not apply to Coyne & Orr, 1989). Sympatry seems to be nonrandomly associated with stronger mating discrimination in many species.

This pattern has been attributed to ‘reinforcement’ — natural selection strengthening sexual isolation in response to maladaptive hybridization following secondary contact of two taxa. Many recent literature reviews have discussed some of the proposed cases of reinforcement (Howard, 1993; Butlin, 1995; Hostert, 1997). However, work in this area has been expanding rapidly. Here, I will review recent work on reinforcement, focusing on studies or developments not covered in earlier reviews. Next, I will present competing theories regarding the causes of greater mating discrimination in sympatry. Finally, I will point to novel directions for studying this pattern, what we can hope to learn about it and what remains to be done.

Reinforcement

Brief history

Traditionally, the pattern of greater species mating discrimination in sympatric taxa has been considered to be evidence for ‘reinforcement.’ Females that preferentially mate with their own species produce more or fitter offspring than those that mate at random. Similarly, males benefit from mating preferentially with females of the same species rather than wasting resources on heterospecific females. Hence, selection will favour alleles that confer mating discrimination. The outcome of this selection is reproductive character displacement — ‘the pattern of greater divergence of an isolating trait in areas of sympatry between closely related taxa than in areas of allopatry’ (Howard, 1993).

Acceptance of reinforcement has resembled stock market fluctuations. The elaboration and popularization of the theory stems from the writings of Dobzhansky 1940, (for example). Enthusiasm for the theory was initially unrealistically high, as allopatric divergence of mate preferences was attributed to occasional heterospecific migrants (Dobzhansky, 1940), and popular writings considered reinforcement to be the final step in speciation (e.g. Lewontin, 1974). Many studies documented experimental support for reinforcement (e.g. Koopman, 1950; Littlejohn & Loftus-Hills, 1968; Waage, 1975; Wasserman & Koepfer, 1977). However, in the 1980s, acceptance of the theory fell drastically. This fall did not stem from new empirical data, but rather, from theoretical and verbal arguments against it (e.g. Paterson, 1982; Spencer et al., 1986; Butlin, 1989). Finally, following Coyne & Orr’s (1989) classic survey, acceptance of reinforcement rose again, and more theoretical and empirical support has appeared in the past five years.

Lack of ubiquity

Evidence of reinforcement has not been ubiquitous (e.g. Walker, 1974; Ritchie et al., 1989; Sanderson et al., 1992). Some studies have documented patterns inconsistent with reinforcement. Many more such studies have probably been performed but not published or refocused onto other results.

One study that failed to find evidence of reinforcement was that of Ritchie et al. (1992) which examined mate preferences and their costs in Chorthippus parallelus grasshopper subspecies. These produce sterile hybrid males when crossed in the laboratory. They predicted that females captured from the zone of hybridization and forced to mate with random males from their population (nonselecting females) would produce offspring that were less fit than females from the same populations that select their mates. Ritchie et al. (1992) failed to find a greater difference between selecting and nonselecting females from the zone of hybridization than between such females from allopatric populations. This observation suggests that selection within the hybrid zone may have favoured alleles at loci that reduce hybrid dysfunction rather than alleles that increase mate discrimination (see Butlin, 1998).

This study presents a novel approach to empirically testing the reinforcement hypothesis, and it also tests for intraspecific sexual selection. However, its execution is cumbersome, and its application should be limited to organisms in which a priori information is available concerning the mode of fitness reduction in hybrids. Laboratory studies of fitness always face the criticism that they may be measuring the ‘wrong’ variables. Furthermore, because these subspecies readily mate with each other in sympatry, they are taxa unlikely to have evolved sexual isolation via reinforcement. Nonetheless, this approach nicely complements the standard mating test of reinforcement.

Role in speciation and recent studies

One of the most contentious arguments about reinforcement is its role in speciation. If one defines species as taxa that are incapable of exchanging genes, then reinforcement occurring in two taxa that produce fully sterile or inviable hybrids has no effect on ‘speciation.’ Initially, reinforcement was documented in species that either produce sterile hybrids or species that were not known to hybridize, and Butlin (1989) correctly argued that its role in speciation was uncertain. Furthermore, a theoretical model suggested that reinforcement was unlikely even in taxa that produce only sterile hybrids (Spencer et al., 1986).

Recently, a series of mathematical models suggested that reinforcement is not as unlikely as previously suggested. A common theme among most of these new models is that they simulate a novel mating preference acting on a display trait that already differs between two interacting populations. Furthermore, these models have delimited parameters that allow reinforcement to occur more readily. Liou & Price (1994) developed a quantitative model of reinforcement based on shifts in female preferences. They found that a genetic covariance between signals and preferences initially generated by reinforcement could drive speciation to completion. Kelly & Noor (1996) found that sterility caused by epistatic interactions between sex chromosomes and autosomes, a more realistic mode of postzygotic isolation than was used in previous models, also enhanced the probability of speciation by reinforcement. Servedio & Kirkpatrick (1997) observed that symmetric migration allows reinforcement to occur more readily than one-way migration (as in a continent–island model). However, Kirkpatrick & Servedio (1999) later identified a wide range of conditions under which reinforcement could occur with one-way migration, concluding that ‘reinforcement of a mating preference is expected quite generally’, even with very weak hybrid incompatibility.

Empirical evidence for reinforcement was also documented in some taxa that hybridize and produce (some) fertile offspring: fruit flies, flycatchers, and sticklebacks. I briefly discuss these in turn.

Females derived from populations of Drosophila pseudoobscura sympatric with D. persimilis display greater reluctance to mating with heterospecific males than females derived from allopatric populations (Noor, 1995), consistent with reinforcement. F1 hybrid males are sterile, but hybrid females are fertile, allowing for gene exchange and selection against mismating. Butlin 1995;(Butlin & Tregenza, 1997) suggested that hybrid fitness might be so low as to stop gene exchange in these species, but later genetic work demonstrated low levels of gene flow (not just hybridization) between these two species (Wang et al., 1997).

Sætre et al. (1997) documented evidence of reinforcement in flycatchers. Pied flycatchers (Ficedula hypoleuca) are black and white in populations allopatric to collared flycatchers (F. albicollis, also black and white), but pied flycatchers are brown in populations sympatric with collared flycatchers. This colour change is consistent with female preferences in these populations. Sætre et al. (1997) also showed that the colours observed in allopatric populations appear to be ancestral, suggesting that the divergence in sympatry results from reinforcement.

Rundle & Schluter (1998) found standard evidence of reproductive character displacement in mating studies of two threespine stickleback (Gasterosteus aculeatus) morphs. Some matings were detected in the wild, and F1 hybrids suffer a foraging disadvantage relative to their parents. One especially good feature of this study was their use of allopatric morphs that came from populations that were ecologically similar to those derived from sympatry, hence reducing the probability that adaptive divergence resulting from other ecological factors played a major role in the observed differences.

Other studies may later yield evidence for reproductive character displacement in taxa not fully isolated by postzygotic isolation; for example, Gerhardt (1994) noted changes in female Hyla chrysoscelis (grey treefrog) mate preferences in populations sympatric with H. versicolor. Although gene flow cannot occur through hybrid males, which are known to be sterile, the fitness of hybrid females has not yet been determined. Similarly, further studies of some of the classic examples of reinforcement, such as Litoria ewingi and L. verreauxi frogs (Littlejohn & Loftus-Hills, 1968), may also yield evidence for natural hybridization and gene flow.

If reinforcement explains the pattern of greater species discrimination in sympatric taxa, then these studies suggest that it contributes to the speciation process, at least in some cases. However, these studies rely primarily on differences in mate preferences of females or in male secondary sexual characters between sympatric and allopatric populations to infer the action of reinforcement. There are other possible causes for these patterns.

Other causes of the pattern of species discrimination

Reinforcement strengthened by runaway sexual selection

Coyne & Orr (1989), reiterating suggestions from other authors (e.g. Lande, 1982), suggested a second possibility: natural selection favouring assortative mating could initiate sexual selection and rapidly increasing discrimination, presumably through a runaway process (as modelled by Liou & Price, 1994). The difference they envisioned was that postzygotic isolation was not required, and hybrids merely having difficulty in securing mates may trigger the process, perhaps because hybrid males have inappropriate mating signals.

This idea is not truly distinct from standard reinforcement because the process is initiated by natural selection. Furthermore, some researchers have adopted a narrow view of postzygotic isolation — only hybrid sterility and hybrid inviability qualify to stop gene exchange. Coyne & Orr (1989) use only these two factors in their analysis, but unlike some investigators, they explicitly note that other forms of postzygotic isolation exist. Failure of hybrid offspring to secure mates could drive natural selection because there is no evolutionary difference between hybrids that are sterile and hybrids that cannot mate. Such hybrid dysfunction has been noted in a variety of taxa (e.g. Davies et al., 1997; Noor, 1997a), including some which have no other noticeable form of postzygotic isolation. Furthermore, as Coyne & Orr (1989) also note, other ecological variables may also cause postzygotic isolation (e.g. Grant & Grant, 1996), and these variables are often ignored in laboratory studies of reproductive isolation.

Differential fusion

Another explanation, termed ‘differential fusion’, suggests that only populations with strong mating discrimination persist in sympatry, while those populations lacking such discrimination fuse and lose their distinctness (Templeton, pers. comm.). Hence, species that are observed in sympatry today are a nonrandom sample of all species that have come into contact with one another.

Coyne & Orr (1989) argued against this interpretation on two grounds. First, the differential fusion hypothesis also predicts greater postzygotic isolation between sympatric species pairs, as postzygotic isolation also prevents fusion. In their data, they observed no difference between sympatric and allopatric species pairs in their postzygotic isolation. Secondly, the differential fusion hypothesis predicts that levels of prezygotic isolation observed among sympatric species should form a subset of those observed in allopatry. However, in Coyne & Orr’s survey, sympatric species possessed higher degrees of prezygotic isolation than observed among any of their allopatric species.

Coyne & Orr’s second argument against the differential fusion hypothesis assumes that they have sampled enough allopatric species pairs to capture all existing degrees of prezygotic isolation. Although the number of species they surveyed is impressive, one could argue that allopatric species with strong prezygotic isolation are exceedingly rare, and fusion of weakly isolated taxa is a common phenomenon. Hence, this argument may not fully invalidate Templeton’s suggestion. In response to the lack of difference in postzygotic isolation between sympatric and allopatric species pairs, Gavrilets & Boake (1998) recently demonstrated theoretically that prezygotic isolation is more effective than postzygotic isolation at preventing fusion following a founder effect. This observation may explain the lack of a difference between sympatric and allopatric species pairs in their postzygotic isolation, although the persistence of many tension zones (e.g. Barton & Hewitt, 1989) suggests that postzygotic isolation also prevents gene exchange in at least some cases.

We cannot exclude the differential fusion hypothesis as a cause for the greater discrimination observed in sympatric species pairs. However, this process probably could not have caused all the cases of greater discrimination between sympatric populations of two species compared with their allopatric populations. Unless gene flow between conspecific populations is very low, the fusion of some populations will probably cause all others eventually to fuse through the influx of recombinant genotypes. This condition is absent in at least one of the proposed cases of reinforcement (D. pseudoobscura, Schaeffer & Miller, 1992).

Noisy neighbours, or facilitated reproductive character displacement

A second hypothesis given by Templeton (cited in Howard, 1993) and Otte (1989) is that reproductive character displacement may result from interference in mate recognition signals of taxa that do not hybridize. This character displacement in mating signals can incidentally cause sympatric populations of two species to be more isolated from each other than allopatric populations. In at least one of the proposed cases of reinforcement, an extensive survey detected no hybrids in nature (Etges et al., 1999). Hence, this process may contribute to the overall pattern. However, this hypothesis cannot explain cases where hybrids are known to occur now or were known to occur historically. While this process may affect taxa that hybridize, it is more parsimonious to assume that selection is a result of hybridization.

Ecological variables

If an ecological variable allows the presence of one of the two species but also affects mating discrimination in the other, then the pattern of greater discrimination may be merely an artefact of either a plastic or selective response to this variable. Various environmental variables affect species discrimination (e.g. Kim & Ehrman, 1999); for example, Brazner & Etges (1993) have shown that differences between taxa in their mating discrimination can be increased or obliterated by merely raising the animals on different media.

Although feasible, this hypothesis is not a parsimonious explanation for the overall pattern — why should ecological variables consistently increase mating discrimination between sympatric taxa rather than sometimes decreasing it? However, the idea is difficult to exclude from all proposed cases of reinforcement.

Similarly, ecological character displacement may incidentally cause changes in mate-recognition signals that make sympatric populations of two species less likely to mate with one another. This hypothesis is extremely difficult to exclude because, unlike the previous one, it does generally predict increased isolation between sympatric taxa (e.g. Grant, 1986, p. 354; Grant & Grant, 1996).

Differentiating among the causes

In our quest for answers to evolutionary questions, we prefer to accept single answers that may explain a whole pattern. In truth, many patterns are composite phenomena, such as the popular Haldane’s Rule (e.g. Wu & Davis, 1993), and numerous processes may contribute to the overall pattern. Hence, the greater discrimination exercised by sympatric populations/species relative to allopatric populations/species should be explored on a case-by-case basis.

Howard (1993) proposed five requirements for attributing mate choice patterns to reinforcement. First, heterospecific matings must occur or have occurred in nature. The latter possibility can be inferred from molecular genetic data — an application that was nearly impossible only 20 years ago. Secondly, selection against hybridization must be identified in the field. Thirdly, if reinforcement is inferred from displacement in a secondary male sexual character, the displacement must be detectable by females. This requirement obviously does not apply to detected shifts in female discrimination itself. Fourthly, the variation must be heritable and capable of responding to selection. Finally, displacement must not have occurred for other reasons, most notably ecological reasons. This last requirement is one where many studies have failed.

Ecological (especially field) studies of reinforcement candidates are lacking (but see Grant & Grant, 1996; Rundle & Schluter, 1998). Specifically, many authors (myself included) have not examined fully the conditions under which species mate in their natural settings as compared to laboratory situations. Given the effects of temperature, food source, etc., on mate preferences, such studies are essential. Many of the proposed alternatives to reinforcement may be accepted or refuted by data on the ecology and behaviour of the species in nature as well as direct observational data on interspecies interactions. Because sympatry is not an all-or-nothing trait, such detailed studies of the extent of interspecific interactions may clarify the relative contributions of reinforcement and other forces.

Some authors have suggested additional criteria for studies to document reinforcement convincingly; for example, Butlin (1995) noted that reinforcement is more convincing if character displacement in a male phenotype is observed. While true, this should not be a requirement, as female discrimination may increase without change in male phenotypes within species. Reinforcement can reduce the range of phenotypes that females accept so that only conspecific males, possibly all conspecific males, are acceptable; for example, studies of brown plant-hopper mate preference found more variation in the range of preferred male phenotypes than mean preferred phenotype (Butlin, 1993). Hence, if reinforcement were to operate in this species, it is likely that the range of preferred male characters would merely decrease, possibly producing no corresponding change in mean male phenotypes.

New directions

Given that there is indeed a pattern of greater species discrimination between sympatric taxa than between allopatric taxa, what questions are left besides presenting new examples or exploring existing examples in more detail? There are three other approaches that seem fruitful: (i) documenting the pattern in a broader sense; (ii) deriving explicit, diagnostic and testable theoretical predictions regarding the competing hypotheses; and (iii) using these theoretical predictions to infer evolutionary processes empirically. I will discuss two applications of these approaches.

Frequency of pattern

Although the pattern of greater mating discrimination in sympatric taxa clearly exists, it is difficult to ascertain exactly how common it is as a result of publication bias. The datasets of (Coyne & Orr 1989, 1997) do not attempt to address the frequency of the pattern, as they document the existence of reinforcement through a statistical test comparing prezygotic isolation in sympatric vs allopatric taxa. The pattern is striking, so one may infer that increased isolation of sympatric taxa is not exceedingly rare. Similar support for reinforcement or other effects of sympatry were documented in a survey of hybrid zones across a wide range of taxa (Howard, 1993) and an analysis of rates of speciation in fish (McCune & Lovejoy, 1998).

One way to determine the frequency of the pattern is to apply the comparative method on taxa where phylogenetic information is available. Consider three species — 2 ingroup species and their outgroup. Imagine that one species (A) is sympatric with the outgroup (C), and the other (B) is allopatric to both species. If sympatric species exhibit greater species discrimination than allopatric species, A and C should be more reluctant to mate with each other than B and C. Furthermore, if many such phylogenies are known and sexual isolation data are available, one can statistically infer maximum and/or minimum frequencies of the effects of sympatry across taxa. Using a small dataset from Drosophila, Noor (1997b) executed this study and estimated with 95% confidence that sympatry affects sexual isolation in at least 21% of such species groups. Further use of this method with larger datasets may more precisely estimate how often sympatry affects sexual isolation.

Using theoretical predictions empirically

Many theoretical models of reinforcement have focused on its plausibility rather than on how to identify it. Even when empirical predictions are given, they often diagnose ‘conditions favourable for reinforcement’ vs ‘conditions unfavourable for reinforcement’. While such diagnoses may be useful, specific predictions regarding the products of reinforcement may be more timely; for example, Haldane postulated that adaptive mutations that reach a high frequency within a species are generally dominant, thus enhancing the efficacy of selection and preventing their accidental loss when initially rare. This principle, termed ‘Haldane’s sieve’, has been applied to examples of adaptive evolution such as mimicry (Turner, 1977). This principle may also apply to the enhanced sexual isolation that evolves in direct response to natural selection (reinforcement).

Populations of D. pseudoobscura sympatric with their sibling species, D. persimilis, seem to have evolved greater species discrimination via reinforcement (Noor, 1995). Haldane’s sieve predicts that the greater species discrimination exhibited by sympatric populations of D. pseudoobscura will be dominant. This prediction has been confirmed by crosses between sympatric and allopatric populations of D. pseudoobscura, as the F1 females exhibit the same degree of discrimination as sympatric D. pseudoobscura (unpublished data). This observation is consistent with Haldane’s sieve and some sort of adaptive evolution.

According to the Haldane’s sieve principle, if species-level sexual isolation results largely from direct natural selection, then hybrid females between species should discriminate against males of either or both parental types. Natural selection within species would have favoured dominant preferences that reduce the frequency of mating with heterospecifics, and this dominance could be expressed in interspecific hybrids. Although genetic data on mate preferences are surprisingly rare (e.g. Bakker & Pomiankowski, 1995; Ritchie & Phillips, 1998), hybrid females of Drosophila species generally mate with males of both parental species, irrespective of whether females from one or both parental species discriminate (Noor, 2000). This finding suggests either that direct selection is not the primary force in driving the pattern of greater sexual isolation in sympatry or that the magnitude of the effect of direct selection in creating the overall observed discrimination is relatively small.

This conclusion is nonetheless premature. First, we have only one example of the product of natural selection on species discrimination behaving as a dominant character. The Haldane’s sieve principle seems to have been violated in the evolution of mimetic wing patterns in Heliconius butterflies (Brower, 1996), and the same may be true for species mating discrimination. Secondly, dominance of mating discrimination may be disrupted in species hybrids. Finally, species discrimination may be caused, in part, by more complicated interactions among loci such that a principle as simple as ‘Haldane’s sieve’ may not be adequate.

Nonetheless, this protocol demonstrates how explicit predictions from theoretical models may be used to elucidate the forces that cause the pattern of greater sexual isolation in sympatric taxa. Future mathematical models should attempt to produce more precise predictions that distinguish between possible explanations for the pattern.

Conclusions

While many accept that reinforcement explains the pattern of greater mating discrimination observed in sympatric taxa, I caution that we have not adequately explored the alternatives in many cases. Generalizations from broad species surveys can only address presence, frequency and magnitude of the pattern (see below). Detailed studies of proposed cases of reinforcement are needed, as are theoretical models that can derive diagnostic predictions of different causes of the pattern.

Even if sympatric taxa are always more discriminating than allopatric taxa, the magnitude of the effect of sympatry may be small; for example, if all taxa that show 99% species discrimination in allopatry evolve the extra 1% in sympatry, the effects of sympatry are slight, despite being frequent. In further documenting the pattern, we must evaluate both the frequency and the magnitude of the effect (see also Hollocher et al., 1997). Indeed, models of reinforcement have consistently suggested that it is most likely where a large initial divergence in mate preferences has already occurred (e.g. Liou & Price, 1994; Kelly & Noor, 1996). The difficulty associated with estimating magnitudes, however, comes from the diversity of opinions regarding experimental mate choice design (e.g. choice vs no-choice) and statistical evaluation of ‘discrimination’. The former should be addressed on a species-specific basis by using the mating design most similar to how matings occur in nature. The latter is a more formidable task.

With the recent growth of molecular population genetics, some of the questions that were difficult to resolve 20 years ago can be answered quickly and easily today. Future theoretical or statistical models of reinforcement, or adaptive evolution in general, will probably yield genetic predictions that can be tested directly. Genetic tools may help us finally to understand how the pattern of greater species discrimination in sympatric taxa came to be.