Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Genetics of speciation

As we approach the milestone of 2009, the 200th anniversary of Darwin's birth and the 150th anniversary of the publication of the Origin of Species, many questions concerning the causes of speciation remain open and speciation continues to be one of the most actively studied topics in modern evolutionary biology (Howard and Berlocher, 1998; Coyne and Orr, 2004). In this special issue, we present a selection of papers addressing aspects of the genetics of speciation. Genetic approaches have always been at the centre of speciation research and, like other areas of evolutionary genetics, the field is currently in a dramatic period of development as it absorbs the new possibilities of the post-genomic era. A rapidly increasing number of fully sequenced genomes are available for analysis and high-throughput sequencing and genotyping have become more routine and affordable. This makes the search for genetic changes associated with speciation more tractable, but it also brings risks. Understanding the genetics of speciation requires much more than trying to identify genetic differences between species; it is about trying to identify the key changes underlying important differentiation and reproductive isolation, the order in which the changes occur, the way they interact and the evolutionary forces that cause their spread. There is a danger that a plethora of genomic information might lead to largely descriptive studies of speciation because we are now at a stage where we can document differences in species’ genomes with unparalleled detail. This will not lead to understanding of the genetics of speciation unless it is possible to identify those changes likely to contribute to divergence rather than those that are carried along in its wake. If these key ‘speciation genes’ can be identified, then analysis of the classes or types of genes (or epigenetic changes) underlying speciation can follow and the changes can be linked to other aspects of the biology of divergence, especially the selection pressures that drive it. This issue, separating causative factors from mere differences, is one that has concerned speciation biologists over the last 150 years whether they are interested in morphology, karyotype, behaviour or ecology, but is now a critically important issue in genetics.

The 10 papers in this issue do not catalogue differences in the genomes of species, but ask questions about what are the key genetic changes underpinning speciation. They cover a wide range of taxa and a wide spectrum of barriers to gene exchange, but they are connected by the nature of speciation genes (which we define as genes that play a disproportionate role in the origin of species, while acknowledging that few if any genes evolve to cause speciation), their interactions and evolution. Some topics are familiar, such as the basis of Haldane's Rule (HR, Haldane, 1922; Turelli and Orr, 2000) and the probability of reinforcement (Butlin, 1995; Servedio, 2004), but there are novel twists. In particular, we have to broaden our view of ‘genes’ to include chromosomal and epigenetic effects, for example, as well as considering an extended range of life histories, including haplodiploids and species that only rarely reproduce sexually.

Two fundamental questions in speciation genetics have been the causes of HR—the sterility or inviability of the heterogametic (usually XY or ZW) sex—and the related possibility of a disproportionate role for the sex chromosomes in reproductive isolation. Qvarnström and Bailey (2008) review the extent of the evidence for and the potential causes of a ‘large X(Z) effect’ in speciation. There are contrasting claims in the literature about this (Reinhold, 1998; Ritchie and Phillips, 1998) and, interestingly, they find evidence for a difference between male- and female-heterogametic systems: a large X-effect on hybrid sterility and Z-linkage for sexual isolation. Sexually limited or antagonistic expression may be the most pervasive factor selecting for sex linkage of loci, but they argue that several factors may act jointly. Genes initially under ecological or sexual selection may be more likely to contribute to hybrid sterility as divergence accumulates. Together with other lines of evidence, this suggests that the distinction between pre- and post-mating isolation is becoming blurred at the genetic level; for example, fast evolution of male-specific loci under sexual selection may contribute to hybrid sterility. Koevoets and Beukeboom (2008) extend our understanding of HR into haplodiploid organisms. They show that HR is observed in several haplodiploid organisms, including the increasingly powerful and genome-sequenced model species Nasonia. The epistatic interactions thought to underlie HR are still expected to act in haplodiploids (although not until the F2 in males), and haplodiploids offer a good opportunity to investigate additional potential explanations of HR. An exciting and novel contributor to hybrid sterility is proposed by Li et al. (2008) who argue that divergence in the ‘pachytene checkpoint’, a mechanism that has evolved to detect aberrant meiotic products, may contribute to hybrid dysfunction, perhaps including HR. If such an error-checking mechanism diverges, then errors in hybrids may be checked less effectively, and hence sterility may become more common. Sexual dimorphism in the operation of the checkpoint might contribute to HR, and the authors argue that variation in the pachytene checkpoint could also contribute to well-established patterns in reproductive isolation such as differences between plants and animals in both the forms of reproductive isolation and the frequency of polyploidy. Widmer et al. (2008) also discuss potential differences in the causes of reproductive isolation between plants and animals. Isolation in plants is more likely to lie in genomic incompatibilities rather than pre-mating isolation, but the nature of these incompatibilities is diverse and probably multifactorial. Extrinsic sources of incompatibilities, such as ecological specialization, need further study (something that is also true for animals). Progress is also being made in our understanding of post-mating isolation between species of yeast, which differ in many ways from the model organisms more widely used in speciation research. Greig (2008) describes how there are a variety of forms of isolation, including chromosomal rearrangements, but he emphasizes a potential role of ‘antirecombination’, where chromosomal divergence precludes pairing and recombination. Surprisingly, the epistatic interactions expected under the standard Dobzhansky–Muller model (Turelli and Orr, 2000) have not been found. This, and the potential role of the pachytene checkpoint, illustrates how varied the genomic mechanisms underlying incompatibility can be: it is important to look beyond traditional Dobzhansky–Muller interactions between pairs of protein-coding loci when looking for the causes of hybrid dysfunction. Michalak (2008) also raises a relatively unexplored but potentially very important issue, the role of epigenetic changes in isolation. Gene expression is influenced in a heritable manner by epigenetic changes such as methylation and imprinting, and misexpression of genes in hybrids is a major cause of dysfunction. Non-coding small RNAs may also be involved. Together, these studies highlight how long-standing questions, such as the role of karyotypic phenomena, and more novel ones, such as the role of epigenetics, are important to modernizing our view of isolation, especially as we widen the range of study organisms.

Servedio (2008) reviews the importance of linkage disequilibrium in providing a bridge between post-zygotic and pre-mating isolation during speciation. Here, we also need to refine our view of the genetic basis of barriers to gene exchange. The completion of speciation requires a link between the initial divergence, which generates selection against hybrids (or against hybridization) and the further evolution of pre-mating reproductive isolation. ‘Magic trait’, ‘one-allele’ and ‘two-allele’ models are well-known alternatives that differ in the way in which recombination can reduce the probability of speciation by reducing linkage disequibrium (Ortiz-Barrientos and Noor, 2005). Servedio (2008) clarifies the distinctions between these models, and the circumstances in which those distinctions are blurred. If we can identify the loci that contribute to reproductive isolation, we can infer the causes of selection on them and the potential for one or a few loci to have a large influence on isolation. This is rarely possible, but mimicry is an area where the sources of selection for gene replacement seem clear. Mimetic wing patterns in butterflies provide a potential example of a ‘magic trait’, under strong natural selection but also influencing sexual isolation, as the wing patterns contribute to mate choice (Jiggins et al., 2008) Therefore, the first quantitative trait locus (QTL) studies of wing pattern divergence in Heliconius butterflies, described by Baxter and Jiggins (2008), are fascinating. They implicate known major colour pattern loci, but also novel genomic regions. The authors discuss how pleiotropic effects of major loci on multiple patterns elements may facilitate gene substitution and how interactions with modifiers are likely as mimicry is perfected. Another area where the loci involved in assortative fertilization are known; and hence multiple selective forces that can be investigated concerns gamete surface interactions in fertilization in free-spawning animals. These proteins have been studied and modelled in the context of speciation for some time now and are arguably some of the best understood speciation genes at a molecular level. Palumbi (2008) brings together a review of their function and evolution. These loci undoubtedly evolve rapidly, and are subject to a wide range of selection pressures. Sexual conflict due to polyspermy may be particularly strong, but more typical processes such as frequency-dependent selection, heterozygote superiority or character displacement all seem to influence their evolution.

The final paper in the issue was not originally commissioned as part of the Genetics of Speciation Special Issue, but is included here because of its clear connection with the issues discussed above. Smadja and Butlin (2008) review the importance of chemosensory variation in pre-mating isolation. Pheromones are well known to play an important role in sexual isolation and they have been studied intensively in lepidoptera and Drosophila, but the role of chemical signalling in speciation is more extensive, including ecological differentiation and a wide range of animals and plants. The mapping of QTLs and the identification of loci involved in chemical communication is making particularly strong progress and candidate gene approaches are now very promising. Loci and gene families involved in pheromone biosynthesis and olfaction and gustation are well characterized in many taxa, allowing detailed studies of their divergence and evolution. These seem to be among the most promising candidates for the identification of individual, well-characterized, ‘pre-mating isolation genes’.

In conclusion, the papers in this special issue of Heredity reveal the breadth of current studies into the genetics of speciation. They contain a fascinating mixture of studies of familiar questions and issues in evolutionary biology, as well as new and exciting ideas and insights. The range of organisms being studied in the context of speciation genetics is broadening and the picture of Dobzhansky–Muller epistasis as an explanation of hybrid sterility is becoming more generalized as the potential influence of chromosomal and epigenetic changes is being explored. Loci of large effect on isolation are being identified and we are, at last, starting to make progress in studying these in an ecological context and addressing the forms of natural or sexual selection responsible for their substitution.


  1. Butlin RK (1995). Reinforcement: an idea evolving. Trends Ecol Evol 10: 432–434.

    CAS  Article  Google Scholar 

  2. Coyne JA, Orr HA (2004). Speciation. Sinauer: Sunderland, MA, 545pp.

    Google Scholar 

  3. Haldane JBS (1922). Sex ratio and unisexual sterility in hybrid animals. J Genet 12: 7–109.

    Article  Google Scholar 

  4. Howard DA, Berlocher SH (1998). Endless Forms: Species and Speciation. Oxford University Press: Oxford.

    Google Scholar 

  5. Jiggins CD, Salazar C, Linares M, Mavarez J (2008). Review. Hybrid trait speciation and Heliconius butterflies. Philos Trans R Soc Lond B Biol Sci 363: 3047–3054.

    Article  Google Scholar 

  6. Ortiz-Barrientos D, Noor MAF (2005). Evidence for a one-allele assortative mating locus. Science 310: 1467.

    CAS  Article  Google Scholar 

  7. Reinhold K (1998). Sex linkage among genes controlling sexually selected traits. Behav Ecol Sociobiol 44: 1–7.

    Article  Google Scholar 

  8. Ritchie MG, Phillips SDF (1998). The genetics of sexual isolation. In: Howard DA and Berlocher S (eds). Endless Forms: Species and Speciation. Oxford University Press: Oxford, UK. pp 291–308.

    Google Scholar 

  9. Servedio MR (2004). The what and why of research on reinforcement. PLoS Biol 2: 2032–2035.

    CAS  Article  Google Scholar 

  10. Turelli M, Orr HA (2000). Dominance, epistasis and the genetics of postzygotic isolation. Genetics 154: 1663–1679.

    CAS  PubMed  PubMed Central  Google Scholar 

Download references

Author information



Rights and permissions

Reprints and Permissions

About this article

Cite this article

Butlin, R., Ritchie, M. Genetics of speciation. Heredity 102, 1–3 (2009).

Download citation

Further reading


Quick links