Introduction

If one accepts some version of the biological species concept, the central problem of speciation is the origin of isolating barriers that prevent actual or potential gene flow among populations (Coyne and Orr, 2004). Its solution requires identification of the traits involved in the reduction of gene flow among populations and understanding of the evolutionary forces that have acted on these traits.

When looking at the numerous studies conducted in the past few decades on speciation, one class of traits appears to play a fundamental role in premating isolation: the sensory systems and the stimuli they receive (which we will call ‘signals’ in this text). A sensory system consists of sensory receptors, neural pathways and parts of the brain that allow the integration of the information mediated by signals (visual, acoustic, tactile or chemical) present in the environment and possibly the initiation of a behavioural response to these signals (for example, discrimination, preference). In the literature, the role of sensory processes in speciation has been mostly illustrated by examples of ‘behavioural isolation’, which involves the divergence of a specific sensory system: the mate recognition system (for example, flycatchers: Saetre et al., 1997; cichlids: Seehausen and Alphen, 1998; Hawaiian crickets: Shaw and Parsons, 2002; house mice: Smadja et al., 2004). However, the role of sensory systems in speciation is not restricted to behavioural isolation. Other modalities of premating isolation can involve the divergence of sensory traits (sensory systems and recognition signals): ‘pollinator isolation’ among angiosperm species due to the divergence of floral traits that promotes pollinator shifts (for example, floral colour divergence in the monkey flowers: Schemske and Bradshaw, 1999); ‘habitat isolation’ due to the divergence of host or habitat preferences (for example, apple maggot flies: Feder et al., 1994). Although habitat and pollinator isolation both involve behavioural change, we will use the term behavioural isolation to refer more narrowly to traits directly affecting species recognition or mate choice. Modalities of ‘sensory speciation’ are thus diverse, according to the type of recognition event (among conspecifics as in mate recognition; with another species as in insect/host plant or plant/pollinator interactions) and the type of sensory system involved (visual, acoustic, chemosensory and so on; Figure 1). Compiling information across taxa and studies is necessary to assess the relative importance and the common features of these types of speciation processes. Most of the reviews that have been published so far have concerned behavioural isolation (for example, Butlin and Ritchie, 1994; Lambert and Spencer, 1995; Ritchie and Philips, 1998; Boake, 2002; Etges, 2002; Coyne and Orr, 2004; Ritchie, 2007), hence focusing on one type of behaviour (mate choice) but compiling examples across different types of sensory systems (Figure 1A). The present review addresses the role of sensory systems in speciation from a different perspective. We analyse the role of one type of sensory system in speciation, chemosensory systems, considering all types of behaviour that can be involved in premating isolation (chemosensory isolation; Figure 1B). To restrict our review, this article will not consider the chemical signalling involved in postmating prezygotic isolation (gamete/gamete or semen/reproductive-tract recognition; see Palumbi, this issue, for example) or chemical recognition among unicellular organisms or viruses, although we recognize the potential importance of these signals in speciation. Behavioural changes may often be the initial triggers for population divergence (Mayr, 1946; Butlin and Ritchie, 1994), and altered recognition, processing and response to chemical cues are expected to be involved in many behavioural changes. Consequently, understanding the role and relationship of chemosensory evolution to behaviour is likely to be fertile ground for understanding speciation. Chemosensory traits have been receiving considerable attention during the past decade but, so far, their role in speciation has not been reviewed specifically. We will use the term ‘chemosensory traits’ to include both signalling and sensory components of chemosensory systems and the terms ‘chemosensory isolation’ or ‘chemosensory speciation’ for premating reproductive isolation or speciation involving these traits.

Figure 1
figure 1

Modalities of ‘sensory speciation’. This figure represents the different premating isolation modalities involving a divergence of sensory recognition systems. In columns, the different behaviours that can be involved in premating isolation are indicated; in rows, the different types of sensory systems potentially implicated in these different behaviours are indicated. ‘Sensory isolation’ modalities can be classified either by the premating barrier involved (for example, behavioural, habitat, pollinator isolation) or by the type of sensory system involved (for example, visual, chemosensory, auditory isolation). Light grey box (A) shows previous reviews that focused on behavioural isolation. Dark grey box (B) shows the present review that focuses on chemosensory isolation.

Chemosensory systems, that is, olfactory and gustatory systems, are nowadays considered to be the most ubiquitous of all sensory systems (Ache and Young, 2005; Penn, 2006). Recent behavioural studies confirm this view. Although the critical importance of chemically based behaviours has been widely recognized in insects and rodents (reviewed in Wyatt, 2003), outstanding examples are now emerging in which odours regulate key life-history processes in various other taxa (for example, birds (Antarctic prions): Bonadonna and Nevitt, 2004; hermit crabs: Gherardi and Tiedemann, 2004; rotifers: Kotani et al., 2001; fish (sticklebacks): McLennan, 2004). Understanding how these complex chemosensory traits evolve is, therefore, a major challenge for biologists. At the intraspecific level, many studies have addressed the phenotypic and genetic characterization of odour-based behaviours (for example, moths: Löfstedt, 1993; Drosophila: Anholt and Mackay, 2001; mice: Brennan, 2004; Brennan and Keverne, 2004; for more examples, see Cardé and Minks, 1997). In comparison, relatively few studies have specifically investigated the evolution of chemically based behaviour at the suprapopulation level and its implications for speciation. Potentially, there are multiple layers of chemosensory change likely to be involved in behavioural shifts associated with premating isolation. First, the physiological changes can be due to alterations in peripheral signal input and/or processing of information by the central nervous system. Moreover, for example in insects, there can be changes affecting long range recognition or short range behaviours, changes affecting the gustatory or the olfactory systems. All these different possibilities interact ecologically and provide opportunities for evolutionary change leading to reproductive isolation. Here, we review some representative studies documenting chemically based speciation in an attempt to evaluate the importance and the modalities of chemosensory isolation in nature and gain insights into the genetic basis and the mechanisms of such divergence.

These evolutionary questions are addressed in the context of considerable recent progress in the molecular characterization of chemosensory systems and their stimuli. First identified in rats (Buck and Axel, 1991) and Drosophila (Clyne et al., 1999), olfactory and gustatory receptor genes are now being characterized in a range of species as genome sequence data become available (Bargmann, 2006). As a result, the genomics of chemosensory systems can now help to unravel molecular features of chemoreception and to trace the mechanisms of chemical perception from molecules to behaviour (see for review in insects: Rutzler and Zwiebel, 2005; Benton, 2006; Hallem et al., 2006; mammals: Dulac and Torello, 2003; and across phyla: Mombaerts, 1999; Firestein, 2001; Matsunami and Amrein, 2003; Ache and Young, 2005). By linking chemistry and physiology at one end with ecology and evolution at the other, ‘chemogenomics’ provides new opportunities to dissect the genetic basis of complex behaviour (Fitzpatrick et al., 2005; Kurtovic et al., 2007) and the functional genetic variation that underlies adaptation and reproductive isolation (Moyle, 2005; Clark, 2006; Noor and Feder, 2006; Storz and Hoekstra, 2007).

Chemosensory traits in speciation

Of moths and flies

When dealing with speciation based on chemicals, two groups of insects provide the classical examples: moths (Insecta: Lepidoptera) and drosophilid flies (Insecta: Diptera; all examples are described in Table 1).

Table 1 Examples of chemosensory speciation

In most moth species, mate finding involves long-distance signalling by female-emitted sex pheromones. The great diversity of pheromone structures used by moth species, and the extensive radiation seen in moths, led to consideration of the role of these pheromones in the speciation process (Linn and Roelofs, 1995; Greenfield, 2002). Closely related taxa often use the same major component, but the pheromones of related species can differ in the complement of minor components in the blend, or in the ratio of components produced (for example, specific ratios of geometric isomers usually of the major component). Numerous studies have analysed the production of specific pheromone blends by the females as well as the detection of the signal (electrophysiological antennal responses) and the induced behavioural responses in males (attraction to pheromone blends) among closely related moth species. In most cases, interspecific divergence in female pheromones correlates with a stronger attraction of males to conspecific signals. As examples, this has been shown in species of the genus Yponomeuta (Löfstedt et al., 1991), Spodoptera (Monti et al., 1997) and Heliothis (Groot et al., 2006; see Table 1, with other examples). Assessing the variation of these systems within species is fundamental if one wants to understand how they have evolved during the process of speciation. Some studies have shown divergence in pheromone blends and responses among populations of a given species (Table 1; for example, Agrotis segetum: LaForest et al., 1997; Lymantria monacha: Gries et al., 2001; Hemileuca eglanterina: McElfresh and Millar, 2001; Zeiraphera diniana: Emelianov et al., 2001). The best example is probably the case of two sympatric races of the European corn borer, Ostrinia nubilalis, in which pheromonal divergence is believed to cause partial behavioural isolation (Table 2).

Table 2 The European corn borer, Ostrinia nubilalis

In flies of the genus Drosophila, courtship and mating behaviours can involve visual and acoustic signals, and also olfactory and gustatory signals (reviewed, for example, in Markow and O'Grady, 2005). Chemical communication during courtship is thought to be mediated by the hydrocarbons found in the adult epicuticle (cuticular hydrocarbons; CHCs) that function as contact signals. CHCs can exhibit a remarkable degree of variability: they can differ in chain length, in the presence or absence of double bonds and in the position of the double bonds. Each individual produces a blend of CHCs and the characteristics of this blend can vary genetically as well as with age, sex, diet and geographic origin within a species (Ferveur, 2005). Some blends show diurnal rhythms (Kent et al., 2007). Analyses of divergence in CHC blend among species suggest a role of CHCs in species recognition and speciation in Drosophila (for example, in Drosophila mojavensis group; Etges and Jackson, 2001). In the Drosophila melanogaster group, closely related species fall into two pairs with respect to CHCs: D. simulans and D. mauritiana are sexually monomorphic (with 7-tricosene as the predominant cuticular compound) whereas D. melanogaster and D sechellia are sexually dimorphic (males: 7-T in melanogaster, 7-T+6-T in sechellia; females: 7,11-heptacosadiene; Jallon and David, 1987). Asymmetric reproductive isolation occurs among these species: males of sexually dimorphic species will court females of all species, whereas monomorphic males will only court conspecific females. Experiments involving the transfer of CHCs from one species to the other demonstrated that sexual isolation in this group is largely caused by differences in female CHCs (Coyne et al., 1994; Coyne, 1996a). Other pairs of closely related species show quantitative and/or qualitative differences in CHC blend and associated premating isolation (Table 1; for example, D. virilis and D. novamexicana: Doi et al., 1996; D. serrata and D. birchii: Howard et al., 2003; D. santomea and D. yakuba: Mas and Jallon, 2005 for preliminary results). Between D. pseudoobscura and D. persimilis, the fixed difference in CHC composition has not been shown to play a role in the sexual isolation of these species (Noor and Coyne, 1996), but the reinforced mating discrimination between the two taxa in sympatry seems to rely on olfaction (Ortiz-Barrientos et al., 2004). Other studies demonstrate between-population variation within a given species, as in D. mojavensis (Etges and Ahrens, 2001). Within D. melanogaster, divergence in CHC blend is observed between African and Caribbean populations (whose major CHC is 5,9-heptacosadiene) and ‘cosmopolitan’ populations from elsewhere in the world (whose major CHC is 7,11-HD). This divergence potentially underlies partial sexual isolation among populations of this species (Table 3).

Table 3 Drosophila melanogaster

These studies on moths and Drosophila underline how chemical signalling systems involving either long-range pheromones or CHCs often underlie behavioural isolation among closely related insect species. However, even in such systems that have received particular attention, it has to be noted that establishing the correspondence between assortative mating and the signals on which assortment is based is not straightforward. In D. melanogaster, uncertainty remains on the role of divergence in CHC blends (5,9-HD versus 7,11-HD) in assortative mating between Z and M races (see Table 3). In O. nubilalis, the divergence in long-range pheromones has long been considered the key determinant of assortative mating among races. However, a recent study revealed that assortative mating does not rely solely on assortative meeting by long-range pheromones, but also involves short-range signals used when sexual partners are in contact (Pelozuelo et al., 2007; see Table 2).

Enlarging our view on chemosensory speciation

Beyond studies on moths and Drosophila, we report in Table 1 some representative studies that have examined patterns of speciation based on chemosensory traits in various species. We have selected these examples on the basis of one main common feature: they all involve the differential recognition, and subsequently preference, for differentiated chemical signals, this preferential interaction between a given type of receiver and a corresponding type of chemical signal leading to premating isolation among populations or species. This compilation allows us to enlarge our view of chemosensory speciation and to evaluate the possible modalities of chemosensory isolation in nature.

Modalities of chemosensory isolation

The first information that emerges from these examples is that chemosensory speciation can involve different forms of premating reproductive isolation: mate choice, host choice or pollination. Therefore, we have classified these examples of chemosensory speciation according to the type of isolation in which chemoreception processes are involved (Table 1).

Chemosensory behavioural isolation

The examples previously described in moths and Drosophila species all referred to behavioural isolation, the mechanism by which isolation occurs through a reduction in attraction and/or probability of mating following contact among heterospecific individuals during the breeding season (Coyne and Orr, 2004). To date, chemosensory behavioural isolation has been studied much more widely than other modes of chemosensory isolation (Table 1). However, chemosensory traits can also be involved in other forms of premating isolation, all in the general category of ecological isolation.

Chemosensory habitat isolation

In animals, habitat isolation can limit reproductive encounters among heterospecific compared to conspecific individuals, premating isolation occurring as a byproduct of adaptation to different habitats (Coyne and Orr, 2004). Although many studies have documented habitat isolation by host shift, mainly in phytophagous insects (reviewed in Berlocher and Feder, 2002; Dres and Mallet, 2002), few of them have identified the cues underlying host preference. For those that have, it is clear that individuals recognize their host mainly through chemical stimuli. Examples of chemosensory habitat isolation are largely restricted to host-specific insects that recognize their host plant through chemicals emitted by the plant. In two sympatric host races of the pea aphid (Acyrthosiphon pisum) that are highly specialized on alfalfa and red clover (Via, 1999; Caillaud and Via, 2000), chemoreception seems to play a fundamental role in host acceptance and there is growing evidence that plant volatiles are important for host location (Park and Hardie, 2003). One of the most intensively studied cases is the habitat isolation among sympatric host races of the apple maggot fly, Rhagoletis pomonella, in which differential preference for host plant volatiles has been behaviourally and physiologically characterized (Table 4). Finally, some species that have been shown to use chemical cues in species recognition also recognize their host plant through chemosensory mechanisms (D. sechellia specialized on Morinda fruit, reviewed in McBride, 2007; O. nubilalis races on maize or mugwort/hop: Bethenod et al., 2005; larch budmoth races on larch or cembran pine: Emelianov et al., 2001; D. mojavensis: Newby and Etges, 1998). Another potential case concerns the homing behaviour of salmon. Generally, homing is precise and fidelity to the natal site results in reproductive isolation of spawning populations and in specialized adaptations of these populations for their natal habitat (Hendry, 2001). We know that homing is governed by olfactory recognition of home stream water (Dittman and Quinn, 1996). This information suggests that chemosensory habitat isolation could drive speciation in salmon, but this possibility has not been directly tested.

Table 4 Rhagoletis pomonella

Chemosensory pollinator isolation

In flowering plants pollinated by specific pollinators, pollinator shifts can reduce the relative amount of heterospecific pollen reaching the stigma (Grant, 1994; Cozzolino and Widmer, 2005; Rieseberg and Willis, 2007). Floral odour is an important trait used by many plants to attract pollinators. For instance, in sexually deceptive orchids, the pollination mechanism is based on mimicry of the sex pheromones released by the female insect and it has been speculated that the highly specialized nature of pollination by sexual deception has led to adaptive radiation accompanied by changes in floral scent (Ayasse et al., 2000). Schiestl and Ayasse (2002) analysed the composition of floral odour and its olfactory detection by the pollinators in two sympatric sexually deceptive orchid species (Ophrys fusca and Ophrys bilunulata) and demonstrated that divergence in biologically active odour compounds in plants is responsible for pollinator isolation. Other examples of chemosensory pollinator isolation concern sympatric species of the genus Gymnademia and Silene (Table 1). We also know that pollinating fig wasps are attracted specifically to volatile chemicals emitted by receptive figs (for example, Grison-Pigé et al., 2002) and that non-pollinating fig wasps locate their host through the recognition of fig chemicals (Proffit et al., 2007). However, investigations in these species-rich systems have not specifically addressed the role of chemical recognition in affecting reproductive isolation among either wasp species or fig species.

These examples show that the modalities of chemosensory speciation are diverse, the divergence affecting signals and behavioural responses being displayed either by individuals of the same species (mate recognition) or by different protagonists of a parasitic or mutualist interaction (for example, plant/insect). All these mechanisms have in common the evolution of chemosensory systems: olfaction is a candidate chemosensory process for sexual behaviour (long-range pheromones), host localization and pollinator attraction whereas gustation is expected to be involved in short-range recognition between sexual partners and for host acceptance.

Diversity of taxa influenced by chemosensory speciation

The second inference from our review is the wide diversity of taxa involved. Although the data are heavily biased towards Drosophila and lepidopteran species, the phenomenon of speciation mediated by chemicals is documented in species belonging to various taxonomic groups, from mammals to plants (Table 1 for a complete list). Given that, as noted before, insects frequently employ chemical signals during courtship or host choice, it is not surprising that chemosensory speciation is documented in many insect species (for example, bees: Vereecken et al., 2007; beetles: Peterson et al., 2007; walking sticks: Nosil et al., 2007; other references in Table 1), including some species primarily known for their use of acoustic cues in mate and species recognition (for example, meadow grasshoppers: Buckley et al., 2003). Along these lines, divergence in CHCs has recently been shown among Hawaiian cricket species (Mullen et al., 2007), although there is no evidence yet that CHCs serve as pheromones in these species or that this divergence could account for sexual isolation. There are likely to be many other examples of chemosensory speciation in insects: for instance in mosquitoes, in which a high degree of variation in chemical signals is known and used for taxonomy, speciation is largely unexplored (but see Caputo et al., 2007). We found only one example in other groups of invertebrates: in a marine polychaete (Neanthes acuminata), in which aggressive behaviour among populations, inducing premating isolation, relies on the olfactory detection of ‘strangers’ (Sutton et al., 2005). There must surely be many others because it is known that arachnids, crustaceans and molluscs, among many others, use chemical cues in both habitat and mate choice. Many parasites must also use chemical cues in host location and so may be subject to chemosensory speciation through habitat isolation. At present, we have no studies in these taxa that have directly addressed the role of chemical recognition in reproductive isolation.

In vertebrates, olfactory-based behavioural isolation has been demonstrated in some species of rodents (mainly in mole rats: Nevo et al., 1976; house mice: Smadja et al., 2004), squamates (sea snakes: Shine et al., 2002; red-sided garter snakes: Lemaster and Mason, 2003; wall lizards: Martin and Lopez, 2006) and fishes (for example, cichlids: Plenderleith et al., 2005; sticklebacks: Rafferty and Boughman, 2006; see Table 1). However, given the widespread interest in other aspects of olfactory communication in rodents (Johnston, 2003; Brennan and Keverne, 2004), it is surprising how few studies have addressed the role of olfactory divergence in rodent speciation. Although some studies on rodents have demonstrated species-specific odours, very few have really tested the role of such species divergence in olfactory-based reproductive isolation (for example, Apps et al., 1990). Conversely, although the most famous examples of speciation in fishes refer to visual cues (for example, three-spine sticklebacks: Rundle and Schluter, 1998; African cichlids: Seehausen and Alphen, 1998), olfaction also plays a significant role in speciation in some fish species. A recent example shows that adaptive divergence in MHC alleles influences odour-mediated mate choice and thus could have led to reproductive isolation among closely related African cichlid species (Blais, 2007).

Chemical characterization of the signals involved in divergence allows the qualitative and quantitative comparison among blends emitted by different populations or species. Characterization requires the collection and purification of the secreted chemicals and their identification by gas chromatography and sometimes mass spectrometry. Bioassays then allow the categorization of active compounds (Wyatt, 2003). Among examples of chemical reproductive isolation, chemical characterization has been achieved mainly in insects, reptiles and plants, in which methods are well established (for example, Gries et al., 2001; Schiestl and Ayasse, 2002; Lemaster and Mason, 2003). In the other groups (rodents, fish, annelids), experiments during which individuals are presented with olfactory stimuli only (urinary stimuli for rodents; water perfumed with odours for aquatic animals) have demonstrated olfactory mate preferences (for example, Smadja et al., 2004; Sutton et al., 2005; Rafferty and Boughman, 2006). In the house mouse, the use of habituation techniques has allowed an indirect assessment of the extent of divergence in urinary signals within and between two subspecies (Smadja and Ganem, 2008). However, the specific odorant compounds involved in species or population recognition in these groups have not yet been characterized (but see Menzies et al., 1992).

Conclusion and perspectives

Studies on interspecific divergence among isolated taxa can reveal the potential role of chemical divergence in speciation. However, studies on taxa still exchanging genes in nature (Table 1) may be better models of changes contributing to the process of speciation because many differences among species may have accumulated after the speciation event and therefore be incidental to the process. Most of the existing studies have been performed in the laboratory and more studies in natural conditions are needed if one wants to understand how these chemicals act in a ‘noisy’ environment (for example, in the presence of chemicals released by other species). Speciation involving the evolution of chemosensory traits has been documented in a range of taxa, despite the taxonomic bias towards lepidopteran and dipteran insects. Given the number of species that seem to use chemical cues not only in mate choice (Wyatt, 2003; Johansson and Jones, 2007) but also in host choice and pollinator attraction, chemosensory speciation is probably still largely unexplored. Chemical characterization of signals has been achieved in some cases, allowing the subsequent exploration of the mechanisms of signal divergence (see below). However, chemical characterization remains a challenge for the more complex olfactory signals (for example, urinary compounds).

Evolution of chemosensory premating isolation

Given the role that chemosensory traits can play in speciation, one of the main challenges is to understand which evolutionary forces act on chemosensory traits to promote divergence among populations in diverse geographical contexts (allopatry, parapatry and sympatry). Although genetic drift alone can cause divergence, it is slow (Wu, 1985) and can only operate when extrinsic factors result in a low rate of gene flow. Therefore, selection is expected to play a role in the evolution of premating barriers but the form of selection is uncertain (ecological adaptation inadvertently or pleiotropically causing isolation, direct selection for isolation, sexual selection and so on) and interactions with drift may be important, for example following founder events or under Fisherian sexual selection.

Natural selection is expected to play a major role in the evolution of habitat isolation: adaptive differences in habitat use in allopatry or disruptive selection for resource use in the presence of gene flow can incidentally lead to reproductive isolation among populations (Coyne and Orr, 2004). Where local adaptation selects for habitat preferences (for example, Fry, 2003), it can have a more direct impact on the evolution of habitat isolation. The most widely accepted scenario for the evolution of pollinator isolation involves geographic separation of plant populations, divergence in floral traits then occurring as a result of disruptive selective pressures exerted by pollinators: for instance, plant populations experiencing completely different groups of pollinators or experiencing very different pollinator's preferences due to changes of the chemical environment (Coyne and Orr, 2004). Finally, the evolution of behavioural isolation has long been debated. The high specificity between the two components of the mate recognition system suggests that the signal/response channel should be highly canalized, with strong selection pressures against novel signals and preferences that would decrease the ability of individuals to obtain matings (Paterson, 1985; Lambert and Spencer, 1995; Greenfield, 2002). This view has led to the prediction that divergence in mating signals is more likely to occur through major shifts than through adaptive changes in small steps. However, it is now clear that variation in the mate recognition system commonly occurs within species and that selection acting on mate recognition systems can promote divergence of these traits among populations. Sexual selection is a force that acts specifically on mating systems and that can lead to accelerated diversification in mating preferences and signals within and among populations (reviewed in Ritchie, 2007), although the other types of selection can also be responsible for the evolution of behavioural barriers (for example, selection against hybridization, selection for species recognition, ecological selection and so on Coyne and Orr, 2004). Here, we review studies addressing the evolution of divergence in chemosensory traits (phenotypic and genetic studies) and ask what these studies tell us about the mechanisms of chemical divergence and more generally about the evolution of premating barriers.

Inferences from phenotypic studies

Indirect evidence from geographical patterns of divergence in natural populations

By analysing patterns of geographical variation in a mating signal or preference among different populations of the same species, some studies have provided arguments in favour of the role of natural selection in the divergence of chemosensory traits. Gries et al. (2001) suggested that divergence in a mating signal in L. monacha populations from eastern Asia, relative to Central European populations, could have resulted from interspecific competition with the coseasonal Lymantria fumida. Similarly, the two pheromone types observed in Hemileuca eglanterina may have resulted from a reproductive character displacement (that is, greater divergence in a mating signal in a contact zone than in allopatry) in H. eglanterina populations in contact with a related species, H. nuttali (McElfresh and Millar, 2001). As these sympatric species do not hybridize in nature, it is unlikely that reinforcement (that is, selection against hybridization) has caused reproductive character displacement in these cases. Nevertheless, these examples suggest that communication interference among sympatric species that use similar premating signals can generate selective pressures on mating signals and lead to their divergence. Direct tests of this hypothesis are needed.

Other studies have analysed variation in chemical mating signals and preferences among populations of a single species that first diverged in allopatry and then met in secondary contact areas where they can still hybridize. These studies show patterns of reproductive character displacement affecting mating signals and/or mate preferences in various species: for example, between D. pseudoobscura and D. persimilis (Noor, 1995), between two subspecies of the house mouse (Smadja and Ganem, 2005, 2008), between the leaf-beetle species Chrysochus cobaltinus and Chrysochus auratus (Peterson et al., 2005a) or among walking-stick populations (Nosil et al., 2003). Evidence for reproductive character displacement, coupled with evidence for lower fitness of hybrids in the contact areas, argues in favour of reinforcement: selection against hybridization may have favoured divergence in mating signals and/or preference in contact zone populations to reduce the cost of production of unfit hybrid offspring.

These examples demonstrate, first, that variation in mate recognition system components can occur within a species' range. Second, they suggest a role for selective mechanisms (reinforcement, interspecific competition) in divergence among populations. Selection for efficient transmission of signals in different environments, known to play a role in the evolution of acoustic signals (for example, Ryan et al., 1990), might also influence the divergence of chemical signals (Linn and Roelofs, 1995) although it has not been extensively addressed, as far as we know. However, one issue with analysing geographical variation is that it can only provide arguments in favour of this role and not a direct demonstration. Many alternative scenarios can explain patterns such as reproductive character displacement and it is difficult to distinguish the alternatives with observational approaches (for a review, see Noor, 1999; Coyne and Orr, 2004).

Direct evidence from experimental approaches

In rare cases, experimental approaches have been used to evaluate the role of selection in the evolution of chemosensory speciation. Higgie et al. (2000) used experimental sympatry to demonstrate the role of natural selection in the generation of a field pattern of signal variation in D. serrata. Field populations of D. serrata display reproductive character displacement in CHCs when sympatric with D. birchii. By exposing field-sympatric and -allopatric populations of D. serrata to experimental sympatry with D. birchii for nine generations, they showed that CHCs of field-allopatric D. serrata populations evolved to resemble the field-sympatric populations, whereas field-sympatric D. serrata populations remained unchanged. Their experiment was one of the first to directly demonstrate that natural selection on mate recognition can result in a pattern of reproductive character displacement.

On the same species, Rundle et al. (2005) also used an experimental evolution approach—involving an ancestral laboratory environment and two novel resource environments—to evaluate the role of divergent selection in the evolution of female mating preferences. They found that CHCs evolved in response to the new environments and demonstrated that female mating preferences for these same CHCs also diverged among populations under different environmental treatments. This direct experimental manipulation showed that divergence in mating preferences for CHCs among D. serrata populations, expected to be an important source of premating isolation, can evolve, at least in part, as a byproduct of adaptation to the environment.

Finally, another study addressed experimentally the evolution of pheromone divergence between Heliothis virescens (Hv) and Heliothis subflexa (Hs). Although previous studies (Gries et al., 2001; McElfresh and Millar, 2001) had only demonstrated the pattern of reproductive character displacement among moth species, Groot et al. (2006) conducted experiments to determine whether communication interference from males of these closely related species could exert strong enough directional selection to cause evolution of chemical mating signals in the face of stabilizing selection by conspecifics. Specifically, they first tested whether interspecific communication interference could be a directional selection force: they introduced quantitative trait locus (QTL) for low production of the acetate compound (typical of Hv female pheromone blend) into an Hs genetic background and measured, in field and cage experiments, the capacity of modified Hs females to attract and mate with Hs and Hv males. They showed that Hv males are more attracted to modified Hs females (with low acetates) than normal Hs females, suggesting that Hv males could have exerted directional selection on Hs females to produce relatively high amounts of acetates and so to avoid interference with the other species. They estimated that the directional selection exerted by Hv males on Hs females to produce relatively high amounts of acetates can range from 0.135 to 0.231, demonstrating that interspecific selection may be intense enough to have counteracted intraspecific stabilizing selection and led to diversification of sexual signals.

Phenotypic studies have sometimes been successful in suggesting or more directly demonstrating the role of selective pressures in the evolution of divergence in chemosensory traits. However, all these studies refer to behavioural isolation. Moreover, evidence from patterns of variation in natural populations can only provide arguments in favour of a mechanism, whereas experimental approaches are limited to a few model species. This could explain why there are very few systems in which both chemical signals used in mate choice and the evolutionary forces influencing sexual isolation among species are known. In the following section, we will see how knowledge on the genetic basis of chemosensory traits can provide complementary information on the mechanisms of evolution of these traits, ideally up to direct tests for signatures of selection at the genomic level.

Inferences from genetic studies

Table 5 summarizes the genetic analyses that have been performed on premating barriers involving chemosensory traits, all of them involving insect species. Most of these genetic studies refer to behavioural isolation, some of them to habitat isolation and none to pollinator isolation. The genes underlying divergence in signals and preferences are likely to be distinct, even in the case of mate recognition systems (Butlin and Ritchie, 1989). As they have often been studied separately (which is unfortunate because they are expected to co-evolve), we present below, in separate sections, the information obtained so far on signals and preferences.

Table 5 Genetic analyses of chemically mediated premating isolation

Most of the studies addressing the genetic basis of chemosensory speciation have used the tools of ‘classical’ genetics (Noor and Feder, 2006). This approach attempts to identify the number, distribution and type of genes that contribute to phenotypes that prevent gene flow among species, with a particular attention on the role of sex-linked genes and of ‘major genes’ (that is, single genes of large effect) in isolation. These methods have evolved to narrow down the search, ideally to the identification of specific genes involved in isolation (for an extensive review of these methods, see Ritchie and Philips, 1998; Coyne and Orr, 2004).

However, an alternative route to the identification of genes involved in chemosensory speciation is now available because of the growth of genomic approaches. Gene families involved in chemical signal production, chemoreception and processing have been described and they can be examined for loci that might contribute to divergence among populations or species. Clearly this candidate-gene approach can be particularly powerful to address the evolutionary genetics of chemosensory speciation when it is combined with classical genetic analysis (Joshi, 2005; Noor and Feder, 2006). Here we begin by reviewing the studies addressing genetics of divergence in chemical signals and in preferences and then describe the first fruits of candidate-gene approaches.

Genetics of divergence in chemical signals

Studies that have been published on the genetic basis of signal divergence concern exclusively mating signals in insects (Table 5, light grey cells). One group of studies has demonstrated that single genes of large effect can be involved in causing signal divergence among closely related taxa. Changes at single loci underlie differences in pheromone blend among noctuid moth species (Monti et al., 1997), races of European corn borers (Roelofs et al., 1987; Dopman et al., 2004) and Ostrinia species (Tabata and Ishikawa, 2005; Table 5). Moreover, an experiment on the cabbage looper (Trichoplusia ni) demonstrated the possible monogenic origin of pheromone divergence in the laboratory: in the course of a study involving collections of volatiles released by females, individuals releasing a new pheromone blend were discovered and one recessive autosomal gene has been shown to be involved in producing the mutant blend (reviewed in Haynes, 1997). When available, information on the biosynthetic pathway of these pheromone compounds helped to understand the mechanisms by which single genetic changes have led to divergence of signals among taxa. In O. nubilalis, the Z/E isomeric mixture is controlled by one reduction step at the end of the pathway, and the difference in the ratio of these isomers between the two races is due to a single change in the activity of the reductase controlling this final step (Roelofs et al., 1987). In T. ni, the single mutation affects enzymes controlling chain shortening of precursors, this modification generating the three major differences observed between the mutant and normal pheromone blends (Linn et al., 1997).

Several studies have also underlined the role of desaturase enzymes, which produce unsaturated fatty-acid precursors, in the emergence of new olfactory signals. The first striking example is the difference in CHC profiles between the two D. melanogaster pheromone races that has been shown to result from one mutation at a desaturase locus (desat2; Dallerac et al., 2000; Takahashi et al., 2001). A subsequent experiment, using site-directed gene replacement, later confirmed that the two alleles of desat2 have a large effect on female CHC profile (Greenberg et al., 2003; but see Table 3 for a complete discussion). In this case, divergence in olfactory signals results from the inactivation of a desaturase enzyme. Research on the desaturases in various moth species revealed that another way to produce a change in pheromone blend is by activation of a non-functional desaturase gene transcript present in the pheromone gland (Roelofs and Rooney, 2003): for instance, pheromonal differences between O. nubilalis and its Asian relative Ostrinia furnacalis result from the activation of an ancestral desaturase gene in O. furnacalis, found to be inactivated in all other Ostrinia species (Roelofs et al., 2002). From these examples on destaturases, it is clear that the change occurred in a single step whereas large differences that map to a single locus, but have not been characterized at the molecular level, could, in principle, have accumulated by a series of substitutions of small effect.

Two conclusions can be drawn from these examples. (1) Signal divergence can occur through major shifts: single genetic changes can be responsible not only for simple signal differences, such as a ratio of isomer components (O. nubilalis, D. melanogaster), but also for alterations in multiple component blends (Spodoptera, T. ni) or qualitative differences (O. nubilalis/furnacalis; T. ni). (2) It seems that major shifts in pheromone composition are most likely to occur when the different compounds that compose the pheromone blend are products of a single biosynthetic pathway: although this system allows tight control over the production of the specific blend, a single mutation at one step of the biosynthetic pathway can lead to dramatic change in the pheromone composition (Linn et al., 1997). Given this possibility for major shifts in chemical signals to be caused by single mutations, it is critical to understand how such mutations can spread when they are generally expected to reduce mate attraction. So far, the model of ‘asymmetric tracking’ offers a partial response: it predicts that a large mutational effect in female pheromone production can be tracked by male responses that are pre-adapted to a broader signal range (Phelan, 1992; Löfstedt, 1993). This prediction agrees with some observations: in T. ni, the male's behavioural response specificity in the mutant line has changed in response to the female's pheromone blend mutation (Haynes, 1997); in O. nubilalis, a few males are attracted to pheromones of O. furnacalis females, suggesting that some males were pre-adapted to respond to a novel pheromone (Roelofs et al., 2002). However, this model does not explain how a new female pheromone can become fixed when only a few males respond to it. Neither is it clear why males respond to a much broader range of signals than is produced by females.

A second group of studies has shown that more than one genetic change (poly- or oligogenic basis) can account for signal divergence among taxa. Three sets of crosses have been performed between species pairs of the D. melanogaster subgroup involving a dimorphic species (either D. melanogaster or D. sechellia, where males predominantly produce monoenes, and only females have dienes) and a monomorphic species (either D. simulans or D. mauritiana, where both sexes produce 7-monoenes). In each cross, the variation of female-specific CHCs segregated with chromosome 3, and at least five non-overlapping genetic factors were detected on its left arm (3L; Coyne et al., 1994; Coyne, 1996b; Coyne and Charlesworth, 1997). The genetic basis underlying the variation of the male-predominant monoene is different: the ratio between 7-T (in D. simulans) and 6-T (in D. sechellia; Cobb and Jallon, 1990) depends on factors dispersed on the five major chromosomal arms (Coyne, 1996a; Civetta and Cantor, 2003). The two species D. pseudoobscura and D. persimilis show a major difference in the ratio of methyl-hexacosane and (Z,Z)-5,9 heptacosadiene. This variation mostly depends on epistatic interactions between the X- and the second chromosomes, in both sexes (Noor and Coyne, 1996). Finally, quantitative variation of the principal CHC, (Z)-11-pentacosene, between a genetically marked strain of D. virilis and D. novamexicana segregated with two of the six chromosomes and one major genetic factor was mapped (Doi et al., 1996; for a review of Drosophila work, see Ferveur, 2005; Gleason et al., 2005). This polygenic basis is also found for female differences in long-range pheromone compounds between H. virescens and H. subflexa (Groot et al., 2004; Sheck et al., 2006).

A poly- or oligogenic basis can suggest that several small changes (gradual evolution) are responsible for signal divergence in these species. From studies on moths, it appears that intrapopulation or geographical variation in pheromone blend is more likely in species that use combinations of components whose biosynthetic pathways are not linked and therefore are more difficult to regulate (for example, A. segetum populations; Ctenopseustis obliquata/herana: Linn and Roelofs, 1995). This variation within the signal channel can provide a source upon which either drift or selection (sexual or natural) can act and promote divergence among populations. Therefore, this genetic architecture is at least consistent with phenotypic observations suggesting that selection can act to promote gradual divergence in chemosensory traits that contribute to reproductive isolation. However, it has to be noted that all these examples refer to species that are already completely isolated, suggesting that these multiple genetic changes could have accumulated after the speciation event and thus might not be the cause of initial isolation. More generally, the distinction between a major gene and polygene is sometimes difficult to define. Several genetic changes can be responsible for the divergence of a trait (here signals) but a few QTLs of large effect, or identified genes, can explain much of the divergence, the ‘polygenic’ effect then becoming a major gene effect (for example, Gleason et al., 2005).

Overall, the results of studies that have addressed the genetic basis of chemical signals suggest that divergence in a chemical mating signal can occur either by major shifts or more gradual changes. These different mechanisms may reflect constraints imposed by the biosynthetic pathways for producing and regulating more or less complex blends of components (Linn and Roelofs, 1995). A growing number of studies now suggest that desaturase genes play a central role in mating signal divergence in various insect species. In moths and flies, the evolution of these genes seems to have been influenced by gene duplication, gene loss and pseudogene formation, these processes being characteristic of the birth-and-death model of multigene family evolution (Roelofs and Rooney, 2003; Greenberg et al., 2006a). As genome sequences become available in many more insect species, the development of genomic studies on desaturase genes is likely to be an important route to gain insights into the evolution of chemical signals and their role in insect speciation. In other groups, such as vertebrates, the genetic basis of pheromone divergence remains to be addressed, as well as the genetic basis of chemical signal divergence in plants.

Genetics of divergence in chemosensory preferences

Several studies have analysed the genetic basis of divergence in chemosensory preferences (involved in mate or habitat choice) that lead to premating isolation in insects (Table 5, dark grey cells). As with mating signals, the results indicate that the genetic basis of divergence in mating preference can be either simple or more complex: several genetic changes have been shown to be responsible for divergence in mate preference between D. simulans/D. sechellia, D. melanogaster Z and M races, D. santomea/D. yakuba and D. virilis/D. novamexicana, whereas difference in mate preference between the two O. nubilalis pheromone races and among Ctenopseustis species map to only one sex-linked locus (for Drosophila species, see Table 5; Takahashi and Ting, 2004; Markow and O'Grady, 2005). QTL studies addressing the genetic basis of adaptation to different host plants in D. sechellia, apple maggot flies and pea aphids also showed that divergence in host preference relies on a few genetic changes (Table 5). Therefore, it seems that divergence in a complex chemosensory behavioural trait can have a relatively simple genetic basis, as was previously suggested by Ritchie and Philips (1998), but there are still not enough comparable cases to find a general pattern and so to make comparisons with other sensory systems. Beyond providing some information on the genetic architecture of behavioural divergence, some of these studies also helped to understand how this divergence may have evolved among taxa that are still exchanging genes in nature. A pair of studies investigated the genetic basis of variation within D. pseudoobscura for female mating discrimination that is enhanced by natural selection in areas of contact with D. persimilis (Ortiz-Barrientos et al., 2004; Ortiz-Barrientos and Noor, 2005). QTLs responsible for reinforced mating discrimination mapped to candidate genes involved in olfaction. Intriguingly, the results indicated that one of these QTLs (called Coy-2) behaves as a one-allele assortative mating locus, that is, a locus where assortative mating is enhanced by increase in the same allele in both populations (Felsenstein, 1981), hence providing a plausible explanation for reinforcement of preferences despite gene flow between the two species. Similarly, the demonstration in the pea aphid that QTLs with antagonistic effects on performance on the two hosts are genetically linked to QTLs responsible for host acceptance suggests that this type of genetic architecture (close linkage or pleiotropy) has facilitated both the evolution of specialization and reproductive isolation among these sympatric populations (Hawthorne and Via, 2001; Via and Hawthorne, 2002).

Studies such as these provide important basic information on the number and location of genetic factors underlying changes in chemosensory behaviour, but the tools of classical genetics have rarely been successful in identifying specific genes at the molecular level. Therefore, it has not been possible to assess patterns of molecular evolution that may accompany evolution of host shifts or assortative mating.

Candidate-gene approach to chemosensory speciation

Since the identification of the chemoreceptor gene repertoire of D. melanogaster (Robertson et al., 2003), several studies have exploited this new opportunity to decipher the molecular and physiological basis of signal integration (for example, Dekker et al., 2006; Kurtovic et al., 2007) and the evolutionary dynamics of chemoreceptor genes (Guo and Kim, 2007; Nozawa and Nei, 2007; Tunstall et al., 2007). It is likely that the genes underlying chemosensory habitat and mate preferences that cause premating isolation are to be found among these chemoreceptor-encoding and chemoreceptor-related genes. We report, in Table 5, two studies that have addressed the genetic basis of host specialization in D. sechellia, relative to its generalist sibling species, using a gene-based approach based on publicly available chemosensory gene sequences. McBride (2007) examined the molecular evolution of D. sechellia's entire suite of olfactory (Or) and gustatory (Gr) receptor genes, presumably including those involved in the specific adaptation of this species to Morinda fruit. Having identified the entire repertoire of functional and non-functional Or and Gr genes in D. sechellia and D. simulans, she found that D. sechellia has lost Or and Gr genes nearly 10 times faster than its generalist sibling and that those receptors that remain functional in D. sechellia have fixed amino-acid replacement mutations at a consistently higher rate, relative to silent mutations, than their D. simulans orthologs. Thus, this study characterized the potential genetic signature of host specialization in an insect chemosensory system, the changes at Or and Gr loci being likely to reflect positive selection and/or relaxed constraint associated with the altered ecological niche of this fly. In the same year, Matsuo et al. (2007), by developing expression analysis and targeted mutagenesis experiments, identified two genes encoding odorant binding proteins, Obp57d and Obp57e, which are responsible for the behavioural differences between D. sechellia and its sibling species in their responses to the toxins contained in the ripe fruit of Morinda citrifolia.

These two pioneering studies underline the potential of the genomic approach: the identification of the genes underlying interspecific differences in chemosensory behaviours, among the whole repertoire of chemosensory genes, can help to gain insights into the evolutionary dynamics of these genes, providing evidence for selective constraint or response to positive selection within particular lineages for example, and thus can contribute to our understanding of the evolution of host adaptation, mate preferences and reproductive barriers. However, these approaches also have their limitations. It is dangerous to attribute differences between a single pair of species, such as D. simulans and D. sechellia, to a cause such as host specialization because there are many other potential causes amongst the differences in their ecology and demographic history. Gardiner et al. (2008) considered one alternative explanation in their analysis of the Or and Gr repertoires of the 12 Drosophila species for which genome sequences are now available. This group includes three host-specialist species (D. sechellia, D. erecta and D. mojavensis). An analysis of the proportion of pseudogenes (representing the rate of functional gene loss) showed that it is strongly correlated with genome size and differs slightly between island-endemic and continental species but does not differ between generalists and specialists. The general problem is that genomic data provide a great deal of information about a few species whereas comparative analysis across multiple taxa is needed to test hypotheses about association with environmental factors. This is true for inferences of selection as well as for genomic patterns such as rates of gene loss. It is also important to note that detection of the signature of a history of positive selection in gene sequences says nothing about the source of selection and this may be difficult to investigate because selection may not be operating in extant populations.

Future directions

Given the availability of genome sequences and the identification of chemosensory genes in a growing number of species (in insects: for example, mosquitoes, moths, honey bees, beetles; in other groups: for example, daphnia, rodents, fishes, platypus), gene-based approaches can now be developed in various biological models, providing new opportunities to address the genetic basis of olfactory and gustatory preferences and of speciation. As chemoreceptors appear to evolve very rapidly (for example, Robertson and Wanner, 2006), it is not always possible to identify their coding sequences by sequencing them directly using other species' primers. However, at least in insects, chemoreceptors share common molecular features across species that allow the use of sequence and structural homology to identify them in new species genomes as genome sequencing is achieved (for example, Krieger et al., 2004; Bohbot et al., 2007). Thus, more non-model genomes are needed to address the molecular basis of chemosensory speciation and more generally of chemically based behaviours. Making the most of gene-based approaches requires combining them with classical genetic tools and, especially, with ecological and behavioural studies that can elucidate the selective factors acting on signals and preferences.

Moreover, the identification of genes underlying chemosensory preferences should be viewed as a source of candidate genes that can be investigated at the population level to assess variation within and among natural populations and their contribution to interspecific differences. Ideally, chemoreceptor genes should not be studied in isolation but in concert with candidate signal production loci such as the desaturase genes. In both cases, there are indications that gene family evolution, for example through gene duplication and loss, may be important as also the selection on individual loci. Do the different families, as well as individual loci within them, co-evolve during divergence and speciation? The availability of candidate loci also makes it possible to ask to what extent the evolution of species differences has involved changes in regulatory rather than structural genes.

A focus on receptor genes should not cause us to forget that divergence in chemosensory preference can rely on genetic changes in neurophysiological pathways. A series of studies on R. pomonella has addressed the physiological basis of divergence in olfactory preferences, showing that variability in peripheral sensitivity and temporal firing pattern could influence host preference and contribute to host fidelity and sympatric host shifts in the Rhagoletis complex (Olsson et al., 2006a, 2006c). Such physiological studies should be developed in other species and comparative analyses undertaken to assess the relative role of receptor and neurophysiological changes in the evolution and divergence of chemosensory preferences.

General conclusion

The evolution of chemosensory systems and chemical signals plays an important role in the occurrence of premating isolating barriers in various taxa, even in those primarily known to use other types of recognition cues. Even though these cryptic and labile signals are often difficult to analyse and their divergence difficult to quantify, chemosensory speciation should be investigated more broadly. Given the growing availability of genomic data related to chemosensory traits, there are great new opportunities for the detailed dissection of the genetics of this specific speciation modality, up to the identification of genes or pathways that contribute to speciation. Especially in model species like Drosophila and moths, full understanding of the mechanisms underlying chemosensory speciation will require ecological and behavioural studies of forces driving divergence in addition to these genomic studies. It will be important to develop comparative studies as well as try to fill some of the current big gaps in coverage, both taxonomic and for forms of isolation other than ‘behavioural’. Moreover, our review focused on cognitive processes but chemosensory speciation is probably not restricted to this: speciation in microorganisms like fungi (Le Gac et al., 2007; Giraud et al., 2008), bacteria (Dettman et al., 2007) or viruses (Duffy et al., 2007) might involve chemical recognition mechanisms as well. Other sensory traits—visual, acoustic—are involved in speciation mechanisms. Comparative studies on the different cues involved in premating isolation could test the relative importance of chemically based mechanisms in speciation, but only when a wider range of species has been studied. Given the probable difference in the genetic and neurophysiological characteristics of these different sensory systems and signals, it would be interesting to go further in the comparison of these different ‘sensory speciation’ modalities and try to make specific predictions on speciation according to the system involved.