Introduction

Over the 225 million years of mammalian evolutionary history, modification of tooth size and associated size variation is a pattern commonly observed in many evolutionary lineages. Dental characters seem to be partly non-independent (Kangas et al. 2004; Harjunmaa et al. 2014), and size and shape changes can be strongly channeled in the course of these evolutionary radiations. Tinkering with pre-existing developmental programs (Salazar-Ciudad and Jernvall, 2010) appears to be one of the main mechanisms (Harjunmaa et al. 2014) of this channeling, leading to numerous examples of parallel evolution (e.g., Charles et al. 2013; Rodrigues et al. 2013), and extreme cases of tooth loss followed by reversal in some lineages (Gingerich, 1977).

At the population level, variation in tooth size is common, especially in distal molars. For instance, in 20% of the human population, only some of the third molars develop, and in 0.1% six or more permanent teeth are lacking (Lan et al. 2014). Tooth formation disorders may appear sporadically, as non-syndromic familial forms or within larger syndromes (Klein et al. 2013). Hypodontia and supernumerary teeth are associated respectively with smaller or greater than average tooth size, while missing teeth are most often the most distal in the morphogenetic field (Brook et al. 2014). In mice, where the dental formula is reduced to only three molars and one incisor per quadrant, the proportion of missing third molars observed is similar to that found in human populations. Likewise, the same association of tooth agenesis with tooth size is observed in some inbred strains (Grüneberg, 1951). Mutations in several genes coding for signaling molecules, receptors or transcription factors have been associated with familial non-syndromic hypodontia (van den Boogaard et al. 2012; Thesleff, 2014). Nonetheless, no tooth-specific regulatory genes have been identified, suggesting that the same conserved regulatory repertoire is used in the development of other organs, which could explain the frequent dental defects found in more general clinical syndromes (Thesleff, 2014).

Developmental biologists have shown that posterior molars originate from successive dental laminae, extending from the preceding tooth, and probably containing progenitor cells initiating tooth development with dental placode formation (Thesleff, 2014). Previously initiated molars seem to express inhibitors balancing mesenchymal activators (Jernvall and Thesleff, 2012), a phenomenon that has been proposed as an Inhibitory Cascade model (IC) to predict molar proportions (Kavanagh et al. 2007), although some objections have been raised regarding the uncritical use of this model (Hlusko et al. 2016). This model has received considerable attention in evolutionary biology (e.g., Renvoisé et al. 2009; Labonne et al. 2012; Halliday and Goswani, 2013; Carter and Worthington, 2016; Evans et al. 2016), and has been generalized as a shared developmental rule for segmented organ systems, such as limbs, vertebrae/somites and phalanges (Young et al. 2015). For mammalian teeth, IC appears to be plesiomorphic, and this developmental bias must have acted on mammal diversification since the early stages, so that the many exceptions to the rule are probably secondarily derived states (Halliday and Goswani, 2013).

Several candidates, Bmp, Activin A, Eda, and Pax9, were initially proposed to be the activators in the IC model (Kavanagh et al. 2007). Based on experimental data and on a computational model including spatial patterning of teeth, a negative feedback loop of Wnt has been proposed as the underlying mechanism, with Shh as a mediator, Sostdc1 as an inhibitor (Cho et al. 2011). This model provides a hypothetical general reaction-diffusion mechanism controlling spatial patterning (Cho et al. 2011). The genetics of this activation/inhibition balance remains nonetheless open (Jernvall and Thesleff, 2012), though it may potentially be a major driver of non-syndromic sporadic hypodontia and supernumerary teeth (Lan et al. 2014). The existence of loci interacting with gene products and thus directly modifying the activation/inhibition balance is an important aspect of IC genetics. However, this piece of evidence is missing from the existing literature. Such loci, named relationship QTL (rQTL), have been identified for allometric relationships between long bones (Cheverud et al. 2004; Pavlicev et al. 2008), but not yet for teeth or other segmented structures. Better understanding of the evolutionary relevance of this balance will be obtained through the validation of such loci. Models show that rQTLs may enhance organismal evolvability by facilitating the alignment of new variation to selection gradients, by generating developmentally channeled variation (Pavlicev et al. 2011). This theoretical model predicts both higher and lower correlations among traits, depending on whether or not they are under the same directional selection (Pavlicev et al. 2011). Such a pattern of correlations is found in teeth, where a reduction of integration between lower and upper molars along the row may be observed in some groups, related to functional constraints of occlusion and mastication and the decreasing role played by teeth along the row in such functions (Gómez-Robles and Polly, 2012). Validating rQTLs will provide an understanding of how a developmental system such as the IC can be modified to release variation, leading to the individuation of parts, and to their divergence according to independent selection regimes (Wagner, 1996). Thus, basing IC genetics on rQTLs will provide a causal mechanism explaining the exceptions observed so far in several macroevolutionary surveys (e.g., Renvoisé et al. 2009; Labonne et al. 2012).

In this study, we demonstrate that, with appropriate statistical modeling and careful phenotyping, it is possible to further improve our understanding of IC genetics by studying standing variation in a population. Here, we developed a computational pipeline to extract the 3D size of all upper (maxillary) and lower (mandibular) molars from high-resolution microCT scans accurately and effectively in a large mouse backcross. We explicitly integrate the IC model of the mouse dentition into our QTL mapping, searching for loci that affect the relationship between successive teeth, as a proxy for the activation/inhibition balance.

Materials and methods

Experimental design and 3D imaging

Three C57BL/6J (B) males and three A/J (A) females were used to derive an F1 generation backcrossed to A males and females. The A (♀) × F1 (♂) backcrosses produced 163 offspring (84 females and 79 males) and the reciprocal F1 (♀) × A (♂) crosses produced 270 offspring (128 females and 142 males). All 433 animals were sacrificed at postnatal day 28. Third molars were all fully erupted. A set of 882 informative SNPs were obtained from a commercial panel. All animal protocols were approved by the University of Washington’s Institutional Animal Care and Use Committee.

All animals were imaged at the Small ANimal Tomographic Analysis (SANTA) Facility at Seattle Children’s Research Institute, using high-resolution microcomputed tomography (model Skyscan 1076C), employing a standardized imaging protocol (0.5 mm Aluminum filter, 55 kV current, 420 ms exposure, 0.7° rotation steps, 3 frames averaged per rotation). Image stacks were reconstructed at 18 μm spatial resolution. A random set of 79 individuals was segmented using the 3D Slicer (Fedorov et al. 2012) with a specific threshold (71–255). This threshold was chosen to represent a good compromise for selecting crowns and roots of molars and not the surrounding tissue, or the root canal space. Manual segmentations were later used as the gold standard to assess the quality of our atlas-based segmentations. More details about experimental setup, genotyping, and imaging may be found in a related paper (Maga et al. 2015). In the following, the convention M1–M3 refers to the lower (mandibular) molars from mesial to distal, whereas M1–M3 refers to the upper (maxillary) molars.

Molar atlas building

The open-source DRAMMS deformable registration software and atlas-building pipeline (Ou et al. 2011) was used to build individual molar atlases. The registration matches a high dimensional vector of multi-scale and multi-orientation Gabor attributes, and uses mutual saliency to up-weight regions of the volume where correspondences can be reliably established, reducing the negative impact of outlier regions on registration quality (Doshi et al. 2013; Ou et al. 2014; Iglesias and Sabuncu, 2015). The atlas-building pipeline was then run in an unbiased population-registration framework, iteratively finding a virtual space representative of the mean anatomy/geometry of the population (Guimond et al. 2000). Segmented molars from individual atlases were back-projected to the individual samples by reversing the deformation.

Particle-based shape modeling

ShapeWorks, an entropy-based particle distribution system (Oguz et al. 2015) was used to describe the surfaces through dense point clouds, and estimate the centroid size of each molar (Dryden and Mardia, 1998). A total of 1024 particles was used for all but third molars, for which only 512 were used. Particle correspondences between structures were established by optimizing the energy function that balances the negative entropy of the particles on the structure with the positive entropy of the population ensemble (Oguz et al. 2015). Sizes of lower and upper rows were computed as the sum of the individual centroid sizes of each molar in the row. Because randomness is involved in the way particles float on the surface, we ran the same ShapeWorks analysis pipeline twice, to assess the repeatability of the results. The process failed for a few samples, so only 413 individuals (201 females and 212 males) were finally used.

Mapping molar size and relationship QTL

The effect of the molar size QTL at locus j was estimated using Haley–Knott regression (Haley and Knott, 1992) by fitting the general linear model (FULL):

$$y_{ik}|{\mathbf{M}}_i\sim N\\ \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\left( {\mu \,{\mathrm{ + }}\mathop {\sum}\nolimits_c {x_{ic}\beta _c{\mathrm{ + }}\mathop {\sum}\nolimits_{m < k} {y_{im}\beta _m\,{\mathrm{ + }\;}p_{ij}\beta _j\,{\mathrm{ + }}\mathop {\sum}\nolimits_{m < k} {y_{im}p_{ij}\gamma _m,\sigma ^2} } } } \right),$$
(1)

where y ik is the centroid size of the kth molar on the row for the individual i; x ic is the value of the covariate c (sex or direction of the cross); y im is the size of any previously developed molars; p ij is the probability of the QTL genotypes given the flanking markers for the individual i; β is the main effect of either the covariate c, the previously developed molars m, or the genotype j; γ m is the interaction between any previously developed molar and the tested locus j. A set of 16 models was used to map tooth size and row size QTL (Table 1), they were based on the preliminary analyses of the covariate effects (Table 2). For models that involve any previous molars as a covariate, an alternative model where the QTL acts only additively with the covariate was evaluated (ADD). An additional model for the second and third upper and lower molars was also evaluated, corresponding to the cases where no effect of previously developed molars was incorporated (NoCOV). The logarithm of odds (LOD) scores for the ADD, FULL and NoCOV models were obtained from the ratio of residual sum of squares of the model to its corresponding null model (i.e., a model including the same predictors but the QTL). The difference of the LOD score between the ADD model and the FULL model (Eq. 1) provides evidence for the Molar × QTL interaction (INTER).

Table 1 Models used for tooth QTL mapping
Full size table
Table 2 Effect sizes (%) of covariates
Full size table

Genomewide thresholds (5 and 10%) were obtained from the distribution of maximum LOD scores from 100,000 random reshufflings (Churchill and Doerge, 1994). The same permutations were used for both FULL and ADD models, and the significance of the INTER LOD scores was then determined, based on the distribution of their differences (Broman and Sen, 2009). This process identified three types of loci that have an effect on a given molar. The first type is a direct locus that may or may not be pleiotropic on several molars \(\left( {Q_j \to \beta _j\left[ {y_m, \ldots ,y_k} \right]} \right)\). These direct QTLs reach the significance threshold in the ADD model. The second type is an indirect locus, with an effect on one molar through the phenotypic effect of other previously developed molars (\(Q_j \to \beta _jy_m \to \beta _my_k\)). These indirect QTLs reach the significant threshold in the NoCOV model but disappear in the ADD model as their effect is mediated by the molar covariate. Because preceding molars mediate also the effect of sex and of the direction-of-cross, the two models are not nested. Therefore, labeling indirect QTL has relied on this differential detection rather than LOD differences such as used with the INTER test. Finally, the third type, detected from INTER, is a relationship QTL (Cheverud et al. 2004) that controls for genetic variation in the influence of previously developed molars on the size of the focal molar (\(Q_j \times y_m \to \gamma _my_k\)).

Genotype probabilities p ij were computed at each centimorgan using R/qtl (Broman et al. 2003). A genotyping error rate of 10−4 and a Carter–Falconer map function were used, the latter providing a good approximation of the level of crossover interference in mice (Broman et al. 2002). Sex-averaged genetic distances were obtained from the Jackson Laboratories Mouse Converter tool (http://cgd.jax.org/mousemapconverter/), using marker ID and the genetic map reported by Cox et al (2009). The QTL regions were defined by their Bayesian credible intervals (Manichaikul et al. 2006). We queried these regions for genes annotated for abnormal tooth morphology and development, using the Human-Mouse disease connection website (http://www.informatics.jax.org/humanDisease.shtml; Accessed 23 Feb 2016). As the genomes of the two parental strains are already sequenced (Keane et al. 2011), we queried the Mouse Genome Informatics databases to identify SNPs that are polymorphic, for each candidate gene, and for regions 2 kb upstream and downstream.

Results

Automated phenotyping

We used the Dice similarity score, the ratio of intersection of two images to the union of two images, as a metric to assess the quality of our atlas-based segmentations in comparison with our manual segmentations, which we considered to be the gold standard. Atlas-based segmentation performed very well (Supplementary Figs. S1 and S2), with generally good Dice scores (Supplementary Fig. S3). There was overall agreement among molars, although the third molars presented slightly lower scores than the other four: 0.78 for third molars and 0.9 for the others, with a coefficient of variation of about 3 to 5% for the first and second molars and of 7% for the third molars (Supplementary Table S1). The average difference was 6.0 × 10−3 between corresponding lower and upper molars. Upon further inspection, we found that the lower quality for third molars was mainly driven by consistent under-segmentation of distal aspects of molar roots in our manual segmentations.

Apart from the few samples that failed during the particle distribution phase of the ShapeWorks pipeline, centroid sizes for all molars agreed very well between the two separate runs, with correlations of at least 0.9998. The root mean square deviations RMSD were calculated as follows:\(\sqrt {\mathop {\sum}\nolimits_{i = 1}^n {\left( {y_{ik,1} - y_{ik,2}} \right)^2{\mathrm{/}}n} }\), where y ik,1 is the centroid size of the tooth k for the individual i for the first run, with y ik,2 for the second run. These RMSD were low on both upper and lower molars, ranging from 4.0 × 10−4 for M3 to 8.0 × 10−3 for M1 once normalized to the mean, demonstrating the repeatability of our results. Relationships between the individual absolute deviation and the average centroid size were observed only for the first molars (Supplementary Fig. S4). Summary statistics on centroid size of molars and rows are provided in Supplementary Table S2.

Mesial molars explain a large proportion of variance in size

Previous studies using the same backcross demonstrated significant gender and direction of cross (DoC) effects on skull and mandible sizes (Maga et al. 2015; Navarro and Maga, 2016). Such significant effects were also evidenced in early experiments on the quantitative genetics of molar size (Bader, 1965). Here, these effects were also evaluated in the presence of previously developed molars as additive covariates (Table 2). Molar covariates accounted for a large amount of variance in size (45–65%), removing all effects of gender and DoC on maxillary M2 and M3 and of DoC on mandibular M2, demonstrating that the mesial molar mediates these effects. However, M1 and M2 explained considerably less variance (16%) in M3 size.

Pleiotropy, specificity, and compensation of molar QTLs

Overall, association signals on rows were very consistent with the signal on the first molars (Fig. 1). Thirteen QTLs were detected for the upper molars (Table 3) and twelve for the lower molars (Table 4). Between two and six QTLs were detected for each individual molar, with the fewest for the third molars. Loci detected for molar rows were also observed for the first or second molars, except for the locus on chromosome 9. The apparent similarity of the second and third molar mappings with the row or first molar mappings disappeared once some tooth covariates were included in the model (Fig. 1). This inclusion led to the discovery of additional loci (Tables 3 and 4).

Fig. 1: QTL mapping for each individual molar and the two rows
figure 1

Focal molar or row is colorized in green on the 3D dental model represented on the upper right corners. The four upper mapping panels are upper molars (row, M1, M2, M3), and the four lower panels are lower molars (row, M1, M2, M3). Horizontal lines represent 5% genomewide thresholds from 100,000 permutations for each modeling. Gray lines are LOD profiles and 5% genomewide thresholds for the upper (~Sex + DoC + Q) and lower molar rows (∼DoC + Q). Black lines represent results for M1~Sex + DoC + Q, and for M1~Sex + DoC + Q. Green lines represent results of the FULL models for second (M2~M1 + Q + M1 × Q and M2~Sex + M1 + Q + M1 × Q) and third molars (M3~M2 + Q + M2 × Q and M3~DoC + M1 + M2 + Q + M1 × Q + M2 × Q); Maroon lines represent the ADD models (i.e., the four preceding models but without interaction) and light green lines represent the INTER mapping (i.e., only the interaction part of the FULL model). Yellow lines represent the results for the NoCOV models (i.e., the four models but without molar covariates). The NoCOV genomewide threshold is confounded with the ADD threshold at LOD = 3 and therefore not visible. In the NoCOV yellow mapping, both direct and indirect QTLs have an effect. In the ADD maroon mapping, only direct QTLs have an effect. In the FULL green mapping, both direct QTLs and rQTLs have an effect whereas the INTER light green mapping corresponds to the rQTL only (Color online)

Full size image
Table 3 QTL effects and candidate genes for upper molars
Full size table
Table 4 QTL effects and candidate genes for lower molars
Full size table

Thirteen of the 25 QTLs appeared pleiotropic, acting on both lower and upper molars. However, considering the relatively large spread of QTL regions (quartile range: 23.76–58.15 Mb), pleiotropy remains an assumption because we cannot be certain that the same gene is causal over such large intervals. Most of these pleiotropic loci also have an indirect effect on the next molar (Tables 3 and 4). It may be noted that ULS.3 classified as indirect QTL on M2 almost reaches the significance threshold (p = 0.06), suggesting the remaining of some direct effect. In contrast, only two of the twelve specific QTLs (i.e., upper or lower molars) also have an indirect effect. Pleiotropic QTLs have stronger effect sizes than specific QTLs (U = 245.5, p = 0.007). One major pleiotropic QTL detected on chromosome 11 had effect sizes between 6.6 and 11.8%, except on M2. After canceling out its indirect effect, this QTL represents only 1.3% of M2 variance. Specific QTLs show effect sizes between 1 and 3%.

Two QTLs having an effect on M2 or M3 on chromosome 7 cancel out in our backcross at row level because of opposite signs: the B allele (i.e., C57BL6/J allele) reduces the size of M2 but increases the size of M3 (Fig. 2). This inverse effect leads to a compensatory phenomenon equalizing the lower row size and canceling out any association signal at row level (Fig. 1).

Fig. 2: Additive effect trace plot along chromosome 7 for M2 and M3 centroid sizes and the centroid sizes of the two possible genotypes at the two QTLs
figure 2

Envelopes and error bars are standard errors of the mean. For the left panel, centroid size is defined conditional on the covariate included in the additive model (M2~Sex + M1, M3~DoC + M1 + M2). Genotype A relates A/J strain and B relates C57BL6/J strain

Full size image

QTL affecting the relationship between molars

The consideration of a FULL model including a QTL × Mm<k term strongly increased the genomewide significance thresholds (Fig. 1). Based on the specific threshold for INTER, we identified three rQTLs that have significant interaction (Table 3, p < 0.05) or marginally significant (Table 4, p < 0.09) with the sizes of previously developed molars. These rQTLs were detected only for the third molars. Relaxing the threshold does not help in detection of rQTL on second molars. More generally, the LOD profile of INTER on second molars did not show any evidence for such loci (Fig. 1). The effects of rQTLs are all negative with the B allele, reducing the effect of the previously developed molar, except for the smallest interaction, ULS.5 × M2, which is positive (Tables 3 and 4).

Genomic variation and candidate genes

Most QTLs contained one or more annotated genes, and only five contained none (Tables 3 and 4). All pleiotropic QTLs contained annotated genes. For all but four annotated regions, at least one of the candidate genes had polymorphic SNPs between the parental strains. Only Dspp, which is implied in odontoblast differentiation, has a coding non-synonymous SNP. Genes Bcl11b, Bmpr1a, Fst, Grem2, Lrp4, Mmp14, Msx2, Nog, Rspo2, Slc39a13, and Sostdc1 have SNPs in mRNA-UTR regions. Noggin, the only candidate gene overlapping with the major QTL on chromosome 11, also presents polymorphic SNPs in a CpG island and 14 SNPs in a birectional lincRNA (Gm2018) located 391 bp to the gene. Sonic hedgehog (Shh) has mostly intronic SNPs but presents five SNPs in three CpG islands and 20 in its antisense long non-coding RNA (lincRNA Gm26894). Similarly, Fst also presents six polymorphic SNPs in two CpG islands within its antisense lincRNA (Gm41074). It is important to note that Grem2 was assigned to US.1 instead of ULS.1, even though the confidence interval of US.1 stops 2 Mb before this gene while the large confidence interval of ULS.1 contains it, because Grem2 remains closer in position to US.1 (7.5 Mb instead of 63.7 Mb). It should also be noted that Pax9, which is known to arrest molar development depending on position in the morphogenetic field (Kist et al. 2005), is found close to ULS.5, although it is not contained in its statistical confidence interval for M2.

Discussion

Overall, the genetic determinants of molar size variation appear somewhat similar for lower and upper jaws. These results temper previously suggested differences between rows in the genetic regulation of tooth size, based either on Eda mutants (Charles et al. 2009) or on the genetic mapping of length and width of the first molars using the SMXA recombinant inbred panel (Shimizu et al. 2004), which also includes A/J as one of the two parental strains. The major QTL segregating in our backcross (ULS.4) appears in the exact position of Noggin, a BMP antagonist, expressed from the bud stage pattern (Hu et al. 2012). Overexpression of Noggin is known to arrest the formation of all lower molars as well as M3 and sometimes M2 (Plikus et al. 2005), by reducing the rate of cell proliferation and down regulating Cyclin-D1 (Wang et al. 2012). The ULS.4 QTL has a direct effect only on M1 for maxillary teeth and on both M1 and M2 for mandibular teeth, which seems consistent with the more widespread effect of Noggin on mandibular molars (Plikus et al. 2005). Alternatively, this pattern may be related to a higher functional redundancy of BMP antagonists for upper than for lower molars, which may maintain BMP homeostasis (Hu et al. 2012). Such redundancy of Noggin, Chordin, and Gremlin has been proposed for teeth, based on their co-expression pattern (Hu et al. 2012). The US.1 QTL discovered here is very close to Grem2 and has a direct positive effect on the maxillary M3. Only one QTL, LS.3, presents a direct effect on M3. This locus seems to be related to Fuz, which is located at 4.6 Mb from the peak of the QTL. This gene is related to planar cell polarity signaling and plays a crucial role to balance Wnt and Hedgehog pathways in craniofacial development (Zhang et al. 2011).

A previous study of QTL mapping for the lower row centroid size using the LG/J × SM/J cross found two QTLs on chromosome 7 and one on chromosome 14 (Workman et al. 2002). These results were not replicated here for the lower row. However, QTLs having an effect on M2 or M3 seem to overlap with the chromosome 7 QTL detected in this other cross at a similar position, but cancel out in our backcross because of opposite signs. Due to the rather low genetic resolution of our backcross, we cannot rule out that these two loci are actually one and the same, showing antagonistic pleiotropy. Although there is an obvious link between dentition and fitness, it has repeatedly been claimed that a small variation in size at the individual tooth level may not have an actual effect on fitness (e.g., Bader, 1965), especially here on laboratory strains. Thus, opposite effects on individual tooth size may not always lead to an opposite impact on fitness, but rather to a compensatory mechanism neutralizing variation at row level.

In early growth-field literature based on phenotypic correlations, Van Valen (1962) hypothesized negative interactions between the third and the other two molars, regulating total row size. Here, we found evidence of such interactions (rQTL) at the genetic level. The theoretical model associated with rQTL predicted both higher and lower correlations among traits, depending on whether or not they are under the same directional selection (Pavlicev et al. 2011). Confidence intervals for the three rQTLs identified in our study contain either Sostdc1, Fst, or Shh. Follistatin (Fst, ULS.6) is an important inhibitor of BMP and Activin during morphogenesis (Iemura et al. 1998; McDowall et al. 2008). This gene is essential for enamel knot formation and impacts distal molars more strongly (causing total absence of third molars, and affecting second molars more strongly than first molars), and has a stronger effect on upper than lower molars (Wang et al. 2004). Here, the apparent effect of ULS.6, direct on M2 and M3, indirect on M3, and null on the M1 and M1, seems consistent with observations from tissue culture experiments (Wang et al. 2004). There is also good empirical support for the two other rQTLs, ULS.5 and US.3, which contain Shh and Sostdc1, respectively. These two genes have been considered as inhibitors of the Bmp signaling (Kassai et al. 2005; Harjunmaa et al. 2014). They have recently been shown to interact (Ahn et al. 2010), and are implied in a negative feedback loop with Wnt (Cho et al. 2011). These authors suggested that this loop is a central mechanism controlling spatial patterning of the molar row. The agreement with our candidate genes behind these two rQTLs, reducing the inhibitory effect of developing molars on the third molars, suggests that this negative feedback loop provides a mechanism by which variation in the Inhibitory Cascade may be produced, and on which natural selection may act. By producing variation in row patterning, the rQTL mechanism may lead to some secondarily derived states of the IC model, as found in diverse mammalian lineages (for example in rodents, Renvoisé et al. 2009; Labonne et al. 2012).

Polymorphic SNPs between the parental strains at the candidate genes for the three rQTLs appear to reside in mRNA-UTR regions, CpG islands, or in the antisense lincRNA of these genes. This result underlines the potentially regulatory nature of the causal variants, though further work is needed for confirmation. This assumption agrees with the recent survey on rQTLs, which concluded that their causal molecular variants seem to be more often cis-regulatory elements than variants associated with the protein-coding genes with which they interact (Pavlicev et al. 2013).

We showed that template-based segmentation of 3D volumetric scans, such as the skulls of inbred mice, is a reliable and effective procedure to obtain 3D size and shape information in studies where a large number of samples is necessary to detect loci of modest effect size. In this study, we considered the total size of molars as our phenotypic unit of analysis. Developmentally, root formation is a separate process, and happens after crown formation. In future studies, templates that distinguish molar crowns from roots could be generated to perform more detailed analyses so as to assess the strength of the association solely for crown or root.

In conclusion, despite their pervasive role in development and probable strong canalization, major developmental genes seem to explain both the main variation in tooth size and the control of the activation/inhibition balance between successively developing molars. Our results support a mechanism producing heritable variation of the Inhibitory Cascade in mouse molars. This result, when combined with the theory of rQTLs, which predicts a channeling of new variation enhancing organismal evolvability, can provide an evolutionary mechanism to explain the great diversity of mammalian tooth sizes and the variety of their relationships.

Data archiving

Genotypes and landmarks are available as an R dataset from the Dryad directory https://doi.org/10.5061/dryad.bm770. Original high-resolution CT images of specimens and segmented molar templates are publicly available through the Genetics of Craniofacial Shape in Mus project housed at the Open Science Framework (https://osf.io/w4wvg/).