Introduction

Social interactions are common in nature, and other individuals are usually the most important part of the environment experienced by an individual (Wolf 2003; Frank 2007). The environment created by social partners through actions, such as competition or cooperation, is referred to as the social environment. Variation in the quality of the social environment may originate partly from genetic variation in the social partners, which would make the social environment heritable (Wolf et al. 1998). The classical example of a heritable environment is the one provided by a mother to her offspring in mammals (Dickerson 1947; Willham 1963; Falconer 1965; Kirkpatrick and Lande 1989; Cheverud 2003).

In the traditional quantitative genetic model, the phenotype of an individual is the sum of the direct effect of its own genes (DGE) and an environmental effect. However, because the environmental effect includes a component due to the social environment, the phenotype of an individual is also a function of the genes of its social partners. The heritable effect of a social partner on the trait value of the focal individual is known as an Indirect Genetic Effect (IGE) (Griffing 1967). IGEs have consequences for trait values and fitness of individuals that interact, and subsequently for the course of the evolutionary processes (e.g., Hamilton 1964a; Moore et al. 1997; Wolf et al. 1998).

In the field of animal breeding, interest in social interactions has increased in recent decades, as both theoretical and empirical studies show that not only fitness but also trait values of individuals can be affected by genes of other individuals (Muir 2005; Bijma et al. 2007a, b). IGEs have been studied in both animal and plant populations, and in a number of those studies social interactions contributed substantially to heritable variation in the trait (reviewed by Ellen et al. 2014). Well-known cases of IGEs in domestic animals include cannibalistic behavior in laying hens, which causes mortality (Muir 1996; Ellen et al. 2008), competition and tail biting in pigs, which is associated with poorer growth (Arango et al. 2005; Camerlink et al. 2013, 2014; Bergsma et al. 2013), and aggression and competition in fish species such as Nile tilapia and Atlantic cod, which reduces growth (Nielsen et al. 2014; Khaw et al. 2016).

In addition to the effects of social interactions on trait values, it has been observed in aquaculture populations that competition for feed and formation of social hierarchies also inflates trait variability (Jobling 1995; Cutts et al. 1998; Hart and Salvanes 2000). Because this pattern is so evident, variability in body weight among individuals has become a standard measure of the degree of competition in aquaculture; the degree of competition is measured by the coefficient of variation (CV) of body weight, where a high CV indicates strong inter-individual competition (Jobling 1995). In farmed fish populations, the CV is usually between 20 and 60 % (Gjedrem 2000; Ponzoni et al. 2005; Gjedrem and Baranski 2009), which suggests moderate to strong competition.

Indications of a close relationship between competition and variability are also coming from the field of plant breeding, where breeders have successfully improved productivity of crops by selecting, partly unintentionally, less competitive phenotypes, which also resulted in more uniform crops (Donald 1968; Austin et al. 1980; Denison et al. 2003). Moreover, the connection between yield, competition, and variability has also been made in game theory, where it was shown that the lowest competition and highest yield is achieved when plants are phenotypically uniform (Zhang et al. 1999). Hence, in plants, there is clear evidence of a genetic relationship, where reduced competition leads to less variability and higher yield.

The variability of trait values of a genotype, measured either repeatedly on the same individual, or on multiple individuals belonging to the same family, has been studied as a quantitative trait in its own right. This trait is often referred to as “inherited variability” or “heritable variation in environmental variance” (SanCristobal-Gaudy et al. 1998; Mulder et al. 2008; Hill and Mulder 2010). The study of variability has been a part of quantitative genetics for several decades already, but it has gained particular attention in recent years due to the development of new methods to estimate genetic variance in variability (SanCristobal-Gaudy et al. 1998; Sorensen and Waagepetersen 2003; Mulder et al. 2009; Rönnegård et al. 2010) and substantial empirical evidence for a genetic basis of variability in livestock, aquaculture, and laboratory populations (reviewed by Hill and Mulder 2010). In several fish populations, for example, it has been found that variability of body weight has a large genetic component (Janhunen et al. 2012; Sonesson et al. 2013; Khaw et al. 2015; Sae-Lim et al. 2015a, b; Marjanovic et al. 2016). However, despite the clear relationship between competition and variability observed at the phenotypic level, inherited variability has not been connected to competition in quantitative genetic models.

As social interactions are often a source of IGEs, the observed relationship between competition and variability on the phenotypic level (Jobling 1995; Cutts et al. 1998; Hart and Salvanes 2000; Denison et al. 2003) strongly suggests an underlying genetic relationship between the two phenomena. At present, little is known of this genetic relationship, both in plants and animals, which may be due to a lack of quantitative genetic models that connect both phenomena. On the one hand, current quantitative genetic models of inherited variability ignore social interactions, since they treat variability as a trait of the focal individual only, ignoring the contribution of social partners. On the other hand, standard IGE-models cannot explain the relationship between competition and variability, since phenotypic variance is independent of the level of IGEs in those models. However, by ignoring IGEs, we may be overlooking an important component of heritable variation in trait variability.

The joint study of IGEs and inherited variability could help us understand observations from animal and plant breeding, and possibly enable utilization of genetic variation that has so far been untapped. In addition, it may bring new insight in mechanisms of canalization or insensitivity of individuals to genetic and environmental changes (Waddington 1942), and broaden our understanding of phenotypic evolution. Therefore, a joint study of IGEs and variability could make a significant contribution to the field of quantitative genetics, and its applications in animal and plant breeding and in evolutionary biology.

As a first step towards unraveling the genetic relationship between social interactions and inherited variability, we present a quantitative genetic model that integrates both phenomena. We use Monte Carlo simulation to evaluate the behavior of the model, and demonstrate that the model mimics the co-evolution of social interactions and variability observed in phenotypic studies.

Theory

Model

The genetics of socially affected traits can be studied using two approaches; variance component models or trait-based models (McGlothlin and Brodie 2009; Bijma 2014). In variance component models, the individual phenotype is divided into a direct genetic component originating from the focal individual, and an indirect genetic component originating from its social partner (Griffing 1967). In this approach, it is not needed to know which traits are causing the IGE. Instead, DGEs and IGEs are estimated as random effects using linear mixed models and information on genetic relationships between individuals (Muir 2005; Bijma et al. 2007b). See Table 1 for notation.

Table 1 Notation key

The trait-based models, in contrast, define IGEs on the phenotype of the focal individual as a function of trait values of its social partners (Moore et al. 1997; Wolf et al. 1998; Bijma 2014). In this case, the traits causing the indirect effects need to be identified. When interaction is between two individuals, and the target trait and the trait causing the IGE, also known as the “effector trait”, are the same, the trait-based model can be written as (Moore et al. 1997)

$$P_i = A_i + e_i + \psi P_j,$$
(1)

where Pi is the phenotypic value of the focal individual i, Ai is the additive genetic effect originating from the focal individual, Pj is the phenotypic value of its social partner j, ψ is the “regression coefficient” of Pi on Pj, and ei is a residual. (With feedback, i.e., when trait levels of interacting individuals are reciprocally affected, ψ is not a true regression coefficient; see Bijma 2014). We will use this model and observations from aquaculture as a starting point to draw a connection between IGEs and inherited variability.

Phenotypic studies in aquaculture suggest that the behavior of a fish towards its social partners depends on its size relative to that of its partners, where larger fish are usually dominant and aggressive, while smaller fish are subordinate and submissive (Doyle and Talbot 1986; Huntingford et al. 2012). In anemonefish, for example, large individuals are dominant members of social groups and display aggressive behavior towards subordinates (Fricke and Fricke 1977; Iwata et al. 2008). Similarly, Oscars (cichlid fish, Astronotus ocellatus) chase and attack smaller conspecifics, but avoid larger individuals (Beeching 2010). Difference in body weight, therefore, affects phenotypes of the interacting individuals, with higher body weight giving a competitive advantage to the individual in terms of growth rate. Thus, to account for the competitive effect of body weight on growth rate, we need to model the evolution of body weight over the life of the interacting individuals.

Therefore, we developed a basic quantitative genetic model involving interactions of two individuals. In this model, our target trait is growth rate between time point t−1 and t, while the effector trait is the difference in body weight between the individuals that interact at the previous time point t−1. The change in body weight, i.e., growth rate, of the focal individual is a function of genetic and environmental effects of the focal individual itself on its growth rate, and of the difference in body weight between the social partner and the focal individual, multiplied by a regression coefficient,

$$\begin{array}{l}P_{t,i} - P_{t - 1,i} = \\ \mu _{{{GR}}} + A_{{{GR}},i} + E_{p,{{GR}},i} + E_{t,{{GR}},i} + b_{ij}\left( {P_{t - 1,j} - P_{t - 1,i}} \right)\end{array}$$
(2)

where Pt,i is the body weight of focal individual i at time point t, Pt−1,i is body weight of i at the previous time point, μGR is the mean growth rate of the population, AGR,i is a (direct) breeding value for growth rate of individual i, Ep,GR,i and Et,GR,i are permanent and temporary non-heritable (“environmental”) effects of individual i, and bij is a regression coefficient.

The meaning of b ij

The bij in our model measures the effect of a difference in body weight between the social partner and the focal individual on the growth rate of the focal individual. Hence, the absolute value of bij reflects the strength of the social interaction. When bij is negative, growth rate of individual i is reduced when j has higher body weight than i, indicating competition. Conversely, when bij is positive, growth rate of i is increased when j has higher body weight than i, indicating cooperation, i.e., “helping the one who lags behind” (Box 1). Thus, b is a measure of cooperation; negative b indicates competition, positive b cooperation, and an increase in b an increase of cooperation (i.e., less competition). The model described by Eq. 2 can be written in matrix form for both individuals simultaneously, which may facilitate analysis of the behavior of the model (Appendix A).

Genetic variation in b

Trait-based IGE models usually assume that the “regression coefficient” ψ is constant within a population (Eq. 1). However, several empirical studies that were able to estimate ψ, show that it may differ between genotypes (Kent et al. 2008; Bleakley and Brodie 2009; Chenoweth et al. 2010). Hence, empirical studies suggest that ψ shows genetic variation, and can thus respond to selection. Following this evidence, we allow b to evolve. Therefore, b is not a fixed parameter, but specific for every interacting couple. We propose that heritable variation in b is a result of a direct genetic effect of the focal individual (AD,i), representing resistance to competition, and an indirect genetic effect of its social partner, representing cooperative effect (AI,j). While b is a property of both the focal individual and its social partner, it affects the phenotype of the focal individual only; we will therefore refer to this b as “the b value of the focal individual”. Thus, for focal individual i with social partner j, the regression coefficient bij, i.e., the b value of the focal individual, is given by

$$b_{ij} = \bar b + A_{{{D}},i} + E_{{{D}},i} + A_{{{I}},j} + E_{{{I}},j},$$
(3)

where \(\bar b\) represents the average regression coefficient, which is a population parameter that is negative under competition and positive under cooperation. The AD,i and ED,i are the direct genetic and environmental effect of individual i on bij, while AI,j and EI,j are the indirect genetic and environmental effect of individual j on bij. Appendix B contains extension of Eq. 2 to accommodate larger group size.

Inherited variability

Note that our model does not include an explicit breeding value for inherited variability. Instead, as shown in the section “Simulation” below, genetic variation in variability is an emerging property of the model, resulting from genetic effects of competition, i.e., the direct and indirect breeding values for b. In other words, our model shows that heritable effects on competition result in inherited variability. In the “Discussion” section, we further investigate how breeding values for b correlate with direct and indirect breeding values for inherited variability (see section “Estimating b” below; see also section “Breeding values for b and variability”).

Competition, cooperation, and the sign of \({\bar{\boldsymbol b}}\)

We use the term “competition” to describe the situations where the larger individual continues to increase in size, while the smaller individual lags behind, leading to divergence of their body weights through time (Fig. 1a). This is typical for populations where \(\bar b\) < 0. We use the term “cooperation” to describe the situation where individuals become increasingly similar in body weight over time (Fig. 1b). This occurs when growth rate of the larger individual decreases, while the smaller one catches up. This is typical for populations, where \(\bar b\) > 0.

Fig. 1
figure 1

Expected growth curves of two individuals in a group under competition (a) and cooperation (b)

Asymmetry: b ij vs. b ji

Note that we distinguish between resistance to competition (AD) and cooperativeness (AI), as these may be different properties of an individual. For example, consider the pair i and j in a population showing competition (\(\bar b\) < 0). Suppose that i is very competitive (AI,i < 0) and also resistant to competition (AD,i > 0), while j is very cooperative (AI,j > 0) but very sensitive to competition (AD,j < 0). Then the effect of j on i will be small, while the effect of i on j will be large (Supplementary File 1, grey cells in Table S2). In other words, an individual that is strongly affected by its social partner does not necessarily also have a strong effect on its social partner. Hence, b is non-symmetric, i.e., bijbji.

Simulation

We used Monte Carlo simulation to investigate whether our model (Eq. 2) predicts the empirically observed relationship between competition and variability, and whether methods for selection against competition (e.g. group selection) also result in a reduction of variability. We considered five values for the average value of b\((\bar b)\), to which we refer as scenarios (Table 2).

Table 2 Parameters used in simulation

Negative values of \(\bar b\) correspond to competition (Scenario 1-strong competition; Scenario 2-moderate competition), while positive values reflect cooperation (Scenario 4-moderate cooperation; Scenario 5-strong cooperation). Scenario 3 represents a neutral environment with \(\bar b\) = 0.

The genetic values of all individuals in the population were simulated as inherited from their parents (base population), assuming Mendelian inheritance, while their environmental values were sampled from independent normal distributions. All individuals were randomly assigned to groups of 2 members. Phenotypes were constructed for 10 time points using Eqs. 2 and 3. Average starting weight was 10 g, and average growth rate between time points was also 10 g. Hence, to illustrate the behavior of our model as simple as possible, we considered absolute growth. Obviously, for the analysis of real data, a more biologically realistic growth model, such as relative growth, may be used. With relative growth, the impact of competition may be higher, since individuals may diverge more in scenarios with competition, as larger individuals grow faster, while smaller individuals slower, compared to absolute growth (see Supplementary file 2 for two small simulation examples).

For each scenario, there were 100 replicates. Table 2 contains parameter values used in the simulation. Appendix C contains a detailed description of the simulation procedure.

Relationship between \({\bar{\boldsymbol b}}\) and variability

The relationship between competition and variability generated by our model was assessed at two levels. First, we considered the average within-group variance of body weight at the last time point. Second, we considered the overall phenotypic variance in the entire population. Results are presented in Fig. 2 as averages over 100 replicates.

Fig. 2
figure 2

Variability of body weight in a population and correlation between group mates across five scenarios i.e., five average b values (\(\overline b\)). Variability is expressed as the average within-group variance of body weight of two group mates and as overall phenotypic variance in the whole population

Across the five scenarios, both average within-group variance and phenotypic variance decreased curvilinear with increasing \(\bar b\), i.e., with increasing cooperation (Fig. 2). The average within-group variance ranged from 376.4 g2 (sd, ±14.4 g2) to 20.9 g2 (sd, ±0.7 g2), which is an 18-fold difference in variability of body weight between scenarios 1 and 5. The phenotypic variance ranged from 457.3 g2 (sd, ±15.7 g2) to 95.1 g2 (sd, ±2.6 g2), showing a 5-fold difference in variability between scenarios 1 and 5. These results show that our model results in a relationship between competition (\(\overline b\)) and variability that is also found in real data.

The difference between the average within-group variance and the phenotypic variance is related to the similarity of group mates. Total phenotypic variance is the sum of between- and within-group variance. When group mates are independent and group size equals two, the average within-group variance is half of the phenotypic variance. Average within-group variance, however, was much larger than half of the phenotypic variance in scenarios with negative b, but much smaller in scenarios with positive b. The correlation between group mates is calculated as \(\rho = \frac{{\sigma _b^2 - \sigma _w^2}}{{\sigma _b^2 + \sigma _w^2}}\), where \(\sigma _b^2\) is between group variance and \(\sigma _w^2\) is within-group variance. In scenarios with negative b, the correlation between group mates was negative, which means that group mates were dissimilar in the competitive environment (Fig. 2). When b was positive, correlation between group mates was positive, indicating higher similarity of group mates in the cooperative environment (Fig. 2). For \(\bar b\) = 0, the average within-group variance was approximately one half of the phenotypic variance.

Growth curve patterns in relation to b values

In this section, we look into how variation in b around its average, affects the variability among group mates. Within every scenario (Table 2) \(\bar b\) was the same for all individuals; however, variation in b values of individuals existed due to variation in direct and indirect genetic and environmental components that make up b (Eq. 3). Therefore, in every scenario some groups would have individuals that both have high b values, some groups would have individuals with low b values, and variations in between. We hypothesize that group mates that both have high b values, i.e., that are both cooperative and resistant to competition, grow more uniform compared to those with low b values, i.e., group mates that are both competitive and sensitive to competition.

To illustrate this, we selected groups that have individuals with the highest and the lowest b values for each of the scenarios. An additional condition when selecting groups was that individuals have an initial difference in their body weight of ~2 sd. The growth curves in relation to the level of b values within a group are illustrated in Fig. 3a, d for scenario 1 (\(\bar b\) = −0.08, strong competition) and 5 (\(\bar b\) = +0.08, strong cooperation). Results for scenarios 2–4 are presented in Supplementary File 3. Supplementary file 4 contains b values of individuals from all the scenarios.

Fig. 3
figure 3

Growth curves of two group mates (one larger than the other) that have lowest sum of b’s (a); the initially larger individual has negative b, the smaller one has positive b (b); the initially larger individual has positive b, the smaller one has negative b (c); lowest sum of b’s (d), for scenarios 1 and 5. Each panel shows one typical replicate

In both scenarios 1 and 5, individuals in a group with the low b values differed substantially in their final body weight (Fig. 3a). Individuals with the high b values, however, maintained a similar body weight through time (Fig. 3d), which is in agreement with our hypothesis.

We also looked into groups that had individuals with positive/negative combinations of b values. When the initially larger individual had a negative b value, its body weight increased over time, resulting in a larger size difference between the two group mates, unless the smaller individual had a positive b value, which allowed it to catch up (Fig. 3b). Similarly, the size difference decreased when the larger individual had a positive b value, even when the smaller individual had a negative b value (Fig. 3c). It was also possible to get re-ranking of the individuals, i.e., the smaller individual can become the larger one. This can happen for example when the smaller individual has a high positive b value, while the larger individual has a low negative b value (Scenario 5, Fig. 3b).

Expressions (Appendix A) for the expectation of the difference in the phenotypic values and the variance of this difference at time point T, i.e. E(PT,i − PT,j|bij,bji) and V(PT,i − PT,j|bij,bji), demonstrate that the phenotypic variance within a group is directly related to the sum of b values within the group. The expressions show that the expected difference is zero if there is no initial difference at T = 0, while the variance depends directly on the sum of bij and bji. More details can be found in Appendix A.

Breeding values for b and variability

If a connection between competition and variability exists not only on the phenotypic level but also on genetic level, we should see less variation in body size among the offspring of sires that have positive direct breeding values (AD) for b, as these individuals should be more resistant to competition. This links our model to the definition of inherited variability, where parents with low breeding values for variability have offspring with lower phenotypic variance. Figure 4a indeed shows that the correlation between AD of sires for b and variability of body weight of their offspring is negative, ranging from −0.55 (sd, ±0.07) to −0.20 (sd, ±0.09) across scenarios. This suggests that individuals that are genetically more resistant to competition are less variable. Moreover, offspring of sires with positive indirect breeding values (AI) for b should be less competitive. The group mates of these “social” individuals should therefore show less variability compared to group mates of individuals with negative indirect breeding values for b. In other words, AI of a sire affects the variability of phenotypes of the group mates of his offspring. As expected, Fig. 4b shows negative correlations between AI of sires and variability of the group mates of their offspring. Figure 4a,b also shows a small negative correlation between AI of a sire and variability of his offspring, and between AD of a sire and variability of the group mates of its offspring. This result suggests a second-order effect; for the direct effect, for example, the AD of a sire first affects the trait values of its own offspring, which subsequently affects the variability of their groups mates in the next time period. For standard errors of the correlations see Supplementary file 5.

Fig. 4
figure 4

a Correlation (r) between direct (AD) and indirect (AI) breeding value of a sire and variance of body weight of his offspring (VarPoff) for each of the five scenarios, i.e., average b values (\(\overline b\)). b Correlation (r) between direct (AD) and indirect (AI) breeding value of a sire and variance of body weight of the group mates of its offspring (VarPgm), for each of the five scenarios, i.e., average b values \(\left( {\overline b } \right)\)

Selection

Individual selection has often been used with great success for improvement of livestock and aquaculture traits. However, this type of selection ignores the contribution of IGE which may hamper the improvement of socially affected traits. An alternative strategy is a group selection, which takes indirect genetic effects into account (Griffing 1976).

To see how variability responds to selection, and whether we can capture direct (AD) and indirect genetic effects (AI) for b with existing selection methods, we performed three types of selection: individual selection for body weight, group selection for body weight, and group selection for lower variance of body weight. In all three cases, selection was done using observations from time point 10. With individual selection, the 11 % of the heaviest individuals were selected as parents of the next generation. With group selection for body weight, the individuals from the 11 % of groups with the highest average body weight were selected. With group selection for lower variance, the individuals from the 11 % of groups with the lowest variance in body weight were selected. We illustrate the effect of selection by using base population with \(\bar b\) = −0.08 (Scenario 1—strong competition, Table 2). Selections were performed for 10 generations. Correlations between AD and AI, AD and AGR, and AI and AGR, were all set to 0. See Appendix C for further details. Figure 5 presents the results as averages over 100 replicates. For standard errors see Supplementary file 6.

Fig. 5
figure 5

Effect of three types of selection on average within-group variance (a), average b (\(\overline b\)) (b), average direct breeding value for b (c), average indirect breeding value for b (d), average breeding value for growth (e), and average body weight (f), in the population

Individual selection increased mean body weight (Fig. 5f), but also decreased AD (Fig. 5c) and AI (Fig. 5d), causing an increase in variability in the population (Fig. 5a). In other words, individual selection increased variability.

Both types of group selection increased AI (Fig. 5d), suggesting that group selection at least partially exploited genetic differences in indirect genetic effects on b. Variability of body weight decreased when group selection was made on variance, but increased slightly when group selection was for average body weight, however much less compared to individual selection (Fig. 5a). This increase in variability with group selection for average body weight originated from a decrease in AD. With group selection on variance, in contrast, AD increased (Fig. 5c). Group selection on the variance, therefore, captured direct and indirect genetic effects on b better than group selection on the average body weight. Group selection on the variance did not change mean body weight, because the correlations between AD and AGR, and AI and AGR were zero (Fig. 5f). Group selection for average body weight, on the other hand, increased mean body weight in magnitude similar to individual selection (Fig. 5f).

Discussion

We have proposed a quantitative genetic model that integrates competition and variability, and have shown through simulation that our model mimics the observation in real populations, where competition for resources increases phenotypic variability among individuals. In our model an improvement of the social environment through an increase in b, which was modeled as a heritable trait in itself, resulted in reduced variability.

Estimating b

The key parameter in our model is the regression coefficient b, which comprises both direct and indirect genetic effects. In other words, b is heritable and can respond to selection. Application of our model requires methods to estimate b and its genetic components. In the following, we discuss the data requirements and propose models that could be used as a first step to estimate the average b and its direct and indirect genetic variance.

Our b connects the difference in trait values between the group mate and the focal individual at the previous time point to the target phenotype of the focal individual at the current time point. Estimating b, therefore, requires data on group-structured populations, where competition occurs within groups, and repeated observations on the phenotypes of the group members (i.e., time-series data).

First, to estimate the overall average level of competition, one could fit single fixed b for all groups, using the model

$$y_{t,i} = P_{t + 1,i} - P_{t,i} = \mu + b\left( {P_{t,j} - P_{t,i}} \right) + e_{t,i}.$$

In genetic analysis of outbred populations, interest is in the genetic (co)variances of growth and the direct and indirect effects on b (AGR, AD, and AI in Eqs. 2 & 3). In animal and plant breeding, for example, knowledge of those parameters would indicate prospects for genetic selection against competition and variability. In outbred populations, the following mixed model may serve as starting point to estimate genetic variance components (ignoring non-genetic terms for simplicity),

$$\begin{array}{l}{\mathbf{y}}_{t,i} = \\ \mu _t + \bar b\Delta {\mathbf{y}}_{t - 1,ij} + {\mathbf{Za}}_{{{GR}}} + {\mathbf{Z}}_{{{D}},\Delta y_{t - 1,ij}}{\mathbf{a}}_{{D}} + {\mathbf{Z}}_{{{I}},\Delta y_{t - 1,ij}}{\mathbf{a}}_{{I}} + {\mathbf{e}}\end{array}$$

where matrices and vectors are in bold and scalars are in italic. y is a vector of phenotypic observations, with elements yt,i = Pt,i − Pt−1,i, μt is an overall mean that may be specific to each time point. The term \(\bar b\Delta {\mathbf{y}}_{t - 1,ij}\) accounts for the average competition in the population, and Δyt−1,ij is a vector of phenotypic differences between the group-mate and the focal individual at the previous time point, with elements Δyt−1,ij = Pt−1,jPt−1,i. The ZaGR are the ordinary (random) additive genetic effects on growth rate. The \({\mathbf{Z}}_{{{D}},\Delta y_{t - 1,ij}}{\mathbf{a}}_{{D}}\) accounts for the direct genetic effects in b, where aD is a vector of random direct genetic effects on b, and \({\mathbf{Z}}_{{{D}},\Delta y_{t - 1,ij}}\) an incidence matrix for direct effects, with elements Pt−1,jPt−1,i in the row and column for focal individual i. The \({\mathbf{Z}}_{I,\Delta y_{t - 1,ij}}{\mathbf{a}}_I\) accounts for the indirect genetic effects in b, where aI is a vector of random indirect genetic effects on b, and \({\mathbf{Z}}_{I,\Delta y_{t - 1,ij}}\) is an incidence matrix for indirect effects, with elements Pt−1,jPt−1,i in the row for the focal individual i and column for its group mate j. Hence, direct and indirect effects on b are so-called random regressions. Note that the above expression merely serves as starting point, and will have to be extended with non-genetic random effects, such random group effects and permanent individual effects (Ep,GR,i in Eq. 2). Moreover, there may be issues with the identifiability of the genetic variance components, which will depend on the family relationships within and between groups (e.g., Appendix of Bijma et al. 2007b).

When time series data are not available, which may often be the case, another approach could offer a solution. Quantitative genetic models for inherited variability can be used to estimate genetic variance in variability from records on within-family variance. Figure 4a shows that variability of sire offspring is correlated with the direct breeding value for b of the sire. Figure 4b shows that variability of the group mates of the offspring is correlated with the indirect breeding value for b of the sire. Therefore, it may be possible to capture direct and indirect effects on b by fitting linear mixed models to the within-family variance, and to the variance of the group mates of a family, with sire as random effect. This analysis requires an appropriate family and group structure, but not time series data. More research is needed to see how breeding values for inherited variability correlate with direct and indirect effects on b, and how those effects can be fully captured.

Evidence for genetic variation in b

To the best of our knowledge, there are no estimates of b available in the literature. However, some indications for variation in b may come from estimates of ψ (psi) in so-called trait-based models of IGE (Moore et al. 1997). When data are available on multiple discrete genotypes, such as inbred lines, fixed b values could be estimated for each genotype, similar to the approach of Bleakley and Brodie (2009), who estimated ψ in guppies (Eq. 1).

This empirical study involved five inbred strains of guppies that differed genetically in their antipredator behavior. One individual from each (focal) strain was paired with three individuals from a different, unrelated strain i.e. social strain. In that way, each focal genotype was tested against different social environments. The results of the study show that the level of ψ differed between the focal strains and in some cases also depended on the social strain, suggesting genetic variation in ψ. In a similar experimental design, where the focal genotype was held constant while social groups varied, the social group effects were estimated for chemical signaling in D. melanogaster (Kent et al. 2008) and sexual display traits in D. serrata (Chenoweth et al. 2010), and both studies estimated and found variation in ψ.

Implications for animal and plant breeding

Phenotypic uniformity is an important trait in animal breeding. In the pig industry, for example, it is desirable to deliver animals within a preferred range to the slaughter house, while deliveries outside that range result in penalties for the farmer (Hennessy 2005; Mulder et al. 2008). In aquaculture, fish that deviate too much from the average size are usually not sold, which reduces revenues. In addition, large size differences in fish populations stimulate competition, which reduces welfare and health of the animals. Better understanding of inherited variability, therefore, is interesting from an economic and animal welfare point of view. In plants, variability may also emerge as a commercially important trait, as some studies suggest that higher uniformity is related to higher productivity (Zhang et al. 1999; Denison et al. 2003).

There is substantial evidence of a genetic basis of variability, which has been obtained through selection experiments and by quantifying genetic variation in variability (reviewed by Hill and Mulder 2010). Recently, methods have been developed to detect QTLs that control variability, so-called vQTLs (Rönnegård and Valdar 2011; Rönnegård and Valdar 2012), and these have been found in studies of litter size in pigs (Sell-Kubiak et al. 2015), several morphological traits and days to flowering in maize (Ordas et al. 2008), and locomotor behavior in fruit flies (Ayroles et al. 2015). Furthermore, several mechanisms, resulting in vQTL effects have been proposed (Rönnegård and Valdar 2011; Rönnegård and Valdar 2012), including: epistatic gene interaction, gene-by-environmental interaction, multi-allelic additive effects underlying a QTL and scale of measurement for the observed phenotype. However, until now, variability has been studied only in relation to direct genetic effects of the focal individual. Here we considered an alternative mechanism that gives rise to genetic variation in variability, which does not only involve the genotype of the focal individual, but also a genetic effect of the social partner, and hence adds another layer to the complexity of inherited variability. The genetic contribution of the social partner is ignored in current QG models for inherited variability, which may reduce accuracy of estimated breeding values and response to selection. When traits are affected by social interactions, selection strategies that accounts for both direct and indirect genetic effects can result in higher response (for example, Griffing 1976; Muir 1996; Bijma et al. 2007b). Our findings suggest that future breeding programs aiming to reduce variability may also need to consider increasing b.

Implications for evolutionary biology

In evolutionary biology, the study of canalization focuses on the absence or suppression of phenotypic variation. Hence, breeding for uniformity can be seen as an analog of the evolution of canalization. Results of our model suggest that canalization may have a social genetic component. Evolution of canalization, therefore, could also be studied in the light of the regression coefficient b. A better understanding of the genetic mechanisms affecting variation may also increase our understanding of the potential for evolutionary change (Flatt 2005). For example, traits may show less variability in some populations than in others, which is often attributed to low genetic variation. With canalization, however, phenotypic variation may be low while the underlying genetic variation is high, which can hinder phenotypic evolution (Flatt 2005).

Mulder et al. (2016) showed that within-nest variability of fledging weight in a natural population of Great Tit (Parus major) has a genetic component and is under stabilizing selection. In that study, phenotypic variability was considered either a trait of the individual, or a trait of its parents, and it was discussed how this view would change the interpretation of the genetic parameters. Here we focused at connecting differences in phenotypic variability between individuals to the level of competition, which may be useful for future studies on variability in natural populations.

Kin-selection theory predicts that individuals should interact differentially with kin vs. non-kin, because this increases their inclusive fitness (Hamilton 1964b). Together with results of our model, this prediction suggests that related individuals should show less variability. In other words, groups consisting of relatives should have higher \(\bar b\) than groups of unrelated individuals. Hence, our findings suggest that canalization may partly evolve by kin-selection.

Conclusion

We presented a quantitative genetic model in which direct and indirect genetic effects lead to inherited variability of trait values on the phenotypic level. The b from our model can respond to selection, and changes in b resulted in changes in variability, indicating the co-evolution of social interactions and inherited variability. Selection results showed that the effect of IGEs on b is ignored in classical mass selection, but can be partly captured by group selection on the mean or the variance. The latter also resulted in a decrease of variability. These findings suggest that we may have been overlooking an entire level of genetic variation in variability, the one due to IGEs. Genetic improvement of social effects, therefore, may be a promising route to reduce variability.

Data archiving

The data are available from the Dryad Digital Repository: https://doi.org/10.5061/dryad.c3k48j0