Introduction

An indirect genetic effect (IGE) is a genetic effect of an individual on trait values of other individuals (Griffing, 1967; Moore et al., 1997; Wolf et al., 1998). The term IGE is commonly used to refer to such effects on individuals belonging to the same species. As conspecifics are often an important component of the (social) environment that individuals experience, IGEs are expected to be a common phenomenon in nature, both in animals and plants, and in domestic, wild and laboratory populations (Sakai, 1955; Frank, 2007). The classical example of an IGE is the maternal genetic effect of a mother on the trait values of her offspring in a mammal (Dickerson, 1947; Willham, 1963; Falconer, 1965; Kirkpatrick and Lande, 1989). Other examples of IGEs are mortality due to cannibalistic interactions in domestic chicken (Muir, 1996, 2005), the effect of competition among trees on growth rate in bark diameter (Brotherstone et al., 2011), the outcome of dyadic interactions in deer (Wilson et al., 2011), social behaviours in microorganisms (Crespi, 2001), size, developmental and fitness-related traits in Arabidopsis (Mutic and Wolf, 2007; Wolf et al., 2011), and growth rate in Medaka (Ruzzante and Doyle, 1991).

The IGEs that an individual experiences can be interpreted as a genetic component present in its social environment. In this perspective, IGEs create an environment that can respond to selection (for example, Hadfield et al., 2011). Thus, in the context of the classical quantitative genetic model where the trait value of an individual is decomposed into the heritable effect of its genotype and a residual labelled as environment, P=G+E, the E-term is partly heritable when IGEs occur. Consequently, compared with predictions based on the classical model, IGEs may substantially alter response to selection, both in magnitude and direction (Griffing, 1967, 1976; Moore et al., 1997; Bijma and Wade, 2008; McGlothlin et al., 2010).

Initially, the pioneering works of Griffing (1967, 1976, , 1977, 1981a, 1981b, 1989) have largely been overlooked. At present, however, IGEs are receiving increasing attention and the number of publications referring to IGEs is increasing rapidly, both in agricultural sciences and in biology. The integration of IGEs in the ordinary linear mixed models used for the estimation of breeding values and variance components has greatly facilitated empirical research on IGEs, both in animal breeding and other disciplines (Muir and Schinkel, 2002; Muir, 2005). In animal breeding, the integration of IGEs within the ordinary quantitative genetic framework has increased their understanding and acceptance, and clarified their potential role in response to selection (Bijma et al., 2007a). In evolutionary biology, the development of trait-based IGE models, often labelled as ‘interacting phenotypes’, has triggered both theoretical work (for example, McGlothlin et al., 2010) and empirical work on the measurement of IGEs (Moore et al., 1997; Bleakly and Brodie, 2009; Chenoweth et al., 2010; Bailey and Zuk, 2012).

This manuscript is a selective review of issues relating to IGEs, with a focus on the modelling IGEs and the interpretation of results. No attempt is made to fully review the existing literature on IGEs. Readers entirely unfamiliar with IGEs are referred to the introduction of Moore et al. (1997) and to Wolf et al. (1998) for a broader introduction. The following contains (i) a basic summary on quantitative genetic IGE models, (ii) a discussion of trait-based vs variance-component models of IGEs, (iii) a discussion on statistical estimation issues, (iv) a section on the interpretation and estimation of Ψ, (v) a section on the interpretation of the magnitude of IGEs in relation to response to selection and (vi) a discussion of the genetic architecture of IGEs.

Basic quantitative genetic results for IGEs

This section briefly summarizes the basic quantitative genetic trait model for IGEs, using the so-called variance-component approach for a group-structured population (Griffing, 1967). Readers familiar with IGE models may skip this section. Consider the simple case of a population structured into groups of n members each, in which interactions among group mates affect the phenotypes of individuals. Then, using the principle of least squares, the trait value of each individual may be decomposed into two additive genetic components. First, a component originating from the genes of the focal individual itself, known as the direct genetic effect (DGE). Second, a component originating from the genes of each of its group mates, known as the IGE. Then the phenotypic trait value of focal individual i is given by

where A denotes additive genetic effects, subscript D direct effects, subscript S indirect effects (S suggesting ‘social’), and j one of the n–1 group mates of focal individual i (Griffing, 1967). When group members are unrelated, phenotypic variance is given by (Griffing, 1967)

From taking the average of equation 1 at two consecutive generations, it follows that response to selection (considering additive effects only) equals . Hence, the quantity relevant for response to selection is given by

which is known as the ‘total breeding value’ of individual i (Moore et al., 1997; Bijma et al., 2007a; Bijma, 2011). Note that, in contrast to the trait value, the total breeding value of an individual is entirely a heritable property of that individual; it does not contain components due to genes of others. Thus, the DGEs and IGEs in equation 3 are effects of the genes of individual i, as indicated by their subscript. This is because an individual transmits its own genes to the next generation, which may differ from those affecting its phenotype (the latter include the ΣAS,j in equation 1).

Response to selection, therefore, equals the per-generation change in mean total breeding value, . Using Price’s theorem, common expressions for response to selection may be generalized to incorporate IGEs (Price, 1970; Bijma, 2011). For animal breeders, response may be expressed as

where ι is the ordinary intensity of selection (Falconer and Mackay, 1996), ρ is the accuracy of selection, being the correlation between the total breeding value and the selection criterion in the candidates for selection and is the s.d. in total breeding value among the candidates for selection. For natural selection, response may be expressed as

where is the genetic selection gradient for the total breeding value, being the regression coefficient of individual relative fitness on individual total breeding value in the population (Lande, 1979; Bijma, 2011). Hence, the expressions for response to selection, equations (4) and (5), have the usual shape, but their components have been redefined to obtain a generally valid expression.

Equations (4) and (5) show that the potential of a population to respond to selection is measured by the variance in total breeding value,

In the absence of IGEs, the ordinary additive genetic variance reflects both the potential of a population to respond to selection, for example, , and the heritable component of phenotypic variance, . A comparison of equations (2) and (6), however, shows that with IGEs the genetic component of phenotypic variance differs from the heritable variance available for response to selection (Bijma, 2011):

As a consequence, traits affected by IGEs may show hidden heritable variation, and heritable variance may in theory exceed phenotypic variance (Bijma, 2011). On the other hand, IGEs may fully remove heritable variation, despite a significant ordinary heritability (Bijma, 2011; Wilson et al., 2011; Costa e Silva et al., 2013). Thus, as interactions among individuals are abundant in nature (for example, Frank, 2007), the heritable variance that determines the potential of a population to respond to selection will usually differ from the ordinary additive genetic variance, but the magnitude of this difference is largely unknown at present.

Taking a simple case as example, the above has briefly illustrated that heritable variance for response to selection is given by the variance in the individual breeding values for the mean trait value of a population (equations (3), (4), (5), (6)). This approach can be extended to more complex cases, including so-called emergent traits that cannot be observed on a single individual, as illustrated by examples in Wilson et al. (2011) and Bijma (2011); see also Linksvayer et al. (2012) and Bailey (2012).

Modelling IGEs

Quantitative genetic models of IGEs have been constructed within two theoretical frameworks, following a historical dichotomy in maternal effects theory (McGlothlin and Brodie, 2009). Following the approach of Dickerson (1947) and Willham (1963), Griffing (1967) developed the so-called variance-component models, where individual trait values are decomposed into a direct genetic component attributed to the focal individual, the indirect genetic components attributed to its social partners, and a residual, as in equation 1 above. Griffing (1967) referred to the IGE as an associative effect, indicating that it originates from individuals associated with the focal individual. This terminology has later been used also by Muir et al. (1996, 2005). The variance-component model does not specify the phenotypic trait values that cause the indirect effects. Instead, using the principle of least squares, it partitions the total phenotypic variance in the trait of interest into a direct and an indirect genetic component, after which DGEs and IGEs can be estimated as random effects, similar to the estimation of ordinary breeding values (Henderson, 1975).

The second class of models follows from an extension of the maternal-effects models of Falconer (1965) and Kirkpatrick and Lande (1989). These models are trait-based, in the sense that they specify the indirect effect on the phenotype of the focal individual as a direct function of the phenotypic trait values of its social partners, and have been referred to as ‘interacting phenotypes’ (Moore et al., 1997; Wolf et al., 1998; McGlothlin et al., 2010). In the simple case where the trait of interest is also the single trait causing the IGE and the interaction is between two individuals, the trait-based model equals (Moore et al., 1997)

where i denotes the focal individual, j its social partner, Ai the additive genetic component of Pi originating directly from the focal individual, ψ (psi) the ‘regression coefficient’ of the focal individual’s trait value on the trait value of its social partner and e a residual. The term regression coefficient is written in quotation marks here because ψ differs from an ordinary least-squares regression coefficient; see section entitled ‘Estimating the interaction coefficient psi’. (For reasons of consistency within this manuscript, the notation used here differs from that in Moore et al. (1997) and related papers.) The ψ is known as the interaction coefficient, and is a generalisation of the regression coefficient of offspring phenotype on mother phenotype in the model of Falconer (1965). The ψ together with Var(A) defines the magnitude of the IGEs, and ψ determines the correlation between DGEs IGEs. For a single-trait model, realistic values of ψ satisfy −1<ψ<1 (see below).

With symmetric interaction, the same model applies to the social partner, Pj=Aj+ψPi+ej, which creates a feedback loop; Pi=…+ψPj+…=…+ψ2Pi+…=…+ψ3Pj+….. Solving the model yields (Moore et al., 1997)

Hence, the trait value is undefined for | ψ |=1. The correspondence to the variance component model is given by (McGlothlin and Brodie, 2009)

Thus, in the trait-based model, the variance of A together with the value of ψ determine the (co)variances of DGEs and IGEs.

Note that, with feedback, the Ai in the trait-based model differs from the classical breeding value as defined by Fisher (1918), which includes the full additive effect that can be explained statistically by the genes of the focal individual, irrespective of whether those effects arise directly or via feedback. The Ai in equation 8, in contrast, contains only the genetic effects of the focal individual that directly affect its own trait value, excluding effects occurring indirectly due to feedback via the social partner. Hence, the Ai in the trait-based model is defined functionally, whereas the ordinary breeding value and additive genetic variance are statistical concepts.

In the general case, the traits causing the indirect effect may differ from the focal trait, and the trait-based model must be specified including all correlated interacting traits simultaneously (Moore et al., 1997; McGlothlin et al., 2010)

where pi is a vector of trait values of focal individual i, ai a vector of corresponding additive genetic effects of the focal individual (defined functionally, not Fisherian), ei a vector of corresponding non-heritable effects, pj a vector of trait values for the jth partner of i, Ψ a square matrix of ‘partial regression coefficients’ and the summation is over all social partners j of i. Hence, Ψ is of dimension ntraits × ntraits, with elements ψkl denoting the ‘partial regression’ of Pk,i on Pl,j, k and l denoting traits (Following a common convention, vectors are in lower-case bold and matrices in upper-case bold).

Both modelling frameworks have their advantages and disadvantages.

  • The variance component model can be specified validly for a single trait, irrespective of whether the IGE originates from that trait or from other traits. In cannibalistic laying hens, for example, the IGE of an individual on the survival probability of its group mates relates to its pecking behaviour. Nevertheless, the variance-component model allows quantifying IGEs solely from survival data and knowledge of which individuals interact with each other (that is., cage composition), without any information on pecking behaviour of individuals (Peeters et al., 2012). The trait-based model, by contrast, requires the inclusion of all traits that functionally underlie the IGE on the focal trait. Some of those traits may be difficult to record quantitatively and in sufficiently large numbers without compromising the organism, such as chemical signals in insects or plants (Bais et al., 2006; Kent et al., 2008).

  • When the traits causing the IGEs are known and recorded, the trait-based model quantifies the strength of the interaction for each trait that causally underlies the IGEs, in terms of the interaction coefficient ψkl. The variance-component model, in contrast, leaves the mechanisms underlying the IGEs entirely unclear; it merely quantifies their total magnitude in terms of a variance component. From this perspective, the variance-component model is a typical breeder’s tool, as it allows estimating the effects that determine response to selection without knowledge of the underlying biology. In evolutionary biology, however, understanding the mechanisms causing the interactions is a research goal in its own right, which is facilitated by the trait-based approach.

  • The variance-component model specifies the strength of the interactions as a genetic parameter, the indirect genetic variance Var(AS), whereas the trait-based model specifies the strength of the interactions as a phenotypic parameter, the interaction coefficient ψ. In principle, estimation of phenotypic parameters requires smaller sample size than estimation of genetic parameters. Hence, when researchers have reliable clues on which traits cause the IGEs, the trait-based model should yield more precise estimates of the strength of the interactions or require smaller sample size. Note that this greater precision may not extend to response to selection, because response involves the product of ψ and genetic parameters (Morrissey et al., 2012).

The distinction between the variance-component and trait-based approach is an analogy of that between two expressions for response to selection, being Price’s theorem and the Lande—Arnold expression for response in multivariate phenotypic selection analysis (Price, 1970; Lande and Arnold, 1983). Price’s theorem specifies response to selection in a trait, say k, as the covariance between an individual’s breeding value for the trait and its relative fitness,

This expression is valid univariately, irrespective of whether selection is based on a single or on multiple traits. In multivariate phenotypic selection analysis, response is given by (Lande and Arnold, 1983)

where r is a vector of responses in each trait, G the matrix of additive genetic (co)variances between all traits and β the vector of partial phenotypic regression coefficients of individual fitness on individual trait values (the so-called selection gradients, not to be confused with in equation 5 above). This system is valid only when all correlated traits affecting fitness are included. The variance-component IGE model is an analogy of Price’s theorem, whereas the trait-based model is an analogy of the Lande—Arnold model.

Estimating IGEs

In the variance-component framework, it is in principle relatively straightforward to estimate genetic parameters for IGEs by extending the so-called animal model with IGEs (Muir and Schinkel, 2002; Arango et al., 2005; Muir, 2005; Van Vleck and Cassady, 2005; Bijma et al., 2007b). Such models are flexible, as they allow for the inclusion of many fixed and random effects, for variation in the number of individuals that interact (‘group size’), and for scaling of IGEs due to variation in the intensity of interaction (Muir, 2005; Gilmour et al., 2006; Cantet and Cappa, 2008; Bijma, 2010a). Nevertheless, quantifying the magnitude of IGEs from field data is often non-trivial, as illustrated in the following.

In group-structured populations, the statistical power for identifying IGEs is determined mainly by the number of groups rather than the number of individuals (Bijma, 2010b). The ideal data, therefore, consists of many small groups of equal size, all kept in the same macro environment, such as, for example, a large laying house with hens kept in cages of a few individuals (for example, Peeters et al., 2012). Estimating IGEs becomes difficult when groups are large and few, or present in different environments. In dairy cattle, for example, estimation of IGEs is challenging despite the availability of records on millions of cows, because of the large group sizes, the variation in environment between herds and the continuous changes in group composition due to replacement of individuals. Even in populations of domestic pigs accurate estimation of genetic parameters may be difficult, because group sizes are increasing and groups are often kept in a limited number of larger units. A farm may, for example, consist of units of 8–12 groups each, which introduces the risk of confounding environmental unit or unit × batch effects with IGEs, even when random group effects are included in the statistical model. Compared with a previous study (Bergsma et al., 2008, Bergsma et al. (2013), for example, found substantially smaller IGEs in growth rate in domestic pigs due to an increase in the data set and a change in the statistical model.

As interacting individuals by definition share the same environment, there is an obvious risk of confounding IGEs with environmental effects. To account for shared environment and for a non-heritable component of the indirect effect, the statistical model should at least allow for a non-genetic covariance among interacting individuals. In populations consisting of small groups this covariance may be negative, which requires fitting a correlation between the residual terms of group mates (Bijma et al., 2007b). With large groups, the non-genetic covariance between group mates is likely positive, and can be accounted for by including a random group-effect in the model (Arango et al., 2005; Bergsma et al., 2008). In populations of plants, including trees, spatial residual variance structures may be fitted to account for local environmental trend and for the non-genetic component of the indirect effect. In tree breeding, advanced modelling strategies have been developed to account for such effects (Gilmour et al., 1997; Stringer et al., 2011; Costa e Silva et al., 2013).

Besides environmental effects, there may be other indirect effects due to differences in sex, age or maternal environment experienced early in life, for example, that may bias the genetic estimates of interest (for example, Bouwman et al., 2010). Thus, apart from the IGE of interest, the full indirect effect due to a social partner may consist of a number of fixed and random components. Hence, a mixed model for the estimation of IGEs may be interpreted as a combination of a direct and indirect sub model,

Some of the fixed or random direct and indirect effects may be fully confounded and can be grouped. For example, because interacting individuals are generally present in the same herd, fitting a single-herd effect will account for differences between herds because of both direct and indirect effects.

Another critical factor in the estimation of IGEs is the genetic relatedness among interacting individuals. When interacting individuals are equally related, for example, when interactions occur within sib-families, the direct and indirect genetic variance are not statistically identifiable (Appendix in Bijma et al. (2007b); Cheng et al. (2009)). This situation may be common in both natural and domestic populations. In mammals, for example, litter mates may affect each other’s growth rate due to competition for maternal care. However, without cross-fostering, such effects cannot be estimated from phenotypic records and pedigree information. In principle, genomic information offers a solution, as it allows to estimate the variation in relatedness between pairs of full siblings, around its expected value of 0.5. For species with reasonably large genomes, however, the variation in relatedness among full siblings will be small, so that large data sets will be required to accurately estimate the variance due to IGEs (Visscher et al., 2006).

Another complication is a potential dependency of IGEs on relatedness. Kin selection theory predicts that IGEs may differ systematically between kin vs non-kin, so that IGEs with positive effects on fitness are primarily expressed towards kin (Hamilton, 1964). Related individuals may recognize each other because they grew up in the same litter or nest (Komdeur and Hatchwell, 1999), and this may affect their mutual behaviour later in life. In domestic pigs, for example, litters from different mothers are often mixed at weaning, so as to standardize the size of fattening groups. At mixing, pigs fight with individuals originating from other litters, but not with their familiar litter mates (Meese and Ewbank, 1973). Thus, the IGE an individual pigs expresses goes together with its relatedness to the recipient of the effect. This issue can be partly solved by distinguishing IGEs on kin from IGEs on unrelated individuals, in which case the statistical model has three random genetic effects; DGE, IGE on kin and IGE on non-kin. Our unpublished results, however, show that the DGE and the IGE on family members are fully confounded in this situation. A potential solution may come from cross-fostering strategies, where genetically unrelated individuals are combined into the litter nursed by a (foster) mother (for example, Bouwman et al., 2010; Wolf and Cheverud, 2012). In such data, a set of familiar individuals contains variation in pair-wise relatedness, so that direct effects and indirect effects on relatives can probably be distinguished.

Further complications arise when the set of individuals that interact changes over time, particularly when this change is related to the trait of interest. Examples are: survival time in laying hens, where individuals that die do no longer express their IGE; longevity in dairy cattle, where only cows that are present in the herd at the same time compete for avoiding culling by the farmer (the so-called voluntary culling); and infectious diseases, where only infected individuals can affect the disease state of others. The results of simulated data on disease epidemics (Lipschutz-Powell et al., 2012) and personal experiences with data from laying hens and dairy cattle (unpublished results) suggest that a simple modification of the incidence matrix of indirect effects, so as to account for the duration of the interaction between a pair of individuals, may produce extremely erroneous results. Solving these issues probably requires more advanced statistical approaches that account for the dynamic nature of the interactions among individuals over time.

Estimating the interaction coefficient Ψ

Although there has been considerable theoretical research on the trait-based IGE-model, relatively little attention has been paid to the estimation of the interaction coefficient Ψ (psi). In the simple case where one can distinguish between the donor of the indirect effect and the recipient of the effect, which is possible only in the absence of feedback, one can estimate Ψ as the multiple regression coefficients of recipient trait values on donor trait values (Bleakly and Brodie, 2009). For the more general case, McGlothlin and Brodie (2009) show that Ψ can be estimated by first fitting a variance-component model to the data, and subsequently calculating Ψ from the relationship between the variance-component and the trait-based model. This approach, however, means that a multivariate genetic analysis is required to estimate a set of phenotypic regression coefficients. As estimation of genetic parameters is challenging, particularly when it comes to genetic correlations, one expects that the estimation of phenotypic regression coefficients from estimated genetic parameters causes a loss in precision. (Note that this two-step approach does not solve the issue of whether the set of phenotypic traits is complete in the trait-based model). Hence, the direct estimation of Ψ as a set of phenotypic regression coefficients would seem more attractive.

At first glance, based on equations (7) and (10), one expects elements of Ψ to be partial regression coefficients of focal individual trait values on social partner trait values. However, when feedback occurs, the non-genetic term ei in equation 10, which would seem to be the residual of a regression model, is correlated to the vector of phenotypes of the social partner, Cov(ei,Pj)≠0. This occurs because the expression for pj contains a component Ψei when there is a feedback. This non-zero covariance between the residual and the explanatory variable violates the principle of least-square estimation of a regression coefficient. Consequently, contrary to apparent belief (Moore et al., 1997; McGlothlin and Brodie, 2009), Ψ differs from the matrix of partial regression coefficients of focal trait values on partner trait values when feedback occurs, .

The regression coefficient can be estimated from a linear model, but interest is usually in Ψij. Hence, the challenge is to translate the estimated into an estimate of Ψij. For the univariate case, where the focal trait and the trait causing the IGE are one and the same trait, the relationship between ψ and can be derived easily from substituting phenotypic values in by equation 8, giving

Hence, for small values of ψ, so that ψ2≈0, the regression coefficient of focal trait value on partner trait value overestimates ψ by factor of almost two. For large values of ψ, in contrast, so that ψ2≈1, the regression coefficient is an unbiased measure of ψ; .

For the single-trait case, an estimator of ψ can be obtained by solving equation 11 for ψ, which yields a quadratic equation having two solutions. One solution yields a |ψ|>1 for , and is biologically irrelevant. The relevant solution equals

for ≠0, and ψ=0 for =0. A b=0.8, for example, corresponds to ψ=0.5. Figure 1 shows the relationship between and ψ for the univariate case.

Figure 1
figure 1

The interaction coefficient ψ as a function of the regression coefficient of focal individual trait value on partner trait value, (solid line). For a univariate model with symmetric interaction (equation 12). For comparison, the dotted diagonal line represents the relationship y=x.

Full size image

As the relationship between ψ and response to selection is strongly non-linear (Moore et al., 1997), interpreting an estimate of as an estimate of ψ may cause dramatic errors in the predicted response to selection. For example, for a true value of ψ=0.5, the corresponding regression coefficient equals =0.8. When interpreting this value as an estimate of ψ, the response to selection is overpredicted by a factor of 5.2 (using equation 16b or Figure 3b in Moore et al., 1997). Hence, the error in predicted response to selection is substantially greater than the error in ψ.

Unfortunately, for the multivariate case, obtaining Ψ from a matrix of estimated regression coefficients requires solving a multivariate quadratic equation, which is non-trivial. Moreover, trait values of individuals may be correlated for other reasons, such as environmental covariances among interacting the individuals or among full sibling, which should be accounted for in the statistical model (see examples above). Hence, further research on the estimation of ψ is required, for example, with the use of stochastic simulations. In the meantime, empirical studies on the estimation of IGEs in cases with feedback could rely on the variance-component approach.

Interpreting the magnitude of IGEs

Estimates of the indirect genetic variance are often small compared with those of the direct genetic variance, particularly when the number of interacting individuals is large. This may erroneously suggest that IGEs are unimportant (Arango et al., 2005; Van Vleck and Cassady, 2005; Van Vleck et al., 2007; Chen et al., 2008). The interpretation of the magnitude of IGEs requires careful consideration, particularly when IGEs are multiplied by certain factors in the statistical model fitted to the data. With interactions among trees, for example, the IGE of a neighbouring tree on the focal tree will depend on the distance between trees, and may be modelled inversely proportional to the inter-tree distance , dij, denoting the distance between trees i and j (Muir, 2005). The estimated indirect genetic variance will depend on the units in which d is expressed, such as feet vs meters. As 1 ft≈0.3 m, the estimated value of when d is expressed in meters will be 0.32 times the estimate obtained when d is expressed in feet. The true relevance of IGEs for response to selection is of course independent of the units in which inter-tree distance is expressed. Hence, estimated values of will differ considerably, while the importance of IGEs remains the same, illustrating the need for careful consideration of multiplication factors in the model when interpreting .

In general, the strength of the interactions may vary among pairs of individuals,for example, because of variation in distance among plants or variation in group size in animals. To account for such effects in the statistical model, IGEs may be multiplied by factors (Muir, 2005; Cantet and Cappa, 2008; Cappa and Cantet, 2008; Bijma, 2010a; Costa e Silva et al., 2013),

The f has been referred to as the intensity of competition factor or the intensity of interaction factor in the context of plant breeding (Cappa and Cantet, 2008), and as a dilution factor in models with variation in group size in animals (Bijma, 2010a, see also Hadfield and Wilson (2007)). Statistical identifiability of requires that the scale of f be fixed, to avoid that the model term ΣfijAS,j has two scaling factors; and (Cantet and Cappa, 2008). The scale of f is fixed automatically when the absolute values of f are known a priori, for example when f is taken to be the reciprocal of inter-tree distance expressed in meters. However, when relative f-values are used or when relative f-values are estimated from the data, they may be expressed on any scale. (Absolute values of f cannot be estimated from the data because their scale is confounded with the magnitude of ). Scaling f-values results in statistically equivalent models, but yields different estimates of . Thus the scale at which f is defined needs to be considered when interpreting the biological relevance of the estimated Var(AS).

Irrespective of the scaling of IGEs, the correct interpretation of the relevance of IGEs for response to selection always follows from considering the genetic mean of the population, as explained in Bijma (2011). From equation 13, assuming that fij and AS,j are independent, it follows that the genetic mean trait value equals

Thus, following Bijma (2011) for any scaling, the total breeding value is defined by (Costa e Silva et al., 2013)

This quantity represents the average effect of an individual’s alleles on the mean trait value in the population. Then, total heritable variance is given by

Hence, the contribution of the variance in IGEs to total heritable variance equals

In contrast to , this quantity is independent of the scaling of IGEs. For example, when multiplying each fij by a factor of 10, the increases by a factor of 10, the by a factor of 100, the decreases by a factor 10 and the decreases by a factor 100, so that in equation 15 remains the same. Thus, has the same value irrespective of the scale of f; it is the that differs with difference in scaling. Thus, irrespective of the scaling of IGEs, the contribution of IGEs to heritable variance can always be derived by considering the population-mean trait value, and is given by equation 15.

Although different scalings of IGEs are statistically equivalent, some scalings facilitate interpretation more than others. For example, one may scale IGEs such that they reflect the effect on a single social partner (to be precise, on the average social partner). This requires the condition , n–1 representing the number of social partners of an individual, and yields an expression for total heritable variance that corresponds to the original definition presented in (Bijma et al., 2007a),

Alternatively, one may scale IGEs such that the IGE of an individual expresses the full indirect effect of that individual summed over all its social partners. This requires the condition , and yields an estimate of that directly reflects the full contribution of IGEs to total heritable variation,

Beware that has the same value in equation 16 and 17; it is the that differs because of the difference in scaling.

Cappa and Cantet (2008) suggested to define the scale of f by the condition

where the sum is over all neighbours of a focal individual. This condition scales IGEs such that they contribute a single unit to phenotypic variance (assuming unrelated individuals),

Hence, the scaling Σf2=1 relates to the contribution of IGEs to the observed variance among individuals, rather than to the heritable variance for response to selection. There is nothing wrong with this scaling, as long as the relevance of IGEs for response to selection is judged on the basis of equation 15. (Beware of the difference between the Σf2 in equation 18, which is a sum of squares, and the in equation 15, which is the square of the average of a sum).

Fitness vs trait models of social interactions

The above has focussed primarily on IGEs on trait values, rather than fitness. Surprisingly, the scientific debate on the mechanisms that can explain social evolution is dominated by studies focussing solely on the fitness effects of interactions among individuals (for example, Nowak et al., 2010; Abbot et al., 2011; Gardner et al., 2011, and references therein). In the group selection vs kin selection debate, for example, effects of genes are usually specified entirely in terms of fitness cost and benefit of the interactions, whereas the effects on individual trait values are disregarded (for example, Gardner et al., 2011). A fundamental principle in genetics, however, is that the genotype affects the phenotype and the phenotype subsequently affects fitness (Lande and Arnold, 1983). The ongoing fitness-centred debate hinders scientific progress, both in theoretical and empirical studies (Reeve and Keller, 1999; Okasha, 2010).

Because fitness-based models disregard IGEs on trait values, such models do not predict response in traits subject to IGEs (for example, Gardner et al., 2011). This is a severe limitation, as the effect of competition or cooperation on fitness of individuals will usually work by effects on their trait values, meaning that not only fitness but also traits are subject to IGEs. An example comes from root mass in plants, which is an important factor in competition among plants, and depends not only on the focal individual’s genotype but also on that of its neighbours (Biedrzycki et al., 2010). Another example is social behaviours in animals, which may depend not only on the genotype of the individual expressing the behaviour, but also on its social partners. Reciprocal behaviours are a well-known example (for example, Seyfarth and Cheney, 1984). A further example comes from non-beak-trimmed domestic laying hens, where mortality due to cannibalism has a significant genetic component because of the genotype of the recipient of the behaviour (that is, a direct effect; Ellen et al., 2010). This result suggests that individuals differ genetically in the ability to avoid being pecked to death, that is. suggesting a genetically based ability to mediate the behaviour of their cage mates. Predictions of response to selection ignoring IGEs on trait values can be very misleading, as presence of IGEs can have profound effects on response to selection, such as a reversal of the direction of response (Griffing 1967, 1976; Moore et al., 1997; Bijma and Wade, 2008; McGlothlin et al., 2010).

Ignoring IGEs on trait values may also obscure the mechanisms underlying the fitness effects of competition and cooperation among individuals. An example comes from studies examining the mechanism underlying competitive ability in plants. Most plants are closely surrounded by other plants, and plants interact intensely with each other (Casper and Jackson, 1997). Two hypotheses are frequently proposed to explain the impact of relatedness on competition among plants (File et al., 2012). The kin recognition and selection hypothesis predicts that related individuals cooperate and therefore outperform groups of strangers (Hamilton, 1964). The niche partitioning hypothesis, in contrast, predicts that groups of strangers outperform kin groups because of differential resource use by neighbours (Young, 1981). Studies aiming to distinguish between both mechanisms have often used the so-called group studies, where fitness of individuals is measured when grown with either kin or strangers (see File et al., 2012 and references therein). In this approach, greater fitness of kin groups than of stranger groups is taken as the evidence of kin selection, whereas the reverse is taken as evidence of niche partitioning. File et al. (2012), however, argue convincingly that such studies fail to demonstrate the mechanisms underlying competition, as niche partitioning and differential interactions with kin vs strangers may co-occur. They argue for a trait-based approach, distinguishing between the effects of interactions on trait values (IGE on trait values), and the subsequent effects of the trait values of the individual and its partners on fitness, following the approach of phenotypic selection analysis (Lande and Arnold, 1983) and multilevel selection theory (Bijma and Wade, 2008). A quantitative genetic trait model aiming to investigate both niche partitioning and differential interaction among kin vs strangers may for example look like

where the trait value yi of focal individual i is a function of the proportion of strangers among its neighbours (pS), its own DGE (AD,i) and the IGEs of its neighbours, which are partitioned into IGE originating from kin () and IGE originating from strangers (). In this model, the sign of pS indicates whether trait values of individuals on average benefit or suffer from strangers, whereas a non-unity correlation between and , or a difference in their variance, indicates that individuals show different IGEs on kin vs strangers.

Models of social selection, that is models specifying the fitness effects of interactions among individuals and models considering IGEs on trait values are natural partners in the study of social selection and evolution (Moore et al., 1997; Bijma and Wade, 2008; McGlothlin et al., 2010). A better connection of the main-stream debate on social evolution with the field of IGEs should facilitate scientific progress in this area.

Genetic architecture and sustainability of response to selection

Evidence is increasing that IGEs can make a meaningful contribution to heritable variation in populations (for example, Mutic and Wolf, 2007; Peeters et al., 2012). IGEs can, therefore, contribute to response to selection, at least in the short term. The potential of IGEs to contribute to a long-term response, however, depends primarily on the genetic architecture underlying IGEs, rather than on their contribution to current heritable variation. For DGEs, selection experiments and genetic improvement programmes in agriculture have demonstrated that many quantitative traits can show large responses to selection, often with little decline in the rate of response over generations when populations are kept at large effective size (Hill, 2008). Well-known examples are the Illinois selection lines for oil content in corn (Dudley and Lambert, 2004), the increase in growth rate and breast meat percentage in domestic broiler chickens (Havenstein et al., 2003) and the selection for flying speed in Drosophila melanogaster (Weber, 1996). These results demonstrate that the direct genetic component of complex traits may respond to selection for many generations. Whether similar results can be obtained for the indirect genetic component is largely unclear at present.

The large long-term responses observed for many complex traits are in agreement with the emerging knowledge of the genetic architecture of complex traits. Many complex traits appear to be determined by numerous genes, in the hundreds or even thousands, rather than tens, and a substantial proportion of the genetic variance appears to be additive (Hill et al., 2008). A typical example is genetic variation in human height, where evidence points towards many segregating genes, each of small effect (Visscher et al., 2010).

Do these results extend to the indirect genetic component of trait values? At present, only few studies have addressed this issue, but knowledge is increasing rapidly in this field. Biscarini et al. (2010) investigated the genetic architecture of feather damage in domestic laying hens. Lines of domestic laying hens often show considerable feather pecking and cannibalistic behaviour (Muir, 1996). In total, Biscarini et al. (2010) detected 11 significant direct associations between single nucleotide polymorphisms (SNPs) and feather damage, and 81 significant indirect associations. Mutic and Wolf (2007) investigated the genetic architecture of size-, development- and fitness-related traits in Arabidopsis thaliana, and identified 15 quantitative trait loci (QTL) with direct effects. Of those 15 QTL, 13 also had significant indirect effects on trait expression in neighbouring plants, suggesting wide-spread pleiotropy. Although evidence is very limited at present, these results do not immediately suggest that the number of genes with IGEs is substantially smaller than that of genes with DGEs.

Nevertheless, the nature of traits affected by IGEs may limit response to selection. For mortality due to cannibalistic pecking behaviour in domestic laying hens, for example, one would expect a diminishing return relationship; once the pecking tendency is sufficiently reduced, allelic variants further reducing pecking behaviour may have little effect on mortality. Thus, although many genes may be affecting pecking behaviour, when mortality shows a threshold relationship with pecking behaviour, the long-term response in mortality will be limited. Something similar would be expected for IGEs on the growth rate in domestic pigs. Once detrimental social behaviours have been sufficiently reduced by breeding for IGEs on growth rate, it seems unlikely that growth rate can be further increased by breeding for IGEs, while response from DGEs may continue much longer. In a technical quantitative genetic sense, this would mean that alleles with favourable IGEs should show negative epistatic interactions.

An important factor in response to selection is the correlation between the DGE and IGE. In theory, a negative direct—indirect genetic correlation approaching −1 may fully remove heritable variation in a trait (Bijma, 2011). Such negative genetic correlations may be produced by long-term kin selection in natural populations, which may occur as a result of limited dispersal. In wild populations of Ecalyptus globulus, for example, neighbours are closely related (Hardner et al., 1998), so that local interactions lead to kin selection. This will cause the gradual exhaustion of heritable variation and the build-up of a trade-off between the direct and indirect components of the total breeding value. Results confirming this expectation have been found in the tree species E. globulus and Pinus taeda L., where DGE and IGE for bark diameter show a correlation of around −0.8 (Cappa and Cantet, 2008; Brotherstone et al., 2011; Costa e Silva et al., 2013). Such negative correlations between DGEs and IGEs may severely reduce the potential for genetic improvement of bark diameter in forestry (Costa e Silva et al., 2013). Hence, prospects for genetic improvement of wood production appear to be considerably smaller than judged from ordinary heritabilities of bark diameter. In A. thaliana, in contrast, Mutic and Wolf (2007) found positive pleiotropic direct and indirect effects on size-, development- and fitness-related traits. Results of Peeters et al. (2012) in laying hens suggest that non-additive effects may have substantial impact on the direct—indirect genetic correlation. They found moderately to strongly negative direct—indirect genetic correlations in crossbred individuals, whereas the correlation was near zero in the purebred parental lines. This study also found a large increase in the magnitude of IGEs in crossbred individuals, together with outbreeding depression, suggesting that IGEs are subject to dominance.

In agricultural populations, the environment is largely under human control, which limits the potential contribution of IGEs to long-term response, as the social organisation is largely prescribed by the housing system. In natural populations, in contrast, the potential impact of IGEs for response to selection seems almost unlimited because the social organisation of the population is itself evolving. At present, models including IGEs are finding widespread application in natural populations, which may considerably broaden our quantitative genetic understanding of evolution by natural selection (Bailey, 2012). Examples are the impact of IGEs on the evolution of social interactions due to sexual selection (Chenoweth et al., 2010; Bailey and Moore, 2012) and the extension of gene networks to societies (Linksvayer et al., 2012).

Data archiving

There were no data to deposit.