Introduction

The effective size of a population is a parameter central to understanding evolution in small populations, because the magnitude of this parameter determines the genetic effects of both inbreeding and genetic drift (Falconer, 1989; Caballero, 1994). This parameter is also important for solving some practical issues; it has been used as a key parameter in designing strategies for conservation of endangered animal and plant species (eg, Lande and Barrowclough, 1987; Santiago and Caballero, 2000; Yonezawa et al, 2000) and breeding of domestic animals (eg, Caballero et al, 1996; Bijma et al, 2001).

The effective size of a population depends on various demographic and ecological variables (Nunney, 1991, 1993). Among these, the variation in the number of progeny per parent (family size) is one with the most pronounced effect (Crow and Denniston, 1988; Caballero, 1994). When the variation is due to non-inherited causes, the effective population size is simply a function of the variance of family size, and predictions have been developed for a variety of cases (Caballero, 1994; Wang, 1996; Wang and Caballero, 1999). However, when the differences in family sizes are, at least partially, due to inherited causes, formulation of the effective size is complicated by the fact that the offspring and later descendants of a parent with a large (or small) family size inherit the property.

The problem of heritable variation of family size in the formulation of the effective size was first addressed by Robertson (1961) in the context of artificial selection. He introduced the idea of the accumulation of selective advantages of individuals over generations. Following this approach, formulae applicable to various practical situations have been developed (Santiago and Caballero, 1995; Nomura, 1996, 1997; Wang, 1998; Bijma et al, 2000, 2001). Only one extension to directional selection on fitness was given by Nei and Murata (1966). Based on the approach of Robertson (1961), they worked out an equation for monoecious populations. They also extended their derivation to dioecious populations. The equations have been used to estimate the effective population size of humans (Nei and Murata, 1966) and in wild animals (Ryman et al, 1981; Kelly, 2001).

In this paper, I will first derive an expression of the effective size of monogamous populations with heritable variation in fitness, and will compare it with the equations of Nei and Murata (1966). We will see that their equation for dioecious populations is inappropriate for most animal and human populations. Finally, a more useful equation for practical use is presented. The application will be illustrated with published data on a human community in Japan.

Effective size of monogamous populations

Suppose an unstructured population of monogamous species, consisting of N/2 couples each generation. Random mating and discrete generations are assumed. An autosomal neutral gene unlinked to the genes affecting fitness is considered. Formulation of the effective size is based on the amount of genetic drift in the frequency of the neutral allele. But as emphasized by Hill (1972), the same expression could be derived from the loss of heterozygosity (or the increase in inbreeding) in the neutral locus. For the effective size under selection, the identity has been proved by Nomura (1999) and Bijma et al (2000). Environmental correlation between parental and offspring fitness is assumed to be absent.

Let ki be the number of progeny (which reach reproductive age) of couple i. Following the terminology in the related works (eg, Caballero and Santiago, 1995; Santiago and Caballero, 1995), we will refer to ki as the ‘fitness’ of couple i. Assuming an additive model, the fitness of couple i can be partitioned as:

where gs(s)i is the additive genetic component from parent of sex s(= m or f), and ei is the environmental (including the nonadditive and sampling) component. The subscript s in parentheses in gs(s)i is attached to emphasize the sex of the parent. In general, gs(u)i is the genetic component of fitness of parent of sex u(= m or f), when the genotype is expressed in sex s. Note that for s ≠ u, the genotype is not expressed in the fitness of the couple, but will be expressed when it is transmitted into descendants of sex s. The two genetic components of fitness (gm(m)i and gf(f)i) may be partially controlled by different sets of genes, because some components of fitness, such as prenatal and postnatal maternal abilities, are sex-limited. Thus, we treat the contributions of male and female parents as two different sex-limited traits with genetic correlation rg. The variance of ki is decomposed as

where Vgs and Ve are the variances of gs(s)i and ei, respectively. Putting

the heritability of fitness may be written as

The effective population size (denoted by Ne) after one generation is approximated by

(Crow and Denniston, 1988; Caballero, 1994). The cumulative effect of selection can be incorporated into the above expression by considering the partial persistence of the random association between the relative fitness and the frequency of a neutral allele for which the effective size is defined (Santiago and Caballero, 1995, 1998). Let pi (= 0,1/4, 1/2, 3/4 or 1) be the frequency of the neutral allele in couple i, and fs(u)i(= gs(u)i/2) be the expected contribution of parent of sex u in couple i to the relative fitness of couple when the genotype is expressed in sex s. The frequency of the neutral allele in the population is p = ΣN/2i=1pi/(N/2). Only the two relative fitnesses, fm(m)i and ff(f)i, contribute to the change (Δp) in the frequency of the neutral allele after one generation as

(Santiago and Caballero, 1995, 1998). But in the following generations, all the associations between pi and fs(u)i are relevant to the total cumulative change of the gene frequency. Noting that under weak selection the associations are halved each generation due to segregation and recombination (Santiago and Caballero, 1995, 1998), and half of the persisting association contributes to the change in the gene frequency because of the sex-limited expression of fs(u)i, the total cumulative change (ΣΔp) can be obtained as

where

and

Following the argument of Santiago and Caballero (19951), the variance of Δps(u) can be written as

Analogously, the covariance between Δpm(u) and Δpf(u) is expressed as

With these expressions, the variance of ΣΔp is obtained as

Since the contribution of V(ΣΔp) to 1/Ne is

an expression of Ne is obtained from equation 2 as

If male and female components of fitness are controlled by the same set of genes, ie, h2m = h2f and rg = 1, equation 4 reduces to

which is the same expression as the equation for a monoecious population given by Nei and Murata (1966).

Comparison with published equations

Nei and Murata (1966) extended equation 5 to a dioecious population with equal numbers of male and female parents. Their equation comparable to equation 4 can be written as

which is different from equation 4. In the following, we consider the reason for the difference.

The implicit model assumed in the derivation of Nei and Murata (1966) is a random union of gametes (RUG) model. The RUG model assumed by Nei and Murata (1966) is such that male parent i contributes kmi gametes to a male gametic pool, and female parent j contributes kfj gametes to a female gametic pool. Zygotes are produced by random union of gametes each from the male and female gametic pool. In this model, the expected fitness of a parent is determined only by the genotype of the parent. Although the variation in fitness under this model is assumed to be due to the difference of fertility among parents (fertility selection), the following argument can be applied to the case where the variation in fitness is caused by the difference of viability among progeny (viability selection). The RUG model under viability selection is such that all male and female parents contribute equally their gametes to the respective gametic pool, and after random union of gametes, the survival of zygotes is determined according to their viability. Note that the expected fitness of a parent under this model is not affected by the genotypes of mates, because they are randomly sampled from the population (Caballero and Santiago, 1995). Thus, irrespective of the types of selection, fitness in the RUG model should be defined for individual parents rather than for couples.

This difference of the definition of fitness results in the difference of the accumulation of the change in gene frequency. This can be understood by expressing the frequency of the neutral allele in couple i in our derivation as pi = (pmi + pfi)/2, where psi (= 0, 1/2 or 1) is the gene frequency in parent of sex s in the couple. As seen from equation 3, all possible covariances (eight covariances) among psi and the relative fitness fu(v)i (u, v = m or f) are accounted for in our derivation, while under the RUG model, the relevant covariances are limited to only four covariances [cov(psi, fu(s)i) (s, u = m or f)], because couples are not created. By this omission of the covariances, the amount of the total accumulated effect is halved in the RUG model, compared to monogamous populations. This is reflected in the coefficient 3/4 in the denominator of equation 6, which is half the corresponding coefficient in equation 4. Similar reasoning has been made by Santiago and Caballero (1998) and Nomura (1999) to explain the inconsistency between equations for Ne under selection.

As in monogamous populations, the male and female heritabilities in equation 6 are defined as h2s = Vgs/Vk. But equation 1 does not hold in the RUG model. When male and female components of fitness are controlled by the same set of genes, substituting h2 ≡ h2m = h2f and rg = 1 into equation 6 leads again to equation 5. But we should note that h2 in the equation for monogamous populations is the heritability of fitness of couples, while in the RUG model, h2 is the heritability of fitness of individuals.

Application to estimation of Ne/N in a human population

Before the application, we modify equation 4 to a more convenient form. In quantitative genetic studies of human populations, father-offspring and mother-offspring correlations of sibship size (or equivalently parent-son and parent-daughter correlations of the progeny number) have been estimated (Imaizumi et al, 1970, and references therein). Let COVk,km and COVk,kf be father-offspring and mother-offspring covariances of sibship size, respectively. Assuming monogamous marriages which will be reasonable for most human populations, at least approximately, the two covariances can be written as

where s = m or f. Substituting this expression into equation 4 leads to

Note that by the use of this equation, we can estimate Ne without estimating the heritabilities and genetic correlation.

Imaizumi et al (1970) studied the heritability of human fertility in a rural community in Japan. From their results between 1921 and 1930, I obtained the following values:

Substituting these values into equation 7, we get Ne/N = 0.774. If heritable variation in fitness is neglected (h2M = h2F = 0), the estimate of Ne/N from equation 2 is 0.963. On the assumption of the RUG model, the coefficient of covariance terms in equation 7 is halved, giving Ne/N = 0.858. However, these estimates should be seen cautiously because the covariances of sibship size over generations may be inflated by non-genetic causes, as discussed in the next section.

Discussion

In theoretical population genetics, the RUG model has been frequently assumed. Some organisms do reproduce more or less in this fashion. For example, some aquatic animals release large numbers of gametes into the ocean, and some wind-pollinated plants release a vast number of male gametes (Nunney, 1993). However, in most animal mating systems, individuals (rather than gametes) come together to mate. In the present study, we have limited our concern to the simplest individual-based mating system, but the derived conclusion holds for more complex mating systems. To incorporate the heritable effect of fitness into the formulation of effective size of populations with individual-based mating systems, fitness should be defined for couples (or families in polygamy or polyandry), and all the random associations between the gene frequencies and the parental genetic components of fitness in a couple or family should be accounted for.

In our derivation, we assumed that the random association between the neutral gene and fitness is halved each generation. This assumption will lead to an overestimation of the cumulative change in the gene frequency, since selection reduces the random association through the reduction in the genetic variance of fitness (Santiago and Caballero, 1998). Caballero and Santiago (1995) formulated the effect of selection on the accumulation of change in the gene frequency in monoecious populations.

In human populations, however, traditional continuity or ‘social inheritance’ could have a more important effect on the persistence of the random association between the neutral gene and fitness. As suggested by Cavalli-Sforza and Bodmer (1971), the inheritance of family size might be psychological as well as be physiological. In fact, the family sizes in most human populations are apparently below the potential maximum (Cavalli-Sforza and Bodmer, 1971), indicating that some form of birth control must be in operation. In wild animal populations, environmental correlation is likely an important cause for the resemblance of fitness between parents and their offspring (Crow and Denniston, 1988). The correlation could be positive if parent and progeny both occupy an especially favorable niche, leading to a persisting change in the frequency of the neutral allele in a similar process to inherited causes. For example, from mother-daughter regression, Kelly (2001) estimated the heritability of female fitness of Serengeti cheetahs as 0.89. This high heritability strongly suggests the presence of environmental correlation between fitness of mothers and daughters. The persisting effect of the environmental correlation on the effective population size is kept completely as long as the progeny remain in the same niche and is partially reduced if a part of them migrate to other niches. Statistical analysis of fitness over three generations, such as a comparison of parent-offspring and parent-grandoffspring correlations, will provide an estimate of the persisting proportion over generations. With an analogous argument to the present study, the persisting proportion could be used for expressing the cumulative effect on the effective population size.

Subdivided population structure can be a more important cause for the resemblance of fitness between parents and their progeny, if there are permanent environmental differences among subpopulations and a part of progeny remain in the same subpopulation as their parents (Wang and Caballero, 1999). Since the resemblance is partially reduced by the migration among subpopulations, the persisting effect on the effective size depends on the migration rate. Thus, the effect of population subdivision could be theoretically modeled with an analogy to the model of linkage between the neutral and selected loci, in which the persisting effect on the effective size is formulated with the recombination rate (Santiago and Caballero, 1998). Incorporation of the persisting effects due to such ecological causes could extend our understanding of evolution in small populations.