Impact of migration on the multi-strategy selection in finite group-structured populations

For large quantities of spatial models, the multi-strategy selection under weak selection is the sum of two competition terms: the pairwise competition and the competition of multiple strategies with equal frequency. Two parameters σ1 and σ2 quantify the dependence of the multi-strategy selection on these two terms, respectively. Unlike previous studies, we here do not require large populations for calculating σ1 and σ2, and perform the first quantitative analysis of the effect of migration on them in group-structured populations of any finite sizes. The Moran and the Wright-Fisher process have the following common findings. Compared with well-mixed populations, migration causes σ1 to change with the mutation probability from a decreasing curve to an inverted U-shaped curve and maintains the increase of σ2. Migration (probability and range) leads to a significant change of σ1 but a negligible one of σ2. The way that migration changes σ1 is qualitatively similar to its influence on the single parameter characterizing the two-strategy selection. The Moran process is more effective in increasing σ1 for most migration probabilities and the Wright-Fisher process is always more effective in increasing σ2. Finally, our findings are used to study the evolution of cooperation under direct reciprocity.

For large quantities of spatial models, the multi-strategy selection under weak selection is the sum of two competition terms: the pairwise competition and the competition of multiple strategies with equal frequency. Two parameters σ 1 and σ 2 quantify the dependence of the multi-strategy selection on these two terms, respectively. Unlike previous studies, we here do not require large populations for calculating σ 1 and σ 2 , and perform the first quantitative analysis of the effect of migration on them in group-structured populations of any finite sizes. The Moran and the Wright-Fisher process have the following common findings. Compared with well-mixed populations, migration causes σ 1 to change with the mutation probability from a decreasing curve to an inverted U-shaped curve and maintains the increase of σ 2 . Migration (probability and range) leads to a significant change of σ 1 but a negligible one of σ 2 . The way that migration changes σ 1 is qualitatively similar to its influence on the single parameter characterizing the two-strategy selection. The Moran process is more effective in increasing σ 1 for most migration probabilities and the Wright-Fisher process is always more effective in increasing σ 2 . Finally, our findings are used to study the evolution of cooperation under direct reciprocity.
Populations are usually divided into several subpopulations separated by geographical distance. Migration, linking subpopulations, is one of the oldest adaptation measures in the animal kingdom and human society. The effect of migration on developing such populations has attracted researchers from various fields. Masses of related analytic studies have been performed by geneticists and mathematicians to explain the correlation between genetic distance and geographic distance through the (one-dimensional or two-dimensional) stepping-stone models [1][2][3][4][5] . The update rules most extensively used are the frequency-independent Moran process and the frequency-independent Wright-Fisher process.
The above-mentioned studies assume that all individuals have the same fitness. Nonetheless, most realistic settings do not operate in this way, and individuals' fitness is shaped by the behaviors of themselves together with those who live in the same environment. Evolutionary game theory provides a powerful mathematical framework to deal with such interactions [6][7][8][9][10] . Under the framework, spatial models have attracted more and more attention through games on graphs [11][12][13][14][15][16][17] and populations comprised of subpopulations [18][19][20][21][22][23][24][25][26][27] . For a large class of spatial models, including games in phenotype space 28 , games on sets 29 , games on islands 30 , and games in group-structured populations 31 , the two-strategy selection and the multi-strategy selection under weak selection can be characterized by a single parameter 32 and two parameters 33 , respectively. Weak selection, meaning the fitness varies little among individuals, has been widely used for analytical studies [34][35][36][37][38][39][40][41][42] . Prior research has showed that the results under weak selection, in general, could not be extrapolated to strong selection 43,44 . There are two reasons why weak selection is still so popular is as follows: Weak selection makes it possible to obtain analytical results without additional assumptions; Weak selection is a natural situation in which a particular game makes a small contribution on the overall fitness of individuals, for example, (a) strategies are similar (as in the adaptive dynamics 45,46 ), (b) individuals get confused about payoffs during the strategy update.
When a game between two strategies (say 1, 2) described by the payoff matrix (a ij ) 2 × 2 (a ij is the payoff of an individual using i when interacting with an individual using j) is considered 32 , strategy 1 is more abundant than strategy 2 on average under weak selection if 11 12 21 22 The parameter σ quantifies the influence of the population structure (including the update rule) on the two-strategy selection. When a game among S ≥ 3 strategies (say 1, 2, … , S) described by the payoff matrix (a ij ) S×S (similar to (a ij ) 2×2 ) is considered 33 , the average frequency of strategy k ∈ {1, 2, … , S} over the stationary distribution is greater than 1/S under weak selection if , demonstrates an average over all pairwise comparisons between strategy k and any other strategies, and the sec- , the competition of all S strategies when they have the equal frequency 1/S. Accordingly, the parameters σ 1 and σ 2 quantify the effect of the population structure on the pairwise competition and the competition of all strategies with equal frequency, respectively. Moreover, they quantify the dependence of the multi-strategy selection on these two competition terms, respectively.
It has been showed that the above-mentioned parameters σ, σ 1 , and σ 2 do not depend on the entries of the payoff matrix but rely on the population structure (including the update rule) 32,33 . Two previous studies have investigated the effect of migration on σ in large finite populations 30 and any finite populations 47 , respectively. The values of σ 1 and σ 2 have been obtained for large finite populations 33 , but it is still unknown how varying migration patterns affect them in populations of any finite sizes.
In this paper, we will calculate the concrete values of σ 1 and σ 2 for group-structured populations of any finite sizes. Our study will proceed for the frequency-dependent Moran process (hereafter called the Moran process without ambiguity) and the frequency-dependent Wright-Fisher process (hereafter called the Wright-Fisher process). The Moran process represents an idealized case of overlapping generations and the Wright-Fisher process is a perfect case of non-overlapping generations. The realistic society cannot be fully depicted by either of them, and yet maybe by something in between. Under the assumption of large populations, it is known that the calculation procedures of σ 1 and σ 2 are the same for the Moran and the Wright-Fisher process when the same symbol represents the product of the population size and the mutation probability in the former and twice the product in the latter 28 . However, the two calculation procedures vary significantly in populations of any finite sizes and will be separately given in Supplementary Information. The key point for obtaining σ 1 and σ 2 is to calculate some special probabilities under neutral selection 33 . The corresponding probabilities for the Moran process have been derived in group-structured populations of any finite sizes 31 , and they will be applied to our model to get σ 1 and σ 2 . Meanwhile for the Wright-Fisher process, we will acquire the corresponding probabilities for group-structured populations of any finite sizes, and then use them to calculate σ 1 and σ 2 . For either of the two processes, the expressions of σ 1 and σ 2 given later hold for any 'isotropic' migration patterns, and a particular migration pattern fully captured by the migration range will be employed to clarify how the migration range impacts σ 1 and σ 2 . We will compare the qualitative and the quantitative effect of the two processes on σ 1 and σ 2 . Finally, our findings will be used to study the evolution of cooperation under direct reciprocity by considering the competition of ALLC, ALLD, TFT.

Results
Model description. Consider a group-structured population of size N which is fragmented into M groups (subpopulations). A group can be understood as an island in population genetics, and a particular company or a living community in human society. An individual adopts one of S strategies labelled as 1, 2, … , S and the payoff matrix is given by (a ij ) S×S , where a ij is the payoff of an individual using i against an individual using j. An individual only plays the game with all others of the same group. Interactions produce the payoff of an individual (say k), p k , and further his fitness, f k = 1 + δp k , where δ ≥ 0 is the selection intensity. In this paper, the case of weak selection, δ → 0, is our focus.
The Moran process and the Wright-Fisher process will be analyzed, respectively. In the Moran process, all individuals of the population compete to reproduce one offspring proportional to their fitness, and then one individual is equi-probably chosen from the whole population to die. In the Wright-Fisher process, all individuals compete to reproduce N (population size) offspring proportional to their fitness, and the whole population is replaced by all the newborn offspring. Mutation or migration may happen to the offspring. An offspring mutates with probability u and then he follows the pattern of 'global mutation' to choose one strategy, i.e., one of S strategies is chosen equi-probably; Otherwise (with probability 1 − u), the offspring inherits the strategy of his parent. An offspring migrates with probability v and then he moves to one group according to a pre-defined migration pattern; Otherwise (with probability 1 − v), the offspring remains in his parent's group.
An undirected graph is used to illustrate the pattern of migration among the M groups arranged in a circle. Each node represents a group and an edge is a potential single-step migration path. We focus on the case of vertex-transitive graphs (with or without self-circle), which are homogeneous in the sense that they look the same from every node. Such migration patterns are 'isotropic' and have been widely investigated [1][2][3][4][5]30,31 . Examples include 'local migration' in which a single-step migration occurs equi-probably among all pairs from neighboring nodes or 'global migration' in which a single-step migration is equally likely to take individuals to any other nodes 30 . In this paper, the concrete values of σ 1 and σ 2 will be obtained for any 'isotropic' migration patterns, and then a particular migration pattern fully captured by the migration range r (Fig. 1) will be analyzed. The migration range r, which takes on one of the values  ⌊ ⌋ M 1, 2, , /2 (⌊ ⌋ x is the greatest integer not greater than x), means that all possible displacements generated by a single-step migration form the set Ω(r) = {1, 2, … , r} whose elements are performed equi-probably. The above-mentioned 'local migration' and 'global migration' can be characterized by r = 1 and = ⌊ ⌋ r M/2 , respectively.
In the above model, migration occurs after reproduction. We also consider a second case in which migration occurs before reproduction and the rest follows from the procedure above. Specifically, one individual is chosen equi-probably (from the whole population) to migrate before reproduction in the Moran process, and all individuals migrate before reproduction in the Wright-Fisher process. The corresponding results are qualitatively similar to but quantitatively different from those of the initial case by Monte-Carlo simulations (Supplementary Information), and will not be given in the main text. The quantitative difference emerges because mutation and migration in the second case cannot happen to the same individual during a generation, which happens in the initial case.
The Moran process. The parameters σ 1 and σ 2 can be expressed by the probabilities (under neutral selection) assigned to the event that three randomly chosen (without replacement) individuals use given strategies and locations (see Methods for the detailed calculations). The expression of such probabilities has been derived for any mutation patterns and migration patterns 31 . Applying the expression to 'global mutation' of our model (see Supplementary Information: III and Methods for the detailed calculations), we have σ 1 and σ 2 of the Moran process denoted by σ Mo 1 and σ Mo 2 in the following expressions in which Φ i (f(x)) and Ψ i (f(x)) are abbreviated as Φ i and Ψ i .
. The expressions hold for any 'isotropic' migration patterns described by f(x). To better clarify how the migration range affects σ 1 and σ 2 , we focus on a representative type of migration patterns characterized by the migration range r (Fig. 1) whose corresponding f(x) is Besides the migration pattern, the expressions have no limitations on the non-zero mutation probability, the migration probability, the population size, or the group number. Figure 2 shows that the theoretical values of σ Mo 1 and σ Mo 2 agree well with the simulated values for different mutation probabilities (u), migration probabilities (v), population sizes (N), group numbers (M), and migration ranges (r). It is noteworthy that the number of strategies S does not appear in the expressions of σ Mo 1 and σ Mo 2 , which agrees with the known conclusion that they are independent of S 33 .
After simple calculations, we have σ Mo 1 and σ Mo 2 in the limit u → 0 as if there is a potential single-step migration path between them. In other words, an offspring can migrate to one of the nodes connected to the node in which his parent is located. The distance between two groups takes on one of the values 1, 2, 3. The migration range r means that the displacements generated by a single-step migration form the set Ω(r) = {1, … , r} whose elements are performed equiprobably.  The pairwise competition plays an overriding role in determining the multi-strategy selection for extremely low mutation probabilities. It is intuitive since there exist simultaneously at most two strategies in the population. By letting u = 1 in equation (3), we get σ Mo 1 and σ Mo 2 for u = 1 as The competition of multiple strategies with equal frequency plays an overriding role in affecting the multi-strategy selection for sufficiently large populations and mutation probabilities (the condition for strategy k to be favored becomes ). It is easy to calculate σ Mo 1 and σ Mo 2 for v = 0 as Mo Mo 1 2 As the mutation probability u increases, the pairwise competition fades out of the multi-strategy selection and the competition of multiple strategies with equal frequency gradually dominates the multi-strategy selection. In the absence of migration (v = 0), the long-term population, in which the absorbing state is that all individuals are located in one group, evolves just like the well-mixed population. Therefore, equation (7) also gives the values of are approximated as 1 and Nu respectively, which is in agreement with the previous study 48 .
When there exists migration (v ≠ 0), the comparison of σ Mo 1 and σ Mo 2 (the dependence of the multi-strategy selection on the pairwise competition and the competition of multiple strategies with equal frequency) is still mainly determined by the mutation probability u (Fig. 3a,b). In contrast to the well-mixed population (or v = 0), in which σ Mo 1 decreases monotonically with the increase of u, migration (moderate migration probabilities) causes the change of σ Mo 1 with respect to u to exhibit an inverted U-shaped curve. Similar to the well-mixed population (or v = 0), the value of σ Mo 2 is still proportional to u with a coefficient around the population size N. This verifies the previous conjecture 33 that σ 1 ≪ σ 2 holds for large Nu. Low mutation probabilities which are extremely close to zero lead to σ σ >

Mo
Mo 1 2 , which means that the pairwise competition has an advantage over the competition of multiple strategies with equal frequency in determining the multi-strategy selection. Whereas the remaining vast majority of mutation probabilities result in σ σ <

Mo
Mo 1 2 , which means that the competition of multiple strategies with equal frequency gains the advantage over the pairwise competition.
We now focus on the effect of migration (probability and range) on σ Mo 1 and σ Mo 2 (Fig. 3). The migration probability or the migration range leads to a significant change of σ Mo  Comparing the Wright-Fisher process with the Moran process. Similar to the Moran process, the parameters σ 1 and σ 2 of the Wright-Fisher process can be expressed by the probabilities (under neutral selection) assigned to the event that three randomly chosen (without replacement) individuals use given strategies and locations. We obtain such probabilities of the Wright-Fisher process (see Supplementary Information: IV for the detailed calculations) for any mutation patterns and migration patterns following the example of the Moran process 31 . Applying these probabilities to 'global mutation' of our model (see Supplementary   where Fig. 4. Additionally, they do not involve the number of strategies (S), which is in line with the previous literature 33 .

. The expressions are suitable for any 'isotropic' migration patterns (f(x)), non-zero mutation probabilities (u), migration probabilities (v), population sizes (N), and group numbers (M), and have been verified by Monte Carlo simulations in
After simple calculations, we have  Just like the Moran process, the pairwise competition dominates exclusively the multi-strategy selection for extremely low mutation probabilities, and the competition of multiple strategies with equal frequency for sufficiently large populations and mutation probabilities. For the group-structured population without migration (v = 0) or the well-mixed population, (1 st Taylor expansion) indicating that σ Wr 2 is twice σ Mo 2 (of the Moran process). The result is in accordance with the previous literature 48 , since the expressions of σ 1 and σ 2 are the same for the Moran and the Wright-Fisher process in the sense that Nu in the former and 2Nu in the latter are denoted by the same symbol when the population is sufficiently large 28 .
Most findings of the Wright-Fisher process are qualitatively similar to those of the Moran process (Fig. 5). Compared with the well-mixed population, migration (moderate migration probabilities) causes σ Wr There are two qualitatively distinct ways to change σ 1 and σ 2 between the Moran and the Wright-Fisher process (Figs 3 and 5). The curve of σ Mo 1 with respect to the migration probability v only has one peak at a low v; and yet in addition to this peak, there may appear a new local peak for σ Wr 1 at v = 1. The value of σ Mo 2 is linearly increasing with respect to the mutation probability, and yet σ Wr 2 is increasing with a decreasing speed. We also quantitatively compare the two processes based on the values of σ 1 and σ 2 (Fig. 6). The Moran process is more effective in increasing σ 1 than the Wright-Fisher process for a vast majority of migration probabilities. The Wright-Fisher process is always more effective in increasing σ 2 than the Moran process, which can be seen from the value of σ σ − Wr Mo 2 2 for v = 0 since σ Wr 2 and σ Mo 2 change very little with the migration probability, Application: Direct reciprocity. We now use the above findings to study the evolution of cooperation under direct reciprocity. Assume that any two individuals of the same group play m rounds of interactions. In any one round, a cooperator brings a benefit b to his opponent at a cost c (b > c > 0), and a defector brings no benefits and pays no costs. Each individual adopts one of three strategies: ALLC (always cooperate) meaning one cooperates in all rounds, ALLD (always defect) meaning one defects in all rounds, and TFT (tit-for-tat) meaning one cooperates in the first round and follows his opponent's strategy in the previous round. The payoff matrix is given by From equation (2), the condition for cooperation to be favored over defection (the average frequency of ALLD over the stationary distribution under weak selection, 〈 x ALLD 〉 δ→0 , satisfies 〈 x ALLD 〉 δ→0 ≤ 1/3) is given by Larger critical cost-to-benefit ratio, (c/b) * , shows that the evolution of cooperation is favored more in the sense that more values of c/b allow natural selection to favor the evolution of cooperation. The effects of the mutation probability (u), the migration probability (v), the migration range (r), and the repetition round (m) on (c/b) * are qualitatively similar but quantitatively different for the Moran and the Wright-Fisher process (Fig. 7). For large mutation probabilities, σ 2 is the key determinant of (c/b) * compared with σ 1 since σ 2 ≫ σ 1  ) does not change much with u (large u in Fig. 7a,c), and it is almost identical for the Moran and the Wright-Fisher process irrespective of m and v (Fig. 7f). For small mutation probabilities, σ 1 and σ 2 jointly determine (c/b) * since σ 1 has a similar size to σ 2, and σ 1 is the major determinant. Here, (c/b) * changes a lot with u (low u) and its changing trend with respect to u varies with v (Fig. 7a,c): When the migration probability v is small or large, (c/b) * increases with u, because σ 1 becomes smaller (decreasing the dependence of the multi-strategy selection on the pairwise competition enhances the evolution of cooperation); When v is moderate, the change of (c/b) * with u is roughly decreasing with the increase of σ 1 . Meanwhile for small mutation probabilities (Fig. 7e), the Wright-Fisher process leads to greater values of (c/b) * than the Moran process for small and large migration probabilities satisfying σ σ <

Mo
Wr 1 1 , and the reverse holds for the remaining majority of migration probabilities satisfying σ σ >

Mo
Wr 1 1 . Migration (probability and range) changes σ 2 very little and thus the effect of migration on (c/b) * is similar to its effect on σ 1 (Fig. 7b,d): There exists a moderate migration probability maximizing (c/b) * ; The optimal migration range corresponding to the maximum value of (c/b) * is from the longest to the shortest range as v increases (the advantage of the optimal intermediate range over other ranges is negligible). Larger repetition round (m) increases (c/b) * by enhancing the inherent payoff advantage of TFT and ALLC over ALLD (Fig. 7e,f).

Discussion
It has been proved that the strategy selection under weak selection can be expressed by several parameters (including one) independent of the payoff matrix not only for two-person games 32,33 but also for multi-person games 49 . These parameters play a vital role in determining the strategy selection, but are difficult to calculate for general models. For only a few particular models, they have been obtained for two-person and two-strategy games 47,50 , two-person and multi-strategy games 33 , and multi-person and two-strategy games 51,52 . In this paper, we have focused on two-person and multi-strategy games whose strategy selection can be expressed by two parameters σ 1 and σ 2 . We have calculated the accurate values of σ 1 and σ 2 for the Moran and the Wright-Fisher process in group-structured populations of any finite sizes. In a previous study 33 , the values of σ 1 and σ 2 have been given for large populations. The assumption of large population size guarantees that the calculation procedures are the same for the Moran and the Wright-Fisher process when Nu in the former and 2Nu in the latter are denoted by the same symbol. In finite populations, however, the calculations of σ 1 and σ 2 vary a lot between the For high u (u = 0.9), the pairwise competition exerts a tiny influence on the multi-strategy selection compared with the other competition (σ 2 ≫ σ 1 ), and therefore we only compare σ Wr 2 with σ Mo 2 . σ Wr 2 is greater than σ Mo 2 for all v. (c) For moderate u (u = 0.01), the two types of competition jointly determine the multi-strategy selection, and thus we compare the two processes based on both σ 1 and σ 2 . The comparison of σ 1 here is qualitatively similar to that of σ 1 for u = 0.001. The comparison of σ 2 here is qualitatively similar to that of σ 2 for u = 0.9. Parameters: N = 100, M = 7, r = 1.
two processes. Accordingly, we have separately provided their concrete calculation procedures in Supplementary Information. The key point in the two procedures is how to obtain some special probabilities under neutral selection. The special probabilities of the Moran process have been given by the previous research 31 to investigate the evolution of cooperation in populations with two layers of group structure (whose strategy selection cannot be expressed by σ 1 and σ 2 ). The special probabilities of the Wright-Fisher process are rigorously obtained for the first time in this work. The values of σ 1 and σ 2 we have calculated are appropriate for any migration patterns, mutation probabilities, migration probabilities, population sizes, and group numbers, and have been verified by Monte Carlo simulations. In previous studies 33,48 , the values of σ 1 and σ 2 have been obtained for large populations. A recent study 47 has suggested the population size suitable for these studies 33,48 is related to the mutation probability, the migration probability, and the group number. Our studies can produce their results with the assumptions of 'global migration' and large populations. We also have verified the previous conjecture 33 that σ 2 is far larger than σ 1 when the product of the population size and the mutation probability is large. Moreover, we have obtained some new findings for the Moran and the Wright-Fisher process. Compared with the well-mixed population, migration modifies the relationship between σ 1 and the mutation probability from a monotonically decreasing curve to an inverted U-shaped curve, and maintains the increasing relationship between σ 2 and the mutation probability. The mutation probabilities for σ 1 > σ 2 (the pairwise competition dominates the multi-strategy selection) are extremely close to zero, and yet those for σ 1 ≤ σ 2 (the competition of multiple strategies with equal frequency dominates the multi-strategy selection) are the remaining vast majority.
We have studied how migration (probability and range) affects σ 1 and σ 2 , and the following findings hold for the Moran and the Wright-Fisher process. Migration leads to a significant change of σ 1 and a relatively negligible one of σ 2 . There exists a moderate migration probability maximizing σ 1 . The migration range leading to the largest value of σ 1 decreases as v increases. Prior research 47 has studied two-person and two-strategy games whose strategy selection can be expressed by a single parameter σ, and has analyzed the effect of migration on σ for the Moran process. The way that migration varies σ 1 (of the multi-strategy selection) is similar to how it changes σ (of the two-strategy selection). This is understandable from the known equality 33  , our results can be used to obtain σ of the Wight-Fisher process and further analyze the influence of migration on σ. Our results are independent of the payoff matrix and can be applied to any concrete game with multiple strategies. In particular, we have determined how migration affects the evolution of cooperation under direct reciprocity by considering the competition of ALLC, ALLD, TFT.
Besides games on islands 30 and games in group-structured populations 31 , games on graphs (a single node only holds one individual at most and individuals can only migrate to empty nodes) have also been used to find what kind of migration can promote the evolution of cooperation [53][54][55][56] and maintain the biological diversity 57,58 . Migration can be envisaged as a way to generate the coevolution of the population structure and individuals' strategies. Compared with conventional co-evolutionary rules [59][60][61] , migration here is a very simple rule in which individuals do not have to know any information about the environment, but moderate migration (probabilities) can promote the evolution of cooperation efficiently. This suggests that moderate change of the interactive structure may be the key factor for the evolution of cooperation in the co-evolutionary dynamics.
Nearly all previous studies have only focused on either the Moran or the Wright-Fisher process [28][29][30][31]47,48 . Here, we have analyzed the distinct roles of the two processes in the multi-strategy selection. The curve of σ 1 with respect to the migration probability v has only one peak at a moderate v in the Moran process; yet in addition to this peak, it can have a new local peak at v = 1 in the Wright-Fisher process. The value of σ 2 is almost linearly increasing as the mutation probability expands in the Moran process, and yet it is increasing with a decreasing speed in the Wright-Fisher process. The Moran process is more effective in increasing σ 1 for the vast majority of migration probabilities and the Wright-Fisher process is always more effective in increasing σ 2 .
Our model is a variant or a special case of previous models [28][29][30] , and our calculations of σ 1 and σ 2 can be extended to the multi-strategy case of these models after proper modifications. In contrast to the stepping-stone model 1-5 , our model requires that all subpopulations have changeable sizes instead of a fixed and equal size. Our assumption is more realistic and can bring about more interesting phenomena by introducing the typical feature of group-structured populations, 'the asynchrony in local extinction and recolonization' . Our investigation may provide some insights into the extension of the stepping-stone models. In turn, the newest developments about the stepping-stone models may give us some ideas for analytically studying more complex and realistic models.