This page has been archived and is no longer updated


Game Theory, Evolutionary Stable Strategies and the Evolution of Biological Interactions

By: Charles C. Cowden © 2012 Nature Education 
Citation: Cowden, C. C. (2012) Game Theory, Evolutionary Stable Strategies and the Evolution of Biological Interactions. Nature Education Knowledge 3(10):6
If natural selection is survival of the fittest, why isn't everything a competition? Cooperation and competition have a fitness face off in game theory.
Aa Aa Aa


Cowden banner.

Evolutionary ecologists aim to understand the complex behavioral relationships between organisms as they interact to obtain resources. In general, these interactions range from antagonistic to cooperative, but cases of exploitation and altruism also occur. Interactions are costly: energy is invested by each organism in both confrontation and cooperation as a means to acquire the resource. Indeed, even avoidance can be costly if energy is expended and no resources are obtained. The energy spent is a cost to the organism, and the resources are benefits. Comparing the relative cost to the benefits obtained following an interaction determines the net gain or loss incurred by the organism, and this value is referred to as the payoff. Different interaction strategies, such as combative or cooperative, result in different payoffs based on nature of the interaction. Evolutionary ecologists treat these strategies as phenotypes. The most successful organisms maximize their payoff and increase their ability to reproduce. In short, the organism with the best interaction strategy has the highest fitness. Since the interaction strategy (phenotype) can directly relate to fitness, the optimum strategy will be favored under natural selection.

Interactions between organisms with either competing or identical strategies can be treated as games with multiple players. Since biological interactions involve two or more decision makers (i.e., individuals with strategies), biologists utilize game theory to elucidate evolutionary consequences of interactions1. Game theory is a branch of mathematics that investigates the outcomes of multi-individual interactions, in which the payoff for any one individual depends not only on its own strategy, but the strategies of the other individuals involved. An individual does not control the decisions of others. While much of game theory is based on the interactions of rational decision makers (humans), in evolutionary game theory the payoff is a surrogate for fitness and strategies evolve over many repeated iterations of the game2,3. Estimating fitness for strategies based on the payoff matrix reveals stable payoff equilibriums may be achieved through natural selection, an important feature of evolutionary game theory. Furthermore, stable payoff equilibriums have been shown for cooperation and altruism, behaviors that seem contradictory to the strongly supported individualistic, survival of the fittest mode of evolution3.

Evolutionary games are often visually represented as a payoff matrix (Fig. 1a), in which the payoff for the individual involved is shown for all possible interactions. In the matrix, the payoffs typically are those assigned to the player depicted in the row playing against the strategy in the column; thus, in Figure 1a, if player 1 is a hawk, and encounters a hawk, the payoff for player 1 is ½ (B-C), the value in the upper left quadrant. If player 1 is a hawk and encounters a dove, the payoff for player 1 is B, the value in the upper right quadrant. The payoffs for player 1 adopting the dove strategy against a hawk and against a dove are in the lower left and lower right quadrants, respectively.

Hawk versus Dove.
Figure 1: Hawk versus Dove.
The general payoff matrix for the ‘hawk vs. dove’ game (a). The payoffs are for player 1 employing the strategies in the rows against player 2 employing the strategy in the columns. The symbol B represents the benefit of acquiring the resource, and the symbol C is the cost of combative interaction. In the upper left quadrant, an encounter of hawk-vs.-hawk, the payoff to player 1 is ½ the benefit minus the cost, and assumes an equal payoff for both hawks. A hawk encountering a dove obtains the entirety of the benefit, while the dove receives none, as shown in the upper right and lower left quadrant, respectively. If both players adopt the dove strategy, the mutual payoff is the highest, with both players evenly dividing the benefit without cost of conflict (bottom right quadrant). Shown in (b) are the values of the benefit (4) and the cost of combat (3) used for the simulation for a population of 1000 individuals (c). Although the highest mutual payoff occurs when both players are doves, the dove strategy is not an evolutionary stable strategy. The hawk strategy will invade and displace the dove strategy in a population even if the initial number of hawks is very low relative to the doves if the cost of the combat does not exceed the reward of the benefit. The likelihood of a dove encountering a hawk is initially very low, but as hawks increase in number, that likelihood increases. The population-level outcome at any encounter depends on the population makeup resulting from previous encounters.
© 2012 Nature Education All rights reserved. View Terms of Use

Evolutionary Stable Strategies

In some populations, all individuals may have the same strategy phenotype. Such a strategy is said to be an evolutionary stable strategy (or ESS) if that strategy cannot be replaced, or invaded by any other strategy through natural selection4,5. Formally, and ESS must satisfy two conditions4: 1) an individual employing strategy A must do better against another individual employing strategy A than any other strategy; and 2) should a new strategy evolve (A') that does equally well against strategy A, for A to be an ESS, an individual employing strategy A must do better against an individual employing strategy A' than an individual employing strategy A'.To illustrate an ESS, let us consider a payoff matrix between two organisms each with an aggressive strategy and a passive strategy: the classic ‘hawk vs. dove' game4 (Fig. 1a)4. A hawk is aggressive and initiates a combative interaction when confronted; alternatively, the dove is passive and avoids combative interactions. A hawk may encounter another hawk, or a dove; likewise, a dove may encounter another dove, or a hawk. If the two individuals adopt a dove strategy, the resources are partitioned equally. If a dove encounters a hawk, the hawk acquires the totality of the resource, while the dove receives none. Lastly, if a hawk encounters another hawk, the resources are partitioned equally; however, there is a cost incurred by each hawk due to the aggressive interaction. A population of doves may be invaded by a hawk, and the hawk strategy will displace the dove strategy (Fig. 1c). However, the hawk strategy is an ESS only if the value of the resource is greater than the cost of the conflict (B > C > 0). If the cost of conflict is greater than the reward, then neither the hawk nor the dove strategy satisfies the conditions of ESS as a pure strategy.


The hawk vs. dove game reveals that competitive strategies may reach a stable evolutionary equilibrium, but how can a cooperative strategy persist in an evolutionary equilibrium? Consider another classic game theory problem - the Prisoners' Dilemma. The conditions of the Prisoners' Dilemma are such that two prisoners are captured for, and guilty of the same crime, and each prisoner has the choice to confess or to remain silent. Examining the payoff matrix (Fig. 2a), the largest mutual benefit occurs when both prisoners remain silent. However, the highest possible benefit for a single prisoner occurs when he confesses, and implicates the other, while the second prisoner remains silent. If both confess, there is a reduced punishment for both. For instance, if prisoner 1 confesses and prisoner 2 remains silent, prisoner 1 is rewarded and prisoner 2 receives the harshest punishment. If prisoner 1 always chooses to confess, he avoids the harshest punishment. If we consider only a single encounter between prisoners, then the only evolutionary stable strategy is to always confess and implicate the other prisoner (Fig. 2b).

The Prisoners' Dilemma.
Figure 2: The Prisoners' Dilemma.
The payoff matrix from the perspective of Prisoner 1 (a), the best payoff is for both to remain silent (cooperate, a benefit of 4 each). However, if both players confess (defect), each receives a reduced sentence (a benefit of 2 each). The highest benefit possible is for a defecting prisoner that is rewarded for confessing and implicating the silent prisoner (a benefit of 5 for the defector, 0 for the silent prisoner). The strategy of always defecting is superior (b). In the Iterated Prisoners’ Dilemma, there is a probability that two prisoners will be arrested together multiple times. Cooperation between pairs of prisoners may evolve via a ‘tit-for-tat’ strategy in which the prisoners decision to confess or remain silent is determined by the previous arrest (c). In ‘tit-for-tat’, the prisoner is assumed to remain silent on the first arrest, but then adopt the strategy of his opponent for the next arrest. Cooperation becomes an evolutionary stable strategy if the probability of encountering a cooperating prisoner is high.
© 2012 Nature Education All rights reserved. View Terms of Use

How does this example lead to cooperation in biological interactions? For biological interactions, it is generally assumed that the same two individuals will interact more than once (so called iterated prisoners' dilemma)3. At the population level, the game will be repeated over evolutionary time scales (hundreds to thousands of generations or more). Recall that cooperation results in the highest mutual benefit for both players. Thus, with repeated games cooperation can evolve through the simple strategy of ‘tit for tat' described by Axelrod and Hamilton3. In the ‘tit for tat' strategy, individual 1 begins the game with cooperating (remaining silent). In subsequent iterations of the game, individual 1 uses the strategy of individual 2 from the previous round. That is to say, if individual 2 cooperated in the first round, then individual 1 will cooperate in round two. Likewise, individual 1 will confess in round two if individual 2 confessed in the first round. The strategy was tested using computer simulations, and cooperation emerged as an evolutionary stable strategy if the probability of two individuals meeting more than once was sufficiently high3 (Fig. 2c). Thus, cooperation can evolve in small populations through strategies based on simple reciprocity.

Complex Interactions

The examples so far have used two-player, fixed payoff games to demonstrate the evolution of different interaction strategies. Biological interactions and strategies are often more complex. Strategies are considered pure if the behavior is constant for all interactions, or they may be considered mixed if the behavior expressed is conditional on the organisms involved4,6. For instance, a mixed strategy may be size related: avoid larger individuals and attack smaller individuals. The value or quality of the benefit also influences strategies. From an evolutionary standpoint, the reward for competitive behavior must outweigh the risk and cost of competition. Thus, low quality resources are unlikely to drive intense combat. However, the higher the quality or importance of the benefit, the more intense the competition will be.

Consider that many animal species are strongly territorial, and territories vary in quality. If a larger individual encounters a smaller individual, the larger individual will often begin aggressive gestures towards the smaller individual. The smaller individual will often flee to avoid the conflict. The likelihood of the larger individual being hurt or killed by the smaller individual is minimal, so the risk of the aggressive behavior is minimal. Likewise, the smaller individual must avoid conflict with larger individuals in which they may be severely harmed or killed. Smaller individuals often establish lower quality territories, but still have the opportunity to reproduce. The most intense conflicts occur between individuals of equal size fighting for higher quality territories, and in this fighting both individuals may receive sufficient damage due to combat so that neither can reproduce. The benefit of avoiding these detrimental costs of combative encounters has led to the evolution of ritualized combat for mates and territory where the contestants avoid severe harm. Common examples of ritualized combat include male fiddler crabs that engage in displays of the larger claw to demonstrate superiority, black rat snakes that engage in elaborate instigator-responder movements, and male red deer that bellow and display their antlers to potential combatants competing for mates. These visual cues that animals display often avoid combat when an individual demonstrates physical superiority; however, combat may escalate between evenly matched individuals or when the seemingly inferior combatant continues the aggressive display.

In the prior examples, we assume a uniform environment, and resources are available everywhere. In natural environments, spatial and temporal variability in resource abundance and quality influence biotic interactions7,8; moreover, organisms modify the environment by utilizing resources. Payoff matrices may be either symmetric or asymmetric. In symmetric payoff matrices, payoffs are identical for each individual based on the interaction, whereas payoffs may be different for each individual in an asymmetric payoff matrix. A common asymmetric game occurs between organisms competing for territories in which one player occupies the territory and the other is the intruder. The payoff in the game is the territory, and the asymmetry is that the territory holder may either maintain or lose the territory, while the intruder may or may not acquire the territory. The territory holder is more likely to escalate the conflict via an aggressive strategy to protect the territory, while the intruder is less likely to escalate9. A second asymmetry that arises in animal conflicts is the different abilities to acquire or hold a resource such that the cost of escalating the interaction differs due to combative capability10. Here, the superior competitor is more likely to employ an aggressive strategy.

Payoff modification is possible under both symmetric and asymmetric models in which individuals have the ability to reward more beneficial partners and punish non-cooperators11. The ability to modify payoffs based on partner interactions is a fundamental component to most mutualism stability theories12. Consider a symmetric payoff matrix in which payoffs are modified based on cooperation or cheating (Fig. 3a). Here is a model for two symbiotic organisms involved in trade for resources. For example, consider a legume (bean plant) that associates with bacteria that provide nitrogen the plant can use. These bacteria live within special nodules on the plant root and rely on the plant for energy in the form of sugar. Thus, cooperation involves an investment by both partners. In the payoff matrix, a modification value (m) increases the payoff if both partners cooperate. If cheating occurs (one partner cooperates, the other does not), then the payoff is reduced by m. If a bacterium receives more sugar, it is able to provide more nitrogen to the plant. Given more nitrogen, the plant can photosynthesize more to produce sugars. There is thus a positive feedback for cooperation. So long as the partners trade with fidelity (an increase in benefit is reciprocated honestly), cooperation emerges as an ESS (Fig. 3b).

Reward and Punishment.
Figure 3: Reward and Punishment.
Payoff matrices may include punishment for cheating and reward for cooperation (a). A cheater chooses to defect while the other organism cooperates. If an organism is cheated, the benefit in the next round is reduced by the modifier value (m), representing a punishment for cheating. Alternatively, m is a reward for cooperation. The modification of payoffs based on cooperation and cheating results in rapid evolution of cooperation if both organisms trade honestly (b).
© 2012 Nature Education All rights reserved. View Terms of Use


Evolutionary game theory in which cooperation is rewarded with fidelity provides rationale for altruistic behavioral strategies seemingly incongruent with natural selection. In evolutionary terms, altruism occurs when an individual reduces its own immediate fitness probability (i.e. its likelihood to reproduce in the short term) to improve the fitness of another individual. Altruism is most common in animals that exhibit social behavior, such as warning calls or food sharing, and colony forming behavior. The benefit of warning calls and food sharing is based on reciprocal altruism, namely that the benefits provided to others will be returned in the future13. A bird that provides a warning call is temporarily more apparent to predators, thus reducing its immediate fitness probability. However, the same bird may benefit in the future from warning calls from other birds. Likewise, a vampire bat that regurgitates its food to feed hungry bats will benefit from the same service when it is unable to find food14. Reciprocal altruism relies on organisms interacting multiple times, and game theory provides a framework for the evolution of reciprocal altruism via the prisoners' dilemma described earlier. The iterated prisoners' dilemma typically involves repeated interactions between small numbers of prisoners; however, altruism has been shown to evolve even in large groups through a combination of direct and indirect altruistic reciprocity15. Direct reciprocity occurs when the same two individuals interact repeatedly, while indirect reciprocity is when subsequent interactions are between different partners. In sum, while altruistic behavior may temporarily reduce the fitness of the individual, reciprocal altruism increases fitness over the lifetime of the individual.

(Note: Inclusive fitness is a common explanation for the evolution of altruistic behavior that does not necessarily depend on game theory. Altruistic alleles increase in a population if the cost to the altruistic individual is less than the reproductive benefit of the recipient multiplied by the likelihood of the recipient passing on the altruistic allele to its offspring (Hamilton's Rule)16. Inclusive fitness is often associated with kin selection, as more closely related organisms more likely share the same alleles - such alleles are referred to as ‘identical by descent' as they are from a common ancestor. However, altruism genes may be found in non-related individuals, thus relatedness is not a strict requirement of inclusive fitness.)


The principles of game theory provide a theoretical framework for understanding the evolution of biological interactions. Evolutionary game theory applies to organisms that interact repeatedly, both within a generation and over evolutionary relevant timescales. In special cases, evolutionary stable strategies emerge in which a particular strategy is adopted by all members of a population and alternative strategies (mutant phenotypes) cannot invade and displace the ESS. The flexibility of game theory allows for the testing of complex mixed strategies and incorporating reward and punishment in evolutionary strategies. Importantly, game theory demonstrates the evolution of cooperation and altruism ESSs is consistent with evolution through natural selection.


Altruism: When one organism reduces its own fitness to benefit the fitness of another organism.

Evolutionary stable strategy: A behavioral strategy (phenotype) if adopted by all individuals in a population that cannot be replaced or invaded by a different strategy through natural selection.

Game theory: A mathematical approach to understanding the outcomes of interactions between two or more individuals when benefits and costs of the interactions depend on the strategies of each individual.

Inclusive fitness: The fitness of a gene as measured by the fitness of the individual possessing the gene and the fitness of the individual's relatives bearing the same gene, identical by descent.

Mutualism: A relationship between two individuals from different species that benefits each individual involved in the interaction.

Phenotype: The physical, physiological, behavioral and other traits expressed by an individual.

References and Recommended Reading

1 Smith, J. M. Game theory and the evolution of behavior. Proc. R. Soc. Lond. Ser. B-Biol. Sci. 205, 475-488 (1979).

2 Axelrod, R. & Dion, D. The further evolution of cooperation. Science 242, 1385-1390 (1988).

3 Axelrod, R. & Hamilton, W. D. The evolution of cooperation. Science 211, 1390-1396 (1981).

4 Smith, J. M. Evolution and the Theory of Games. Cambridge University Press, 1982.

5 Taylor, P. D. & Jonker, L. B. Evolutionary stable strategies and game dynamics. Math. Biosci. 40, 145-156 (1978).

6 Crowley, P. H. Hawks, doves, and mixed-symmetry games. J. Theor. Biol. 204, 543-563 (2000).

7 Nowak, M. A. & May, R. M. Evolutionary games and spatial chaos. Nature 359, 826-829 (1992).

8 Enquist, M. & Leimar, O. Evolution of fighting behavior - the effect of variation in resource value. J. Theor. Biol. 127, 187-205 (1987).

9 Hammerstein, P. The role of asymmetries in animal contests. Anim. Behav. 29, 193-205 (1981). doi:10.1016/s0003-3472(81)80166-2

10 Smith, J. M. & Parker, G. A. Logic of asymmetric contests. Anim. Behav. 24, 159-175 (1976). doi:10.1016/s0003-3472(76)80110-8

11 Bull, J. J. & Rice, W. R. Distinguishing mechanisms for the evolution of cooperation. J. Theor. Biol. 149, 63-74 (1991).

12 Doebeli, M. & Knowlton, N. The evolution of interspecific mutualisms. Proc. Natl. Acad. Sci. USA 95, 8676-8680 (1998).

13 Trivers, R. L. Evolution of reciprocal altruism. Q. Rev. Biol. 46, 35-& (1971). doi:10.1086/406755

14 Wilkinson, G. S. Reciprocal food sharing in the vampire bat. Nature 308, 181-184 (1984). doi:10.1038/308181a0

15 Suzuki, S. & Akiyama, E. Evolutionary stability of first-order-information indirect reciprocity in sizable groups. Theor. Popul. Biol. 73, 426-436 (2008). doi:10.1016/j.tpb.2007.12.005

16 Hamilton, W. D. Evolution of altruistic behavior. Am. Nat. 97, 354-& (1963). doi:10.1086/497114


Flag Inappropriate

This content is currently under construction.

Connect Send a message

Nature Education Home

Population Ecology

Visual Browse