Network Modularity is essential for evolution of cooperation under uncertainty

Cooperative behavior, which pervades nature, can be significantly enhanced when agents interact in a structured rather than random way; however, the key structural factors that affect cooperation are not well understood. Moreover, the role structure plays with cooperation has largely been studied through observing overall cooperation rather than the underlying components that together shape cooperative behavior. In this paper we address these two problems by first applying evolutionary games to a wide range of networks, where agents play the Prisoner's Dilemma with a three-component stochastic strategy, and then analyzing agent-based simulation results using principal component analysis. With these methods we study the evolution of trust, reciprocity and forgiveness as a function of several structural parameters. This work demonstrates that community structure, represented by network modularity, among all the tested structural parameters, has the most significant impact on the emergence of cooperative behavior, with forgiveness showing the largest sensitivity to community structure. We also show that increased community structure reduces the dispersion of trust and forgiveness, thereby reducing the network-level uncertainties for these two components; graph transitivity and degree also significantly influence the evolutionary dynamics of the population and the diversity of strategies at equilibrium.

T he earliest formation of the very building blocks of life, such as RNA, may not have cooperatively formed into complex biological systems without the structural scaffolds required for catalytic processes to evolve 1,2 . Just as these building blocks required structured interactions in order to evolve, the structure of social interactions also aide in the emergence of cooperative behavior [3][4][5][6][7] .
Models of emergence and survival of cooperation among selfish agents are often based on game theoretic social dilemma constructs which capture the interaction of multiple players simultaneously or relatively simple pairwise interaction where each player only interacts with a single player at a time. While the former, usually modeled as a public goods game 8,9 , is useful for understanding tragedy of the commons type situations that challenge prosocial behavior [10][11][12] , the latter, adopted in this work, allows exploration of a straightforward strategy space and also lends itself directly to network-based simulation methods. Here agents in a pairwise interaction have two actions to choose from, cooperate or defect. While mutual cooperation provides a Pareto efficient outcome, each player individually gets less than if they were to defect while the other cooperates. This makes mutual defection the Nash Equilibrium of the single shot game, thereby creating a social dilemma where both players together receive a higher payoff if they cooperate but maximize their minimum payoff by mutually defecting. The relative payoffs for each combination of player moves can be changed to result in games with different characteristics or dilemma strengths 13,14 .
The gloomy prediction of mutual defection made by single-shot social dilemma games contrasts with observations of real world social behavioral in which cooperation between players is common 15 . This difference has been ascribed to repeated interactions and incomplete information 16 , as well as the role of interaction structure as our scope widens from two player interaction to the community level. Spatial and structural effects on behavior have commonly been studied based on evolutionary game theory as the underlying theoretical framework and using agent-based simulation. These studies show that structure has a strong effect on cooperative behavior [17][18][19] . Local coupling within network structures enables cooperators to cluster together and survive in an environment where defectors have the upper hand 14 ; however, defectors are weakened when surrounded by their own kind 20 . If strategy and structure are allowed to evolve together, defection tends to break apart tightly clustered cooperative communities 21 .
Since social ties allow for cooperators to cluster together and improve their survival, are more connections always better? Not necessarily, since many studies have demonstrated the opposite effect. Foundational work by Ref. 22 and Ref. 23 demonstrated that cooperative or altruistic genes may dominate a neighborhood when distributed locally due to an increase in the relatedness of individuals; however, this localized dispersion creates competition for resources that naturally limits the spread of these genes. This balance between sufficient structure for interactions between like individuals and the challenges of local competition was later studied by Ref. 17, who found that relatively sparse connectivity encourages cooperation with the intuition that agents with fewer ties value their relations more than agents with more ties and so are more likely to exhibit prosocial behavior in order to maintain those ties. These opposite effects suggest that link distribution rather than merely link density plays a significant role in the survival of cooperation. The effect of link distribution has been shown for some special structures, such as scale-free networks with power-law connectivity, where they support higher cooperation levels than homogeneous networks [24][25][26][27] , though this affect may be limited by participation costs 28 as well as frequency of interaction where more frequent interaction can negate the cooperative advantage of heterogenous networks over homogeneous 29 . Beyond some of these special structures, the current literature lacks a general understanding of the way link distribution affects cooperation.
Cooperation is not a single-parameter behavior, rather it is composed of a host of components such as trust, reciprocity and forgiveness, that together shape cooperative behavior, yet the majority of studies regard cooperation as a whole. Moreover, many of these works are based on deterministic models that naturally neglect a wide range of mixed strategies that emerge in real-life situations and shape the overall cooperative behavior of a social network. A model suggested by Nowak and Sigmund 30,31 can address both of these issues at the same time by introducing stochastic strategies and breaking them down into three different components of cooperative behavior, (y, p, q), where y is the probability of cooperating (c in equation (2)) on the first move of a repeated two-player game, p is the probability of cooperating when an opponent cooperates, and q is the probability of cooperating after an opponent defects (d in equation (2)), hence we characterize each of the components of the (y, p, q) triple as trust, reciprocity, and forgiveness respectively for this work 30 . Despite its advantages, to the best of our knowledge, the use of this model for the study of structural effects is limited to Ref. 32 for lattice structures and Ref. 33 who study the effects of cognitive capacity of agents in scale-free networks. In this paper, we use this construct as the building-block of our model and will refer to it as the (y, p, q) model. The goals of this paper are two-fold. Our first goal is to identify network structural characteristics that have the highest impact on the emergence and survival of cooperative behavior. These characteristics will also give us the necessary tools to tackle the link distribution effect described earlier. Our second goal is to investigate the impact of network structure on individual components of cooperative behavior (namely trust, reciprocity, and forgiveness). To further these goals we employ stochastic agent-based simulations on a wide range of network structures using the (y, p, q) model, and perform multivariate statistical analysis on simulation results. We utilize these methods to provide evidence for the key role of modularity in the evolution of cooperation. Modularity can neutralize high stakes games, similar to how active linking, or giving agents the ability to manage their local connections 34 , can reduce the effective strength of cooperative dilemmas. Finally, we show the leadership role of forgiveness in underpinning sustained cooperation in a noisy environment, where forgiveness emerges first in order to dampen the effect of defectors and this allows for reciprocal behavior and trust to emerge.

Results
Our agent-based 35 model is built upon an evolutionary games on graphs 3 framework and operates on four levels: pairwise interaction between agents, the execution of an agent's strategy, the evolution of strategy over time, and the network structure through which interactions take place. The lowest level is where decisions are made by agents who receive payoffs in a pairwise game according to equation (4). The strategy of each agent, is a three-component stochastic vector, (y, p, q), introduced by Ref. 30. The evolutionary layer is where players imitate the strategies of their neighbors with a probability proportional to their relative payoff (see equation (5)) until an equilibrium of strategies is achieved. Above all, we structure these interactions through a variety of networks where each player occupies a node position and nodes are connected via edges or links through which games are played.
The majority of networks we have analyzed have 24 nodes. Our initial sensitivity analysis showed that graphs of approximately this size balance the trade-off between computational efficiency and sensitivity to structural variations needed for this work. Small deviations from this size include lattice structures (10 and 15 in figure 1) which have 27 nodes (3 3 3 3 3) and 25 nodes (5 3 5) respectively, and the Coxeter 36 graph (12 in figure 1) with 28 nodes. Many of the graphs were generated with a one-dimensional Watts-Strogatz (WSR) 37 model (7,8,13,16-20 in figure 1), followed by Erd} os-Rényi (ER) 38 (2-5 in figure 1), and lattice (6,10,15 in figure 1). The remainder include a variety of cubic graphs including the McGee graph 39 (9 in figure 1), a Goldberg snark 40 (G 3 described in 41 (11 in figure 1)), and the Coxeter 36 graph (12 in figure 1). The Coxeter and Goldberg graphs are non-hamiltonian in that they do not contain a path that visits each vertex exactly once, whereas the McGee graph is hamiltonian 42 . Other tested graphs include a classic ring (21 in figure 1), a fully connected graph (1 in figure 1), and a scale-free graph 43 (14 in figure 1). The variety of structures we chose was sufficient to determine the key structural characteristics, as borne out by the high variance captured in the multivariate analysis discussed later in this work.
To make our results more reliable and generalizable we relaxed three limiting assumptions that are typically used for reducing computational load and analysis complexity. First, each simulation run draws new starting strategies from a random distribution instead of picking from a set of representative strategies, this allows us to explore the strategy space more completely. Second, we randomize all three strategy components so no starting cooperative bias exists. Third, though the simulation system was designed to return only converged results on a run-by-run basis this does not ensure that the final results themselves are stable. Hence we performed a procedure where runs were continually collected until no observable shift in the results was observed. This process required a total of 5.4 core-months of execution time on a Cray XE6 computer to complete successfully.
Which are the key network characteristics effecting cooperation emergence? To help answer this question we applied principal component analysis (PCA) of the graph characteristics degree, transitivity, girth, path length, and modularity in addition to the strategy componentsỹ,p, andq at the final equilibrium of all runs. We then reviewed a biplot 44 representation, shown in figure 2 panel (B), which captures 96% of variation in the data as a whole. We calculated the modularity index based on the measurement of community structure Q developed by Ref. 45, where Q 5 1 is a maximum that indicates a strong community structure, with community detection performed by a random-walk 46 method.
From table 1, the modularity eigenvector correlates well (cos(h mod ) 5 0.995) with the first and most significant principal component, due to its alignment with the horizontal axis (h mod 5 6.75u from table 1); the primary component (PC1) accounts for 83% of variance in the data set as a whole ( figure 2 panel (B)). This indicates that modularity is most representative of the primary variation structure in the data (see Ref. 47  among all network characteristics; degree and transitivity have a similar loading though they are less correlated with PC1 (20.839 for transitivity and 20.788 for degree). Further, sinceỹ,p, andq all correlate positively with the modularity, increases in modularity will be met with increases in each of these cooperation components, which is in general agreement with Ref. 48, who study the emergence of total cooperation versus modularity. Transitivity, and degree are negatively correlated to the cooperation components; however, this holds true even more for trust (y), which directly opposes to transitivity (within 5 degrees) and degree. Hence, relatively sparse graphs with less connectivity should encourage more cooperative behavior than more dense highly-connected graphs, this agrees Ohtsuki's b/c . k rule 17 which Ref. 49 associated with Hamilton's rule 50 and the notion of inclusive fitness 51 for bi-transitive graphs (or graphs that   look the same from any pair of nodes). The direction of girth also agrees with the analytical model results from Ref. 52 who suggest that as graph girth increases cooperation is enhanced due to the improvement in local cooperator cluster formation; however, since the girth vector opposes the trust vector in the vertical direction (figure 2 panel (B)) there is a subtle secondary effect present that reduces trust relative to reciprocity and forgiveness as girth increases.
From figure 2 panel (A) the positive trend of each individual cooperative element is evident;ỹ andp have similar slopes, though y starts at a higher cooperation level. Componentq improves at a much higher rate with modularity than the other components, from figure 2, and so begins to mix withp above the 10-Lattice structure. Indeed all components improve with modularity but forgiveness (q) improves by nearly 3/2 (139%) followed by reciprocity (p) and trust (ỹ) at 1/2 and 1/5 respectively. So as communities form and become more distinct, forgiveness improves the most until a degree of balance is formed with reciprocity.
Cooperation There are significant differences in the evolutionary dynamics of the different structures tested. Indeed, run convergence time varies widely across different networks and its structural dependency can provide some interesting insights. From figure 3, for graphs with zero transitivity, higher average degree graphs tend to converge quicker than lower degree graphs; however, once the graphs become more transitive the convergence time builds significantly-to a factor of five over the fastest converging structure. Since a more transitive graph is more locally homogenous, that is the local structure is less distinctive across the network from a given local agent's perspective 49 , it follows that this lack of distinction requires more time to converge since similar strategies may become locally stable across the network but will compete in the margins for global convergence.

Discussion
All strategic components show improvement with network modularity; however, forgiveness shows the most sensitivity to structure which results in a factor of 2.4 improvement in the probability that an agent forgivesq ð Þ a defection action by an opponent for the 14-BA graph (Q 5 0.6) compared to an unstructured (well-mixed) population (Q 5 0). This makes intuitive sense because forgiveness is necessary in order for cooperation to evolve since the initial conditions are equally populated by defectors and cooperators. Without forgiveness there would be no cooperative acts to reciprocate and so higher levels of forgiveness improvement relative to reciprocity and trust indicates that forgiveness underpins the other cooperative behaviors. Moreover, forgiveness can more easily evolve in more modular structures since community boundaries and limited interaction protect the more forgiving strategies from being exploited by agents in other communities. This finding naturally extends the previous literature on the role of forgiveness in the emergence of cooperation, first demonstrated by Ref. 31 for an unstructured population and later shown to be even more significant by Ref. 32 who showed a factor of 2 improvement in forgiveness for a variety of square lattice structures compared to the unstructured model. Our results agree with this factor of 2 improvement; however, we extend this result to show that the probability of forgiveness can be even higher for structures with higher levels of modularity than lattice structures.
The impact of motivation, or payoffs, changes with modularity as well. Indeed, increasing community structure also increases trust behaviors across a range of fearful and greedy motivators. In other words, community structure improves societal trust in-general, because of the invariance of the improvement inỹ to S and T as modularity increases, shown in figure 1. Since the strategies are reactive in that they react to past moves of their opponents, initial distrusting moves cause an enduring effect that reduces cooperative levels. With higher degree networks that are less modular the effect of these distrusting moves is felt globally across the network; however, with modular structure the impact is contained to a local community level. The level of motivation, or the stakes of the game have little effect on trust because the decision to trust, as we have modeled it, The PC1 Loading column shows the relative contribution weight of each parameter to PC1 (highest independent variable loading is in bold), which is a linear combination of each parameter and weight. The angle of each parameter eigenvector (from figure 2 panel (B)) is shown in column h (degrees). The correlation between each parameter in the biplot by the cosine of the angle difference between the modularity eigenvector and each respective parameter eigenvector, or cos(h mod 2 h param ).  figure 4. Here the dispersion in reciprocityp ð Þ is nearly unaffected by modularity, however trustỹ ð Þ dispersion reduces dramatically along with forgivenessq ð Þ. This suggests that trust and forgiveness behaviors are moderated by modularity or modularity reduces the uncertainty of system-wide global trust and forgiveness behaviors under uncertain conditions. Why does the influence of payoffs on behaviors reduce as modularity increases? Community structures, or modularity, allow more diverse strategies to evolve and become stable since community boundaries reduce the influence of outside strategies on conditional cooperators. Conversely modularity can be thought of as an emergent property that improves survival in the presence of diverse environmental effects and reactive agents 54,55 . We summarize the effect of these diverse strategies as a flattening of the ST-plane gradient. The significant increase in strategy diversity is shown in figure 5 where transitive graphs tend to have less diverse strategies than non-transitive graphs. Intuitively this is because the similar local structures of the transitive graphs provide fewer local niches that support the evolution of diverse strategies.
We summarize effect of modularity and incentives (payoff) together in figure 1 panel (C). Here, the angular gradient can be interpreted with respect to modularity or payoff. From a modularity perspective, less modularity is needed to attain the same level of cooperation improvement when stakes are relatively low; however, as stakes increase more modularity is needed to sustain the same cooperative behaviors. From a payoff perspective, the influence of payoffs on sustained cooperative behaviors decreases as modularity increases, or modularity has the ability to neutralize the effect of payoffs on behaviors such that relatively high levels of cooperation can be sustained into even the strongest social dilemmas (high T).
The effect of modularity on cooperative behavior can also be shown analytically, using a simplified model (see Methods). In this deterministic model, we assume that the temptation to defect is equal to b and the higher this temptation the more modular a connected component needs to be in order for cooperation to survive and propagate. Defining a as the ratio of internal community links k in and external links k out for node i, we can show that Equation (1) aligns qualitatively with the modularity gradient in figure 1 panel (C), where increased modularity reduces the effect of higher T(b) on cooperation for p and q.
Additional insight can be gained by expanding the present work to other two-player games such as the Snowdrift and Stag Hunt games   Ref. 60, presents a two-dimensional plane of payoffs where S is received by by a player (or paid from a player when S , 0) for cooperating while an opponent defects and T is received by a player who defects while an opponent cooperates. Hence, S is representative of fear motivation (the more negative the more fear) and T is representative of greedy motivation (the more positive the more greed) 33 . The shaded region represents the payoff range of the current work, namely the Prisoner's Dilemma game. and studying the effect of modularity across the wider multi-game greed-fear (T 2 S) plane of payoffs, shown in figure 6. Initial conditions may also be varied including studying the the changes in structural effects when initially mean or nice strategies are present. Does the central role of modularity hold for a wider variety of strategies? The three-component strategy framework can be extended to four, from Ref. 31, where (p1, p2, p3, p4) are the probabilities to cooperate after a (C, C), (C, D), (D, C), and (D, D) respectively in the prior round as well as the probability to cooperate in the first round, giving a five component strategy that adds the notion of self-knowledge relative to an opponent's past moves.
Finally, given that selection strength can alter the dynamics of evolving structured populations 56 future work should also focus on testing the robustness of the present results with both strong and weak selection through the use of the more common Fermi evolutionary rule.

Methods
Our model operates at four levels: game (behavioral), strategy, evolution, and network. At the behavioral level we first adopt a simplified symmetric two-player repeated game, shown in equation (2) described by Ref. 57, 58, where the defecting player receives T (temptation to defect) if their opponent cooperates (c) but receives P 5 0 if their opponent defects (d), which is the punishment for mutual defection. In this work we limit the scope of our analysis to the Prisoner's Dilemma range of payoffs (lower right quadrant of figure 6) and reserve the other game structures for future work.
If both players mutually cooperate they each receive R 5 1; however, the cooperating player receives S , R (suckers payoff) if their opponent defects. Though P and R are fixed, we vary S and T over a range of payoffs within the Prisoner's Dilemma (PD) game, which is characterized by 1 , T , 2 and 21 , S , 0 3 , shown in figure 6.
In our simulations we draw T and S from uniform distributions within the range of PD payoffs, shown in equation (3).
All scores that players receive in repeated games are not counted equally; each iteration i of the game contributes less than the previous iteration to the total score since it is common to discount the future relative to the present 14 . We apply a discount factor w to account for this such that the i'th round is worth a mere w i of the first round score. The player with the highest total score upon completion is the winner. Equation (4) describes how the payoff P o is determined, where k o is the degree of node o, w o is the node's discount factor which is drawn from a uniform distribution w o~U (0:8,0:9), g is the number of repeated games played (fixed at 50), and P oi/ni is the payoff received by player o versus its neighbor n in game i.
For the logic behind player's behaviors or their strategy we apply the (y, p, q) cooperation component framework from Ref. 30. Past studies usually simplify this framework by assuming that the first component y is fixed at y 5 1 (i.e., always cooperate on the first move), either with the rationale that a single move becomes less important in an infinitely repeated game 31 or that a two-strategy framework lends itself well to a two dimensional representation 59 . Since our purpose is to study how structure impacts trust as well as reciprocity and forgiveness we do not assume y is fixed and so begin each simulation with a completely random (y, p, q) strategy variant for each network node. This leads both to unique insights and to increased generality of our results since there is no starting bias, whereas the assumption of y 5 1 is a certainly a cooperative one. Moreover, our strategy space is highly stochastic since the initial y, p, q values are each drawn from a uniform distribution for every node, resulting in millions of starting strategies across the simulation runs performed for this work. This also effectively eliminates distribution effects between cooperators and defectors, described by Ref. 53.
The evolutionary process is as follows. We first initialize each node in the network to an initial strategy s o which is defined by the triple (y o , p o , q o ) each component of which is drawn from a uniform distribution U(0,1). From then on, for each time step, the population is evolved through a pairwise comparison rule where the game score of each node in the network P o is sequentially compared with a randomly chosen neighbor P n ; strategies are then updated with the probability W(s n R s o ) defined in equation (5), which was introduced by Ref. 24.
The k max variable in equation (5) is the maximum degree of nodes o and n, and D 5 T 2 S for the PD game. Equation (5) has the nice property of evolving both high and low degree nodes at the same rate, which the commonly-used Fermi function lacks 6 . This is an important characteristic for this work because of the variety of network structures we test; however, equation (5) gives up the ability to adjust the evolutionary selection strength b, present in the Fermi function but we keep evolutionary strength constant for this work and so are unaffected by the inability to control it.
Using equation (5), the evolutionary process continues for multiple generations until no more than one node updates its strategy within a sequential time window of 100 generations. To aide in convergence we also provide a strategy adoption fixed noise threshold E~0:01, below which nodes will not update their strategy.
Finally, nodes that switch strategies adopt the average of their present strategy and the chosen new strategy. This method greatly enhances the simulation convergence time (.103 faster over a simple copy process) by reducing the cycling of strategies through the network while also exploring a greater variety of strategies.
Derivation of modularity relation. To help understand the effect of modularity on agents as game stakes increase we have derived the relation equation (1) which shows that the more temptation to defect (b), the more modular (a) the component needs to be in order for cooperation to survive and propagate, which aligns qualitatively with the modularity gradient in figure 1 panel (C). This section details the derivation of this relation.
Our goal is to demonstrate that modularity increases the chance of survival and expansion of cooperative behavior in a network. We use a Prisoner's Dilemma framework with R 5 1 and T 5 b . 1 and assume there are only two deterministic payoff structures: Cooperation (S i 5 1) and Defection (S i 5 0). The agents learn from their more successful agents with a probability proportional to the difference in their aggregate payoffs. We test the structural conditions under which a group of cooperators cannot be invaded by defectors but a group defectors have a finite probability of becoming cooperators.
For the first goal, we pick any connected component of cooperators within the network and show that this connected component is a module. For each node in this component, either all the neighbors are already cooperators, meaning that the node is fully embedded in the component, or the node has a mixed set of neighbors, in which case we call it a boundary node. Since the boundary node is a member of the connected component of cooperators, it cannot have any cooperative neighbor that is outside of the component. If the degree of the boundary node i is k i , the neighborhood can be divided into a set of cooperating neighbors k in and defecting neighbors k out .
In order for the component to stay cooperative, every boundary node i needs to have a larger aggregate payoff than any of its defecting neighbors (node j). The payoff for node i is simply equal to k in and the payoff for node j is bk' j . We need to have k in wbk' j . The variable k' j , the number of cooperating agents of node j, can take any number between 1 (since i already is a cooperator) to k j , the degree of node j; this condition must hold especially for the k' j~kj since it is the worst case for cooperator survival. Thus we have: Now consider the same component and assume that all the members are defectors. We want to identify the structural condition under which boundary nodes, of this defector component, have a finite chance of becoming cooperators. These boundary nodes exclusively derive their payoffs from cooperators and all of them lie outside of the defector component. This score is bk out and needs to be smaller than the score of a cooperative neighbor. The latter is simply k0 j . So for node i we have: We also have 0 # k0 j # k j 2 1. Thus we have bk out , k j 2 1. Combining this with equation (6) gives: This gives a condition for the relative value of k in and k out for node i. Assuming that bk out is much larger than 1, and defining a as the ratio of k in to k i , that is a measure of how modular the connected component is, equation (8) turns into: www.nature.com/scientificreports