# Developmental Emergence of Sparse Coding: A Dynamic Systems Approach

## Abstract

During neocortical development, network activity undergoes a dramatic transition from largely synchronized, so-called cluster activity, to a relatively sparse pattern around the time of eye-opening in rodents. Biophysical mechanisms underlying this sparsification phenomenon remain poorly understood. Here, we present a dynamic systems modeling study of a developing neural network that provides the first mechanistic insights into sparsification. We find that the rest state of immature networks is strongly affected by the dynamics of a transient, unstable state hidden in their firing activities, allowing these networks to either be silent or generate large cluster activity. We address how, and which, specific developmental changes in neuronal and synaptic parameters drive sparsification. We also reveal how these changes refine the information processing capabilities of an in vivo developing network, mainly by showing a developmental reduction in the instability of network’s firing activity, an effective availability of inhibition-stabilized states, and an emergence of spontaneous attractors and state transition mechanisms. Furthermore, we demonstrate the key role of GABAergic transmission and depressing glutamatergic synapses in governing the spatiotemporal evolution of cluster activity. These results, by providing a strong link between experimental observations and model behavior, suggest how adult sparse coding networks may emerge developmentally.

## Introduction

The proper development of neural networks is strongly activity-dependent1. A common feature of the immature cortex is the generation of synchronized network activity in which discrete events are separated by relatively long quiescent periods2,3. During development, this so-called cluster activity transitions to a sparse firing pattern as typically observed in adult networks4. Sparse firing is supposed to represent an efficient coding regime for processing and storing information in the mature cortex during adulthood5,6. The developmental transition from dense to sparse coding has been observed in all cortical areas examined to date2,3,7 and, in addition, in human cortex8. Moreover, since, unlike adult networks, eliciting large cluster activity is a ubiquitous feature of many immature neural structures1,9, sparsification is believed to be a universal phenomenon of neural network maturation. Although sparsification is thought to represent an essential aspect of cortical development, the mechanisms underlying this process are currently not understood. Data obtained from rodents led to the conclusion that the process of sparsification is largely, though not entirely3, independent of sensory inputs2,7. Strikingly, the time period of development during which sparsification occurs coincides with major changes in both intrinsic neuronal and synaptic properties. These include, for example, a profound decline in membrane resistance2,10, a steep increase in the number/density of both GABAergic and glutamatergic synapses11,12, an acceleration of the kinetics of postsynaptic currents12 as well as pronounced changes in short-term synaptic plasticity10,12.

Here, we describe a computational modelling approach based on experimentally measured trajectories of intrinsic neuronal and synaptic parameters in order to provide mechanistic insights into the generation of cluster activity and the transition from dense to sparse coding during development. To this end, we use the well-established extended Wilson-Cowan-type model accounting for mechanisms of short-term synaptic plasticity13. Despite the lack of well-established immature neuron models, as still many unspecified parameters need to be measured first, using this biophysically interpretable mean-field model enables the study of developing networks based on the average effects of these parameters at a network level. By combining this model with experimentally reported trajectories of neurobiologically plausible parameters during postnatal development, we found that we can emulate typically observed in vivo features of developing network function.

By using the mathematical tool of stability analysis, we derive three key results which help to better understand the sparsification process. Firstly, in the model we find that at early postnatal days, while the network is resting at its low-activity spontaneous state, an unstable state is built up in its firing activities. This is important, because we find that this transient, hidden unstable state is key to the emergence of large postnatal cluster activity. Using the model, we show how sparsification is driven developmentally, and how during this process the network’s information processing is refined. Secondly, we address the open question what mechanisms govern the spatiotemporal evolution of postnatal cluster activity. Thirdly, we quantify the effect of different maturational processes on sparsification. Together, this study provides the first mechanistic insights into the in vivo biophysical mechanisms underlying sparsification, and the implications of this process for the refinement of information processing.

## Results

To study developmental changes in postnatal neuronal activity, we used simulations of a spatially localized recurrent neural network with short-term synaptic plasticity (STP-RNN)13, where STP renders synaptic efficacies dynamic over time14,15; see Methods. This network is a mean-field Wilson-Cowan-type model of one excitatory (E) and one inhibitory (I) neuronal population with recurrent dynamic synaptic connections (Fig. 1a). This model has the advantage of being biophysically interpretable and mathematically accessible. The activity rates (E r and I r) can be properly scaled to represent locally the average recorded activities in the populations. Here, we use experimentally reported, postnatal changes in neurobiologically plausible parameters (Table 1) to assess how the network’s spontaneous behavior and sensory processing properties are refined toward the postnatal onset of sensory transduction. To study the developmental states of visual cortical networks from before to after eye-opening, we selected four postnatal days (P) for modelling: P3 (period of physiological blindness), P10 (a few days before eye-opening), P14 (the day after eye-opening) and P20 (a few days after eye-opening)3. During this period, immature networks not only undergo the sparsification process but also a dramatic development in intrinsic neuronal and STP properties10,12; see Table 1 and Supplementary Methods. To derive the results, we used system dynamics methods described in the Methods section (see also Supplementary Methods). For quick reference, all important technical terms are explained in Table 2.

### Transient unstable state hidden in firing activities

We first address what mechanisms may underlie the emergence of cluster activity in immature networks, by analyzing model behavior at P3 as a representative early postnatal stage.

Previous experimental16,17 and modeling studies18,19 usually considered adult networks to be bi- or multi-stable where, e.g., spontaneous (stimulus-absent) network activity can transition between two stable activity states: A relatively quiet activity state, and a higher activity state. For early postnatal stages, our stability analysis and simulations indicate that the proposed developing network model is actually mono-stable whose only stable state is a quiescent, spontaneous state (Fig. 1b). To demonstrate this, we plotted in Fig. 1b the so-called phase plane of network dynamics in terms of the average activity in the E- and I-populations (E r-I r -plane; Table 2). Figure 1b shows that the network’s fixed point (FP; Table 2) is located at E r = I r= 0 which is essentially a stable activity state of the network (see Supplementary Methods). In the following, we will call this specific FP the rest state of the network. This analysis result fits well with experimental findings of near-zero hertz spontaneous activity of immature populations3,20, and the absence of spontaneous network’s persistent activity states mainly during periods of physiological blindness7,21.

How can a quiescent, immature, mono-stable network generate large cluster activity? By analyzing network perturbations, we found strikingly that while a P3 network model was relaxed at its stable rest state, threshold-crossing perturbations of E-population were amplified profoundly (Fig. 1c, Supplementary Fig. 1c). This amplification resulted in population spikes (PSs, Table 2), which, in the model, we assume is the analogue to experimentally observed cluster activity. Note that this perturbation can be due to, e.g., a single-shock electrical stimulation22, the onset of a longer-lasting external input (Supplementary Fig. 2c), or a random input driven by spontaneous retinal waves or the thalamus3,20.

While previous studies showed the underlying mechanism of PSs generation mainly in adult bi-stable network models18,19, here we sought to address this mechanism in an immature mono-stable network (Fig. 2). As the initial phase of PSs is known to be governed by the network’s fast (i.e. firing activity) dynamics18, we re-plotted the E r-I r-plane after freezing the network’s slow (i.e. STP) dynamics at the rest state (Fig. 2d); thus, it is turned to a Static-RNN (Table 2) with frozen synaptic efficacies (Frozen STP-RNN). Surprisingly, we found that in addition to the stable rest state, there is an unstable FP hidden in the network’s fast dynamics (i.e. in the respective Frozen STP-RNN, Fig. 2e), which does not exist in the (non-frozen) STP-RNN (Fig. 1b). This unstable FP (black dot, Fig. 2e), which is located close to the rest state, can confine the attraction domain (grey region, Table 2) of the stable rest state in the E r-I r-plane of the corresponding Frozen STP-RNN (Fig. 2d); but, clearly, not in the E r-I r-plane of the STP-RNN itself (Fig. 2c). We call the outer border of this domain in Fig. 2d (before perturbation) the amplification-threshold. Almost exactly the same border exists for activity-perturbation domains of the STP-RNN (Fig. 2c). Cluster activity is initiated by an activity perturbation exceeding this threshold, where the activity will be moved far from the rest state in both corresponding networks with frozen (Frozen STP-RNN) or dynamic (STP-RNN) synapses. Strikingly, when freezing synaptic efficacies at different times during a cluster activity (Fig. 2a), we can see that this unstable FP disappears roughly after the peak of cluster activity (Fig. 2e), thereby allowing network activity to converge back to its stable rest state in the STP-RNN (Fig. 2a). This disappearance is mainly because of the weakening of the excitatory synapses at high activity rates (Fig. 2b); these synapses provide positive feedback to the network. Importantly, as it can be seen in Fig. 2e (panel IV), while the network is again at the origin after cluster activity, sufficient recovery time is required for the re-emergence of the unstable FP.

Our analyses further show that the experimentally observed variability in size (i.e. the number of network neurons recruited by cluster activity) and duration of spontaneous cluster activities at each postnatal day2,3 is effectively governed by the relatively long recovery time of immature excitatory synapses (Supplementary Fig. 1). In addition, in our model, cluster activity was generated only if the perturbation strength was sufficiently large (Supplementary Fig. 1c and d) and was able to push network activity beyond the amplification-threshold (Fig. 2c and d). This is consistent with experimental reports of immature networks7,23, and may underlie a relatively all-or-none characteristic of postnatal cluster activity.

### The route towards sensory processing

The sparsification of activity patterns sets in around eye-opening3,7. We now use our model at the stages P10 to P20 to address the question what refinements of the developing network enable sparsification.

Firstly, similarly as for P3, at all later stages up to P20 in the model we found that for a developing network relaxed at its stable rest state (Fig. 3a), again an unstable FP is hidden in the network’s fast (i.e. firing activity) dynamics (Fig. 3b). Consequently, a threshold-crossing perturbation of the E-population at rest state led to cluster activity (Fig. 3c). In addition, our analysis revealed that the amplification-threshold tends to move toward higher E-activity rates, from P3 to P20 (Fig. 3b, Supplementary Fig. 3). This result suggests that a stronger input may be required to trigger (large) cluster activities at late development stages.

Experimental in vivo findings show that the cluster activity size starts to drop dramatically around the end of the second postnatal week2,3. Here, we consider this developmental decrease in cluster activity size as an indicator for the developmental transitioning from dense to sparse coding (sparsification process); similarly to experimental studies2,3. In our model, we approximate the cluster activity size by a measure of the average activity of both E- and I-populations ($$P{S}_{{\rm{net}}}^{{\rm{amp}}}$$; see Table 2 and Methods). Using this measure, we observed, similarly as in experiments, that cluster activity size undergoes developmental reduction between P10 to P20 (Fig. 3c and d, see also Supplementary Fig. 2c): It was relatively large at P10 ($$P{S}_{{\rm{net}}}^{{\rm{amp}}}\approx 85$$), but starkly reduced after eye-opening (P14; $$P{S}_{{\rm{net}}}^{{\rm{amp}}}\approx 30$$), followed by some further reduction to P20 ($$P{S}_{{\rm{net}}}^{{\rm{amp}}}\approx 15$$).

Secondly, the results also show that immature networks probably lack any spontaneous, non-quiescent stable FP (i.e. a spontaneous attractor or persistent activity state) up to a few days prior to eye-opening. In Fig. 3a this is indicated by the existence of only one green dot at P3 and P10. However, after eye-opening, new spontaneous FPs with higher activity rates than the rest state emerged (note the non-origin FPs at P14 and P20 in Fig. 3a). This is compatible with experimental data showing the existence of spontaneous persistent cortical activity for stages mostly after the second postnatal week3,21,24 and their absence during early development7,21. In our model, in the absence of a stimulus, these FPs emerge mainly because of the developmental increase and decrease in background activity2,3,7 and membrane resistance2,10, respectively. Their combined effects determine the increase in population activity thresholds θ E and θ I (see also Supplementary Fig. 4). While in the model the network’s FP domains (FP-domains; Table 2) are not affected by these thresholds (see “Characterization of operating regimes” section in Supplementary Methods and also Supplementary equation (17)), the developmental reduction in the network instability (yellow regions, Fig. 3a) allows for these FPs to be stable. This means that attractors start emerging, under a wide range of E-activity; note the non-origin green dots at P14 and P20 in Fig. 3a.

Thirdly, the emergence of spontaneous attractors not only turns the developing mono-stable network (at P3 and P10) into a bi-stable (at P14 and P20) network (Fig. 3a), but also has a striking effect on the dynamical trajectory of cluster activity. That is, in the model, we found a fundamental difference between spontaneous postnatal cluster activities and those in more mature networks (Fig. 3c): before eye-opening (P3 and P10), the cluster activity is of the so-called mono-stable type (see Supplementary Methods), while after eye-opening the cluster activity can be also of the bi-stable type. The difference is that a mono-stable cluster activity (P3 and P10 in Fig. 3c) is initiated and terminated at the stable rest state, while a so-called bi-stable cluster activity is initiated at the stable rest state but converges to a spontaneous attractor (P14 and P20 in Fig. 3c, green dots). This difference may be interpreted as a first expression of information processing where the newly emerged attractors, which a bi-stable cluster activity converges to, can be seen as representative states that are the basis of perception. In the model, the time required for an effective transition to the attractor, and thus an effective representation of perceptual stimuli, is probably equal to the duration of cluster activity (see dashed grey lines, Fig. 3c): ~240 ms at P14 and ~180 ms at P20; these transition times are consistent with experimental observations at these stages17,24.

Moreover, our simulations showed that the spatiotemporal characteristics (i.e. size and duration) of mono-stable cluster activities are in general considerably more robust to interfering perturbations (P3 and P10, Supplementary Fig. 5), as compared to the bi-stable ones (P14 and P20, Supplementary Fig. 5). This suggests that the trajectories underlying mono-stable cluster activities (P3 and P10) can be seen as signals with high signal-to-noise ratio, which may act as a reliable neuronal communication mechanism in neonatal networks. The decreased robustness of bi-stable cluster activities after eye-opening (P14 and P20) can in principle augment the network’s information processing capability by making it more flexible in responding to sensory input.

In addition, we found that at the new spontaneous attractors (non-origin green dots; Fig. 3a) the network operates as a so-called inhibition-stabilized network (ISN, light-yellow regions, see also Table 2)25,26. An ISN regime is thought to allow the cortex to process complex computations27, and some experimental evidence indicates that adult cortical networks operate as ISNs25,28. Our results show that the ISN regime becomes effectively accessible after eye-opening, as the unstable FP-domain, which is confined to low E-activity ranges, is developmentally substituted by an ISN FP-domain. This indicates a developmental increase in the ratio of areas of the ISN (AOD ISN) and unstable FP-domains (AOD Unstable), which we quantified by AOD ISN/Unstable (Table 2) in Fig. 3e. In practice, this means that in parallel to the strengthening of the sensory inputs after eye-opening, a larger set of potential stimulus-evoked ISN attractors will be accessible for the network, presumably to perform complex sensory computations.

Overall in our model, the process of sparsification is translated not only to a potent reduction of postnatal network instability but also to a potential emergence of new attractors and information processing capabilities.

### Spatiotemporal evolution of postnatal cluster activity

What mechanisms govern the spatiotemporal evolution (i.e., size and duration) of cluster activity during development in vivo? We addressed this question by analyzing how a postnatal cluster activity, once on a trajectory away from the rest state, is brought back to either the rest state or another FP with an activity level higher than the rest state.

In our model, there are two important factors: (i) the inhibitory transmission, which is believed to play a critical role in the stabilization of adult networks25,28, and (ii) STP, which can dynamically control the gain of neuronal responses14 and acts strongly depressing in immature networks10,12.

Firstly, we found that cluster activity initiated at the rest state can still converge back to this state even when blocking GABAergic receptors, at all stages from P3 to P20 (Fig. 4a). This finding suggests that inhibitory processing may not be necessary for stability in the developing network with depressing excitatory synapses. However, blockage of GABAergic receptors yielded an increase in cluster activity size (Fig. 4a and c), consistent with in vivo reports20,29. This blockage effect on cluster activity size became more pronounced during the course of development (Fig. 4c), in spite of the developmental reduction in the network instability (e.g. see AOD ISN/Unstable in Fig. 3e). This finding may underlie a developmental enhancement of an effective contribution of inhibitory transmission to network activity (see also Supplementary Fig. 2d showing the ability of inhibitory inputs to turn off activity at the attractor22, after eye-opening). In addition, we found that the blockage of GABAergic receptors shortened the cluster activity duration, for example, from approximately 330 to 320 ms at P3 (Fig. 4a and b) and from approximately 265 to 210 ms at P10 (Fig. 4a). The range of these cluster activity durations in our model is compatible with previous experimental reports7,30. Moreover, we found that this blockage effect on cluster activity duration increased from P3 to P20 (Fig. 4d).

Secondly, in contrast to GABAergic transmission, we found that at P3 the simulated removal of the STP effect led to run-away excitation (Fig. 4e). After freezing the synaptic efficacies (see Supplementary Methods) at the stable rest state, a threshold-crossing perturbation of the E-population (similarly as in Fig. 1c) caused a seizure-like surge of network activity (Fig. 4e). Importantly, this means that the rest state in the corresponding Frozen STP-RNN is a locally, but not globally, stable FP (see Methods). Similar results were observed for P10 to P20 (data not shown). Accordingly, we conclude that the strongly depressing immature STP, but not inhibitory synaptic transmission, guarantees the stabilization of postnatal cluster activities during the course of development. This finding also holds, when GABAergic transmission is considered as excitatory at the network level during the first postnatal week (see Supplementary Methods and Supplementary Fig. 6b). Under this assumption, blocking GABAergic receptors led to a decrease in cluster activity size, which is not compatible with in vivo data20,29. Therefore, our modeling results (Fig. 4b) are in agreement with recent in vivo findings that while GABA acts as a mainly depolarizing transmitter during the first postnatal week, it exerts an inhibitory effect on the underlying network activity in the immature cortex and hippocampus20,31.

Moreover, cluster activity was abolished by simulated blockage of glutamatergic receptors (data not shown), consistent with experimental in vivo data20,29. This is because the main source of the network instability in our model is determined by glutamatergic synapses which provide positive feedback to the network. Therefore, blocking glutamatergic receptors removed all unstable FPs as well as the attractors in Fig. 3a and forced spontaneous network activity to the rest state (as observed experimentally17,22), and removed the unstable FP hidden in the network’s fast dynamics in Fig. 3b, throughout P3 to P20 (data not shown). In addition, note that the GABAergic transmission can play an important modulatory effect on this instability (see Supplementary Methods), where an increase in the efficacies of inhibitory synapses can effectively attenuate the instability (not shown). This implies that relatively weak GABAergic inhibition (Table 1) is permissive for the generation of the unstable states (e.g. see Fig. 3a and b) in developing networks; note that in our model these synapses strengthen during development (Table 1).

In sum, we found that strongly depressing excitatory synapses have a key role in the termination of postnatal cluster activities, whereas GABAergic transmission mainly regulates their spatiotemporal evolution.

### Key maturational processes mediating sparsification

The most influential, maturational processes mediating sparsification are still not clearly understood2,3. By using our modelling approach, we mechanistically quantified the impact of different network parameters on the emergence of sparse coding.

To do this, we replaced single parameters or parameter combinations of the P10 network model by their respective values at P20 and measured to what extent this modification can account for the normal decrease in $$P{S}_{{\rm{net}}}^{{\rm{amp}}}$$ and normal increase in AOD ISN/Unstable, when transitioning from P10 to P20. To quantify effects we computed $$rati{o}_{PS}$$ and $$rati{o}_{AOD}$$ as the ratios of the modification-induced changes in $$P{S}_{{\rm{net}}}^{{\rm{amp}}}$$ and AOD ISN/Unstable, relative to their respective normal developmental changes during this period (Fig. 5). We found that only three parameters caused a substantial decrease in $$P{S}_{{\rm{net}}}^{{\rm{amp}}}$$ (Fig. 5a): the two absolute synaptic efficacies ($${J}_{{\rm{E}}}$$ and $${J}_{{\rm{I}}}$$) caused the largest decrease close to that in the normal transition and, to a minor degree, the excitatory release probability $${U}_{{\rm{E}}}$$. Strikingly, for the excitatory synaptic time constant ($${\tau }_{{\rm{E}}}$$) the modification led to an increase in $$P{S}_{{\rm{net}}}^{{\rm{amp}}}$$, i.e. when this parameter is changed on its own, it tended to reverse sparsification (see also Supplementary Figs 6c and 7). We obtained qualitatively similar conclusions when analyzing the effect of replacing single parameters or parameter combinations of the P20 network model by their values at P10 (Supplementary Fig. 8a). When testing for changes in AOD ISN/Unstable, we found that the inhibitory synaptic depression time constant $${\tau }_{{{\rm{r}}}_{{\rm{I}}}}$$ had the strongest contribution to the normal transition from P10 to P20, whereas all other single parameter substitutions had relatively weak effects (Fig. 5b). Note that the change in $${\tau }_{{{\rm{r}}}_{{\rm{I}}}}$$ appears to be a necessary but not a sufficient condition for the normal transition from P10 to P20, as substituting $${J}_{{\rm{E}}}$$ and $${J}_{{\rm{I}}}$$ in the P20 model by their respective P10 values virtually abolished the developmental change in AOD ISN/Unstable (Supplementary Fig. 8b). Figure 5 further shows that the modification effect of the population activity thresholds (θ E and θ I) was virtually negligible for $$rati{o}_{PS}$$ and zero for $$rati{o}_{AOD}$$. However, as shown above, their maturation plays an important role for the emergence of spontaneous attractors (see also Supplementary Fig. 4).

## Discussion

We modelled the in vivo activity of a developing cortical network during the first postnatal month by combining an extended Wilson-Cowan model with experimentally reported trajectories of neuronal and synaptic parameters. We revealed mechanistically that a particular combination of a transient, hidden unstable state in firing activities and strong synaptic depression enables an immature network to generate large cluster activity while otherwise being mostly silent. We further found that the normal developmental transition from dense to sparse coding is strongly dependent on an elaborate, parallel refinement of absolute synaptic efficacies, both short-term synaptic plasticity (STP) and intrinsic membrane properties, and background activity. Strikingly, in our model, sparsification translates not only to a reduction of postnatal instability of network activity but also to an effective availability of the inhibition-stabilized network (ISN) regime as well as the emergence of spontaneous attractors, providing a novel mechanistic explanation for how the network’s information processing is refined towards eye-opening.

How can immature networks be quiescent for relatively long periods and occasionally generate large cluster activity3,32? Surprisingly, we found that while the developing networks (P3 to P20) are operating at their rest state (Fig. 3a), an unstable state is formed in their fast (i.e. firing activity) dynamics (Figs 2e and 3b). This may be a key to this biphasic behavior (Fig. 3c and Supplementary Fig. 2c). In addition, we found that at early stages prior to eye-opening (P3 and P10) immature networks are mono-stable, where the only FP of the network is its stable rest state (Fig. 3a). Accordingly, the underlying mechanism of cluster activity emergence in these networks is in stark contrast to models of cluster activity proposed for adult networks, based on the usual assumption of a bi-stable (or multi-stable) network18,19. Our finding about the strong effect of the hidden unstable state (Fig. 3b) during the initial phase of network activity at the rest state (see Figs 2 and 3c) may provide an explanation why neonatal networks, e.g. at P3 and P10, are more susceptible to seizures than mature networks32,33. Moreover, the lack of any spontaneous attractor in these networks (Fig. 3a) might also contribute to this susceptibility, since the attractors can, in principle, aid in stabilizing a network’s activity.

Sparsification coincides with the peak of GABAergic and glutamatergic synaptogenesis11,12 and overlaps with a developmental reduction in release probabilities, particularly at glutamatergic synapses10,12. Using a computational study, we here revealed how these changes are suited to drive the transition from dense to sparse coding during network maturation (e.g. see Figs 3 and 5). Since sensory deprivation affects the total synapse numbers only modestly34, our results might also explain previous experimental findings: While sensory experience may have a modulatory effect on sparsification during the first days after eye-opening3, this process is largely mediated internally, i.e. independent of sensory inputs2,3,7.

How does the sparsification process prepare in vivo developing networks for effective sensory processing? While the answer to this question remains poorly understood by experimental studies, our computational study provides two mechanistic insights: Firstly, we found that Non-ISN and instability may play key roles as the dominating operating regimes prior to eye-opening (Fig. 3a), possibly reflecting an immaturity of inhibitory transmission. That is, our modeling results imply that during stimulation (e.g. in response to long-lasting external inputs) the immature networks will operate under a Non-ISN regime, rather than an ISN regime (P3 and P10; Supplementary Fig. 2b), due to the lack of an effective availability of the ISN regime during early development (P3 and P10; Fig. 3a). The Non-ISN regime will enable the immature networks to maintain their stability without requiring inhibitory transmission effects (see Methods). For adult networks, however, some experimental evidence indicates that cortical networks operate as ISNs25,28. Strikingly, in the model, during sparsification, the unstable FP-domain is effectively replaced by that of an ISN regime (Fig. 3a and e). This may enable cortical networks to process complex sensory computations27 in parallel to the developmental strengthening of sensory inputs.

Secondly, we found that spontaneous attractors start emerging around eye-opening, which, in our model, is mainly due to the combined effect of developmental increase in background activity2,3,7 and the developmental decrease in membrane resistance2,10. In general, attractors can represent the solution to a specific neural computation35. They are also thought to be the substrate of, and used to model, e.g., the working memory36, eye position stability37, and orientation selectivity38. Besides, in our developing model, the attractor emergence phenomenon can also underlie three developmental mechanisms. First, it provides the basis for transitioning between FPs (here, via bi-stable cluster activity at P14 and P20; Fig. 3c) and may be thought of as an early step towards representation of perceptual stimuli. Second, it enables inhibitory transmission to contribute effectively more to network activity, e.g. through providing a sufficiently strong balancing of the intrinsic instability of the E-activity under an ISN regime26,27 (P14 and P20; Fig. 3c), or by becoming amenable to terminate activity at the attractor22 (Supplementary Fig. 2d). This may be important for a presumably more effective processing of sensory inputs. Third, this phenomenon may also initiate the effective interaction of spontaneous activity with sensory cortical responses, as observed experimentally17.

Evidence from in vitro studies suggests that GABA depolarizes neonatal neurons, e.g. in rodents during the first postnatal week20,32. However, the effect of GABA at the network level is still an open question. While most of the previous in vitro studies reported that GABA is excitatory (or both excitatory and inhibitory) at the network level32, recent in vivo studies harnessing new experimental techniques found that GABA inhibits intact neonatal networks in neocortex and hippocampus20,29,31. In particular, GABAergic transmission was shown to limit the spatial extent of cluster activity20. In our model, we tested these two hypotheses by blocking the GABAergic receptors in two P3 networks with either excitatory (Supplementary Fig. 6b) or inhibitory GABAergic transmission (Fig. 4b). This simulated manipulation led to an increase in the cluster activity size only when GABAergic transmission was inhibitory at the network level. Therefore, our results support the recent in vivo findings31,33. Importantly, this finding implies that even when GABA depolarizes immature neurons, GABA still can attenuate the instability effect on network activity, thereby restricting cluster activity size.

To our knowledge, there are only few previous modelling studies covering this developmental period (e.g. see refs39,40), where the authors’ main focus was on the potential mechanism of the cluster activity generation under in vitro condition. These studies were focused on a single postnatal stage and this stage was mostly restricted to the first postnatal week, like P5. Moreover, these authors probably did not consider some important biophysical considerations which have only recently been revealed by in vivo studies: the abolishment of cluster activity following the removal of glutamatergic synapses20,29, the existence of a profound inhibitory effect of GABAergic transmission at the network level20,29 and its non-excitatory (but depolarizing) effect at the neuron level20,31 during the first postnatal week, or that the loss of large cluster activity occurs around eye-opening2,3 and not around P741. In contrast, here, (i) we used a unified computational model which takes into account most of these recently reported biophysical considerations (see Supplementary Method and Table 1), (ii) our work shows results based on, and mainly in accordance to the recent in vivo studies, where these studies revealed some critical features different from those reported for in vitro data, (iii) we studied the whole developmental period, comprising the time course of sparsification, and not only a single postnatal stage, (iv) we not only address the possible underlying mechanism of the postnatal cluster activity generation, but also the developmental changes in the operating regimes and information processing capabilities, as well as the possible refinements in network properties governing the sparsification.

There are advantages and limitations of our modeling approach. Clearly, as compared to a mean-field model as employed here, a spiking network model can in principle provide more detailed results, e.g. one could model the temporal sparseness of spiking activities, or the specific patterns of sensory inputs at the neuron level. However, for this early developmental period, we are not aware of any established neuron model. For such a model, a considerable amount of unspecified neurobiological parameters is required to be measured experimentally first. Instead, using a mean-field model enabled us to forgo such detailed parameterizations by considering only their average characteristics over the network. Although this procedure comes at the price of removing biological details, this enabled us to use an extended Wilson-Cowan-type model13; a well-established model and extensively used before to study adult networks behavior18,42. This model enabled us to incorporate the available, experimentally reported developmental trajectories of intrinsic neuronal and synaptic parameters (see Supplementary Method and Table 1), including short-term synaptic plasticity. In addition to its biophysical interpretability, this model is in general mathematically tractable which allowed us to derive analytical expressions, e.g. in our stability analyses (see Supplementary Methods). Moreover, this model is readily extendable to incorporate other biophysical mechanisms like spike-frequency adaptation, or to build a spatial graph of multiple homogenous and heterogeneous inter-connected networks to study, e.g. the activity propagation over different brain areas.

In sum, we have shown that by establishing and extending a novel application of an existing computational model to immature networks and integrating recent experimental findings, new mechanistic insights into the development of neural networks can be obtained. We expect that, in the future, this modelling approach can also guide research by providing for concrete predictions that can be tested experimentally.

## Methods

Here, we briefly describe the main components of our model and the analyses. A detailed description can be found in Supplementary Methods.

### STP-RNN model

This model (Fig. 1a) is an extended version of Wilson-Cowan’s recurrent neural network (RNN) mean-field model43, for which the short-term plasticity (STP) of synaptic connections were also modeled13. The equations governing the model dynamics over time are (dots denote the time derivatives):

$$\begin{array}{rcl}{\tau }_{{\rm{E}}}{\dot{E}}_{{\rm{r}}}(t) & = & -{E}_{{\rm{r}}}(t)+{G}_{{\rm{E}}}({J}_{{\rm{EE}}}\,{u}_{{\rm{EE}}}(t)\,{x}_{{\rm{EE}}}(t)\,{E}_{{\rm{r}}}(t)-{J}_{{\rm{EI}}}\,{u}_{{\rm{EI}}}(t)\,{x}_{{\rm{EI}}}(t)\,{I}_{{\rm{r}}}(t)+{e}_{{\rm{E}}}(t)-{\theta }_{{\rm{E}}})\\ {\tau }_{{\rm{I}}}{\dot{I}}_{{\rm{r}}}(t) & = & -{I}_{{\rm{r}}}(t)+{G}_{{\rm{I}}}({J}_{{\rm{IE}}}\,{u}_{{\rm{IE}}}(t)\,{x}_{{\rm{IE}}}(t)\,{E}_{{\rm{r}}}(t)-{J}_{{\rm{II}}}\,{u}_{{\rm{II}}}(t)\,{x}_{{\rm{II}}}(t)\,{I}_{{\rm{r}}}(t)+{e}_{{\rm{I}}}(t)-{\theta }_{{\rm{I}}})\\ {\dot{x}}_{{\rm{ij}}} & = & {\tau }_{{\rm{r}}{}_{{\rm{ij}}}}^{-1}(1-{x}_{{\rm{ij}}}(t))-{u}_{{\rm{ij}}}(t)\,{x}_{{\rm{ij}}}(t){A}_{{\rm{j}}}(t)\\ {\dot{u}}_{{\rm{ij}}} & = & {\tau }_{{{\rm{f}}}_{{\rm{ij}}}}^{-1}({U}_{{\rm{ij}}}-{u}_{{\rm{ij}}}(t))+{U}_{{\rm{ij}}}(1-{u}_{{\rm{ij}}}(t)){A}_{{\rm{j}}}(t)\end{array}$$

where j and $${\rm{i}}\in \{{\rm{E}},{\rm{I}}\}$$, $${A}_{{\rm{j}}}\in \{{E}_{{\rm{r}}}\,,\,{I}_{{\rm{r}}}\}$$, j is the index of presynaptic population, $${E}_{{\rm{r}}}$$ (=A E) and $${I}_{{\rm{r}}}$$ (=A I) are the average activity (in hertz) of E- and I-populations which receive, respectively, the external inputs $${e}_{{\rm{E}}}$$ and $${e}_{{\rm{I}}}$$, e.g., from other brain regions, $${G}_{i}$$ is the linear input-output gain above population activity threshold $${\theta }_{i}$$ (otherwise $${G}_{i}=0$$), and $${x}_{{\rm{ij}}}$$ and $${u}_{{\rm{ij}}}$$ are the dynamic variables of short-term synaptic depression and facilitation mechanisms. The parameter definitions and values are listed in Table 1. A STP-RNN with synaptic efficacies frozen at the FP is called Frozen STP-RNN (a Static-RNN-type model).

### Phase plane components

E r-I r-plane is the 2D phase plane of E- and I-population activity rates ($${I}_{{\rm{r}}}$$ vs. $${E}_{{\rm{r}}}$$), in which the $${E}_{{\rm{r}}}$$- and $${I}_{{\rm{r}}}$$- nullclines are represented as the set of points for which $$d{E}_{{\rm{r}}}/dt=0$$ and $$d{I}_{{\rm{r}}}/dt=0$$, respectively. For a STP-RNN, the depicted quasi $${E}_{{\rm{r}}}$$ and $${I}_{{\rm{r}}}$$ nullclines in E r-I r-plane are based on the reduced STP-RNN (Supplementary equation (7)). Any intersection of these two quasi nullclines (i.e. the FP) can represent the steady state of the whole network (i.e. the 10D STP-RNN).

### Stability of FPs

To determine the stability of any FP of interest we applied linear stability analysis to our 10D STP-RNNs (or 2D Frozen STP-RNNs): We investigated whether all eigenvalues of the network’s system of equations linearized around the FP (using the Jacobian matrix) have strictly negative real parts (if so, the FP is stable), or whether at least one eigenvalue with positive real part exists (if so, the FP is unstable). This can be re-stated as: A FP is stable if following a small perturbation from that FP (at which the network was before perturbation), the network dynamics converge back to it. Conversely, if the dynamics move away from the FP or dies out, that FP is unstable.

### Operating regimes and FP-domains

The stable operating regimes of a RNN at a FP can be classified as an inhibition-stabilized network (ISN) vs. a Non-ISN25,26. To discriminate between these two regimes three criteria were defined: (A) Excitatory instability: For the inhibitory activity rate fixed at the FP, the recurrent excitation is strong enough to render the E-population intrinsically unstable. (B) Excitatory stability: In contrast to (A), the E-population is stable per se, i.e. even with a feedback inhibition fixed at its level at the FP. (C) Overall stability: The dynamic feedback inhibition to the E-population is strong enough to stabilize the whole network activity. At a FP, a network operating under the (A) and (C) criteria is an ISN, while a network operating under the (B) and (C) criteria is a Non-ISN. A network, which is neither ISN nor Non-ISN at the FP, operates under an unstable regime.

We partition the E r-I r-plane into different domains of operating regimes (FP-domains). Each FP-domain contains all potential steady states (i.e. FPs) at which the network could operate under the corresponding regime. The area of each FP-domain is computed numerically as AOD (area of domain). Note that the borders between the FP-domains (Fig. 3a) were determined by using numerical simulations, when we plotted the FP-domains of operating regimes based on their stability criteria (A-C) obtained analytically in Supplementary Methods (see “Characterization of operating regimes” section in Supplementary Methods).

### Cluster activity size and duration

In a RNN, the cluster activity (network spike; PSnet) usually involves the population spikes44,45 in both E (PSE) and I (PSI) populations. To approximate cluster activity size, i.e. the total number of neurons synchronized during a cluster activity, we calculate $$P{S}_{{\rm{net}}}^{{\rm{amp}}}=\omega \times ({A}_{\,{\rm{sum}}}^{{\rm{amp}}}-{A}_{\,{\rm{sum}}}^{{\rm{0}}})$$ (a dimensionless parameter), where A sum = E r + I r is the sum of the activity of E- and I-populations (in hertz), $${A}_{\,{\rm{sum}}}^{{\rm{amp}}}$$ and $${A}_{\,{\rm{sum}}}^{{\rm{0}}}$$ are the maximal activity during the cluster activity and the preceding activity, respectively, and $$\omega$$ is a scaling factor (in units of [Hz−1]) to convert the activity rate to an approximate number of activated neurons during the cluster activity. For simplicity, we set $$\omega =1$$, since its veridical value is not defined for the mean-field model. We measure the cluster activity duration as its termination time minus its onset time.

### Computation of $${\boldsymbol{rati}}{{\boldsymbol{o}}}_{{\boldsymbol{PS}}}$$ and $${\boldsymbol{rati}}{{\boldsymbol{o}}}_{{\boldsymbol{AOD}}}$$

To investigate the impact of specific maturational processes on sparsification (as used in Fig. 5), we first substituted single parameters or a small combination of parameters at P10 by their values at P20 (Table 1), followed by the computation of the $$rati{o}_{\gamma }=100\times \varpi \,({\gamma }_{{\rm{P10}}}^{{\rm{res}}}-{\gamma }_{{\rm{P}}10})/({\gamma }_{{\rm{P}}20}-{\gamma }_{{\rm{P}}10})$$, where for $$rati{o}_{PS}$$ we set $$\gamma =P{S}_{{\rm{net}}}^{{\rm{amp}}}$$ with $$\varpi =-1$$, and for $$rati{o}_{AOD}$$ we set γ = AOD ISN/Unstable with $$\varpi =+1$$. $${\gamma }_{{\rm{P10}}}^{{\rm{res}}}$$ is the value of γ measured after the parameter(s) value(s) substitution.

### Data availability

Custom Matlab and Mathematica codes for our model are available upon request from the corresponding author.

## References

1. 1.

Khazipov, R. & Luhmann, H. J. Early patterns of electrical activity in the developing cerebral cortex of humans and rodents. Trends Neurosci. 29, 414–418 (2006).

2. 2.

Golshani, P. et al. Internally mediated developmental desynchronization of neocortical network activity. J Neurosci. 29, 10890–10899 (2009).

3. 3.

Rochefort, N. L. et al. Sparsification of neuronal activity in the visual cortex at eye-opening. Proc Natl Acad Sci USA. 106, 15049–15054 (2009).

4. 4.

Kerr, J. N., Greenberg, D. & Helmchen, F. Imaging input and output of neocortical networks in vivo. Proc Natl Acad Sci USA 102, 14063–14068 (2005).

5. 5.

Olshausen, B. A. & Field, D. J. Sparse coding of sensory inputs. Curr Opin Neurobiol. 14, 481–487 (2004).

6. 6.

Shadlen, M. N. & Newsome, W. T. Noise, neural codes and cortical organization. Curr Opin Neurobiol. 4, 569–579 (1994).

7. 7.

Colonnese, M. T. et al. A conserved switch in sensory processing prepares developing neocortex for vision. Neuron. 67, 480–498 (2010).

8. 8.

Vanhatalo, S. et al. Slow endogenous activity transients and developmental expression of K + -Cl- cotransporter 2 in the immature human cortex. European Journal of Neuroscience. 22, 2799–2804 (2005).

9. 9.

Luhmann, H. J. et al. Spontaneous Neuronal Activity in Developing Neocortical Networks: From Single Cells to Large-Scale Interactions. Frontiers in Neural Circuits. 10, https://doi.org/10.3389/Fneir.2016.00040 (2016).

10. 10.

Etherington, S. J. & Williams, S. R. Postnatal development of intrinsic and synaptic properties transforms signaling in the layer 5 excitatory neural network of the visual cortex. J Neurosci. 31, 9526–9537 (2011).

11. 11.

De Felipe, J., Marco, P., Fairen, A. & Jones, E. G. Inhibitory synaptogenesis in mouse somatosensory cortex. Cereb Cortex. 7, 619–634 (1997).

12. 12.

Feldmeyer, D. & Radnikow, G. Developmental alterations in the functional properties of excitatory neocortical synapses. Journal of Physiology-London. 587, 1889–1896 (2009).

13. 13.

Tsodyks, M., Pawelzik, K. & Markram, H. Neural networks with dynamic synapses. Neural Comput. 10, 821–835 (1998).

14. 14.

Abbott, L. F., Varela, J. A., Sen, K. & Nelson, S. B. Synaptic depression and cortical gain control. Science. 275, 220–224 (1997).

15. 15.

Zucker, R. S. & Regehr, W. G. Short-term synaptic plasticity. Annu Rev Physiol. 64, 355–405 (2002).

16. 16.

Haider, B., Duque, A., Hasenstaub, A. R. & McCormick, D. A. Neocortical network activity in vivo is generated through a dynamic balance of excitation and inhibition. J Neurosci. 26, 4535–4545 (2006).

17. 17.

Petersen, C. C., Hahn, T. T., Mehta, M., Grinvald, A. & Sakmann, B. Interaction of sensory responses with spontaneous depolarization in layer 2/3 barrel cortex. Proc Natl Acad Sci USA. 100, 13638–13643 (2003).

18. 18.

Barak, O. & Tsodyks, M. Persistent activity in neural networks with dynamic synapses. PLoS Comput Biol. 3, e35, https://doi.org/10.1371/journal.pcbi.0030035 (2007).

19. 19.

Millman, D., Mihalas, S., Kirkwood, A. & Niebur, E. Self-organized criticality occurs in non-conservative neuronal networks during Up states. Nat Phys. 6, 801–805 (2010).

20. 20.

Kirmse, K. et al. GABA depolarizes immature neurons and inhibits network activity in the neonatal neocortex in vivo. Nat Commun. 6, https://doi.org/10.1038/ncomms8750 (2015).

21. 21.

Rigas, P. et al. Spontaneous Up states in vitro: a single-metric index of the functional maturation and regional differentiation of the cerebral cortex. Front Neural Circuits. 9, 59, https://doi.org/10.3389/fncir.2015.00059 (2015).

22. 22.

Shu, Y., Hasenstaub, A. & McCormick, D. A. Turning on and off recurrent balanced cortical activity. Nature. 423, 288–293 (2003).

23. 23.

Prida, L. M. & Sanchez-Andres, J. V. Nonlinear frequency-dependent synchronization in the developing hippocampus. J Neurophysiol. 82, 202–208 (1999).

24. 24.

Cossart, R., Aronov, D. & Yuste, R. Attractor dynamics of network UP states in the neocortex. Nature. 423, 283–288 (2003).

25. 25.

Ozeki, H., Finn, I. M., Schaffer, E. S., Miller, K. D. & Ferster, D. Inhibitory stabilization of the cortical network underlies visual surround suppression. Neuron. 62, 578–592 (2009).

26. 26.

Tsodyks, M. V., Skaggs, W. E., Sejnowski, T. J. & McNaughton, B. L. Paradoxical effects of external modulation of inhibitory interneurons. J Neurosci. 17, 4382–4388 (1997).

27. 27.

Latham, P. E. & Nirenberg, S. Computing and stability in cortical networks. Neural Comput. 16, 1385–1412 (2004).

28. 28.

Chagnac-Amitai, Y. & Connors, B. W. Horizontal spread of synchronized activity in neocortex and its control by GABA-mediated inhibition. J Neurophysiol. 61, 747–758 (1989).

29. 29.

Minlebaev, M., Ben-Ari, Y. & Khazipov, R. Network mechanisms of spindle-burst oscillations in the neonatal rat barrel cortex in vivo. J Neurophysiol. 97, 692–700 (2007).

30. 30.

Rahmati, V., Kirmse, K., Markovic, D., Holthoff, K. & Kiebel, S. J. Inferring Neuronal Dynamics from Calcium Imaging Data Using Biophysical Models and Bayesian Inference. PLoS Comput Biol. 12, e1004736, https://doi.org/10.1371/journal.pcbi.1004736 (2016).

31. 31.

Valeeva, G., Tressard, T., Mukhtarov, M., Baude, A. & Khazipov, R. An Optogenetic Approach for Investigation of Excitatory and Inhibitory Network GABA Actions in Mice Expressing Channelrhodopsin-2 in GABAergic Neurons. J Neurosci. 36, 5961–5973 (2016).

32. 32.

Ben-Ari, Y., Gaiarsa, J. L., Tyzio, R. & Khazipov, R. GABA: a pioneer transmitter that excites immature neurons and generates primitive oscillations. Physiol Rev. 87, 1215–1284 (2007).

33. 33.

Kirmse, K., Witte, O. W. & Holthoff, K. GABAergic depolarization during early cortical development and implications for anticonvulsive therapy in neonates. Epilepsia. 52, 1532–1543 (2011).

34. 34.

Bakkum, B. W., Benevento, L. A. & Cohen, R. S. Effects of light/dark- and dark-rearing on synaptic morphology in the superior colliculus and visual cortex of the postnatal and adult rat. J Neurosci Res. 28, 65–80 (1991).

35. 35.

Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci USA. 79, 2554–2558 (1982).

36. 36.

Wang, X. J. Synaptic reverberation underlying mnemonic persistent activity. Trends Neurosci. 24, 455–463 (2001).

37. 37.

Seung, H. S., Lee, D. D., Reis, B. Y. & Tank, D. W. Stability of the memory of eye position in a recurrent network of conductance-based model neurons. Neuron. 26, 259–271 (2000).

38. 38.

Ben-Yishai, R., Bar-Or, R. L. & Sompolinsky, H. Theory of orientation tuning in visual cortex. Proc Natl Acad Sci USA 92, 3844–3848 (1995).

39. 39.

Jeong, H. Y. & Gutkin, B. Synchrony of neuronal oscillations controlled by GABAergic reversal potentials. Neural Comput. 19, 706–729 (2007).

40. 40.

Luccioli, S., Ben-Jacob, E., Barzilai, A., Bonifazi, P. & Torcini, A. Clique of functional hubs orchestrates population bursts in developmentally regulated neural networks. PLoS Comput Biol. 10, e1003823, https://doi.org/10.1371/journal.pcbi.1003823 (2014).

41. 41.

Garaschuk, O., Linn, J., Eilers, J. & Konnerth, A. Large-scale oscillatory calcium waves in the immature cortex. Nat Neurosci. 3, 452–459 (2000).

42. 42.

Mongillo, G., Barak, O. & Tsodyks, M. Synaptic theory of working memory. Science. 319, 1543–1546 (2008).

43. 43.

Wilson, H. R. & Cowan, J. D. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys J. 12, 1–24 (1972).

44. 44.

Loebel, A. & Tsodyks, M. Computation by ensemble synchronization in recurrent networks with synaptic depression. J Comput Neurosci. 13, 111–124 (2002).

45. 45.

Tsodyks, M., Uziel, A. & Markram, H. Synchrony generation in recurrent networks with frequency-dependent synapses. J Neurosci. 20, 1–5 (2000).

## Acknowledgements

This work was supported by the Open Access Publication Funds of the TU Dresden, the Priority Program 1665 of the German Research Foundation (HO 2156/3-1/2, KI 1816/1-1/2 and KI 1638/3-1/2), the Collaborative Research Center/Transregio 166 of the German Research Foundation (B3 to KH and KK), the Research Group FO 1738 (WI 830/10-2 to KH), the Federal Ministry of Education and Research (01GQ0923 to KH) and the Interdisciplinary Centre for Clinical Research Jena (to KK and KH). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

## Author information

Authors

### Contributions

V.R., K.K., K.H., L.S. and S.J.K. designed the study. V.R. designed the model, derived analytical expressions, and ran the experiments. V.R., K.K., K.H., L.S. and S.J.K. participated in the interpretation of the data and findings. V.R., K.K., K.H., L.S. and S.J.K. wrote the manuscript.

### Corresponding author

Correspondence to Vahid Rahmati.

## Ethics declarations

### Competing Interests

The authors declare that they have no competing interests.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Rahmati, V., Kirmse, K., Holthoff, K. et al. Developmental Emergence of Sparse Coding: A Dynamic Systems Approach. Sci Rep 7, 13015 (2017). https://doi.org/10.1038/s41598-017-13468-z

• Accepted:

• Published:

• ### Progressive alignment of inhibitory and excitatory delay may drive a rapid developmental switch in cortical network dynamics

• Alberto Romagnoni
• , Matthew T. Colonnese
• , Jonathan D. Touboul
•  & Boris S. Gutkin

Journal of Neurophysiology (2020)

• ### Somatostatin Interneurons Promote Neuronal Synchrony in the Neonatal Hippocampus

• Tom Flossmann
• , Thomas Kaas
• , Vahid Rahmati
• , Stefan J. Kiebel
• , Otto W. Witte
• , Knut Holthoff
•  & Knut Kirmse

Cell Reports (2019)

• ### Neuronal Synchronization Can Control the Energy Efficiency of Inter-Spike Interval Coding

• Siavash Ghavami
• , Vahid Rahmati
•  & Lars Schwabe

IEEE Transactions on Molecular, Biological and Multi-Scale Communications (2018)