Abstract
Musicians can perform at different tempos, speakers can control the cadence of their speech, and children can flexibly vary their temporal expectations of events. To understand the neural basis of such flexibility, we recorded from the medial frontal cortex of nonhuman primates trained to produce different time intervals with different effectors. Neural responses were heterogeneous, nonlinear, and complex, and they exhibited a remarkable form of temporal invariance: firing rate profiles were temporally scaled to match the produced intervals. Recording from downstream neurons in the caudate and from thalamic neurons projecting to the medial frontal cortex indicated that this phenomenon originates within cortical networks. Recurrent neural network models trained to perform the task revealed that temporal scaling emerges from nonlinearities in the network and that the degree of scaling is controlled by the strength of external input. These findings demonstrate a simple and general mechanism for conferring temporal flexibility upon sensorimotor and cognitive functions.
Main
Mental capacities such as anticipation, motor coordination, deliberation, and imagination lie at the heart of higher brain function. A fundamental feature of these capacities is that they are not tied to immediate sensory or motor events and unfold at different timescales. To support such temporal flexibility, the brain must control the dynamics of ongoing patterns of neural activity. An example of such flexible behavior is the control of selfinitiated movements. Humans can precisely control the timing of their movements and can make rapid adjustments based on instruction. However, the mechanisms that confer such flexibility are not well understood.
We investigated the neural mechanisms underlying flexible temporal control. We developed a task in which monkeys were instructed to produce different time intervals using different effectors. While monkeys performed the task, we evaluated the causal function and signaling properties of neurons across three brain areas that have been strongly implicated in timing: (i) the medial frontal cortex (MFC), which has been implicated in the inhibition^{1}, initiation^{2, 3}, and coordination^{4,5,6,7} of movements, (ii) the caudate nucleus downstream of MFC, which is thought to play a major role in timing tasks^{8,9,10,11,12,13,14,15}, and (iii) thalamic regions that project to MFC and causally influence selfinitiated movements^{16}.
Neurons exhibited a diversity of complex response profiles that could not be reconciled with dominant models of timing^{13}, including clockaccumulator models^{17, 18}, oscillationbased models^{19}, and population clock models^{20, 21}. Instead, responses were unified under a general principle of temporal scaling that was evident at both individual and population levels. Specifically, when animals produced longer intervals, the population activity evolved along an invariant neural trajectory but at a slower speed. Notably, speed was adjusted on a trialbytrial basis and in accordance with the instruction provided to the animal. Although these findings are at odds with classic models of timing, they corroborate observations of temporal scaling in other tasks and areas^{8, 22,23,24,25}.
To investigate the mechanisms underlying such flexible speed control, we analyzed the dynamics of recurrent neural network models capable of using graded input to produce different time intervals. Analysis of these models revealed a previously uncharacterized yet simple mechanism for flexible temporal scaling: degree of scaling was controlled by an external input acting upon the nonlinear activation function of individual neurons in a recurrent network.
Results
Behavior
On each trial, monkeys fixated a central spot with their hand resting on a button and produced either a Short (800ms) or Long (1,500ms) interval using one of the effectors (Eye or Hand). The desired interval and effector changed on a trialbytrial basis and was cued throughout the trial by the color and shape of the fixation point (Fig. 1a). Production intervals (Tp) were measured from a brief ‘Set’ flash to the time of movement initiation. Animals learned to flexibly switch between conditions (Fig. 1b) and produced accurate intervals whose variability increased for the Long condition compared to the Short condition (Fig. 1c). This is consistent with Weber’s law and is a wellknown property of timing behavior^{26, 27}. The Weber fraction was significantly larger for button presses compared to saccades (onetailed pairedsample t test, for monkey A, n = 31, t _{30} = 1.80, P = 0.041, and for monkey D, n = 35, t _{34} = 6.44, P < 0.001).
Causal experiments and singleunit electrophysiology
Reversible inactivation of MFC (Fig. 2a) with muscimol, a GABA_{A} agonist, significantly impaired performance for both Long and Short intervals (Fig. 2b). This was evident from a comparison of the distributions of withinsession increases in the meansquared error after the muscimol injection versus before the injection (for statistics, see Table 1). The drop in performance was due to a combination of changes in both bias and standard deviation (Fig. 2b). No significant impairment was measured after saline injection (Fig. 2b and Table 1). Furthermore, muscimol inactivation had no significant effect on reaction times during a memory saccade task (Table 1). Based on these results, we concluded that MFC played a causal role in the main motor timing task.
Temporal scaling of complex response profiles
To estimate each neuron’s firing rate, we binned trials based on Tp and computed average spike counts after aligning trials to the time of the motor response (Fig. 2c). Across neurons, response profiles were highly heterogeneous and included linear, nonlinear, monotonic, nonmonotonic, and multimodal activity profiles (Fig. 2d). We tested each neuron’s activity profile against predictions of various models of motor timing using a crossvalidation procedure (Fig. 2e). We considered three variants of the clockaccumulator model: one in which flexible timing was achieved by adjusting a threshold over a ramping process, one in which the clock was adjusted, and one in which both were adjusted. Since clock models can only accommodate neurons with linear ramping profiles^{17, 18, 28,29,30}, they failed to capture the nonlinear profiles exhibited by the majority of neurons in the population. Crossvalidated polynomial fits of different degrees of freedom indicated that only 11% (47 of 416) of responses increased linearly; the rest were explained by higherorder polynomials. This number increased by only 4% when the starting and terminating points of the linear ramps were allowed to vary by up to 200 ms.
We also tested two oscillationbased models of interval timing, in which the response time is determined by the collective phase of oscillators and different frequencies^{19}. In one variant, a single sinusoid was fit to the response of each neuron, and in another, multiple sinusoids (up to four) of different frequencies were used. These models were also unable to capture the diversity of MFC responses (Fig. 2e).
Finally, we tested MFC responses against a simple variant of the populationclock model^{20, 21}, in which the response profile of each individual neuron is unique and contextindependent, and the collective activity of the population determines movement initiation time. Accordingly, we modeled each neuron by the bestfitting polynomial (crossvalidated) that captured the activity across both the Short and Long contexts. This model performed better than the clockaccumulator and oscillation models. However, MFC data violated a key qualitative prediction of the population clock model: unlike in the population clock model, the vast majority of MFC responses differed for the Short and Long conditions from early on after the Set cue (Fig. 2d).
Our initial inspection indicated that response profiles were selfsimilar when stretched or compressed in accordance with the produced interval (Fig. 2c,d and Supplementary Fig. 1). This was true for both random fluctuations of Tp within each temporal context (i.e., 800 ms or 1,500 ms) and deliberate adjustments of Tp across the two contexts. Consistent with this observation, a temporally scaled polynomial function fitted to the data for different conditions clearly outperformed all other models in terms of explanatory power (Fig. 2e; oneway ANOVA, F _{6, 2,859} = 125.2, P < 0.001).
Speed control across the population
We quantified the degree of scaling by a scaling index (SI) that was computed as a coefficient of determination (R ^{2}) across temporally scaled responses associated with different Tp bins. This analysis revealed a wide range of SI values across the population (Supplementary Fig. 1a). When activity of a population of neurons is plotted in a coordinate system in which each axis represents the firing rate of one neuron, also known as the state space, the response dynamics of the population can be depicted as a highdimensional neural trajectory. In this representation, perfect temporal scaling would result in perfectly overlapping neural trajectories evolving at different speeds. When we plotted MFC neural trajectories within the space spanned by the first three principal components (PCs) of neural activity, responses did not overlap perfectly, indicating that MFC responses comprised a mixture of scaling and nonscaling signals (Fig. 3a), which was also evident from the distribution of SI values across individual neurons (Supplementary Fig. 1a).
We hypothesized that perfect scaling might be found within a subspace of the population activity, i.e., a scaling subspace (Fig. 3b). As a first step, we examined the degree of scaling in the first few PCs. Using the same SI metric used for single neurons, we found that the first two PCs that explained nearly 40% of the variance (Fig. 3b) had scaling indices of 0.91 and 0.97, respectively (Fig. 3a). The third PC, however, did not exhibit temporal scaling and had a SI of 0.20. This provided initial evidence that certain highvariance dimensions in the state space exhibit strong scaling. However, scaling dimensions need not coincide with PCs, since PCs correspond to dimensions of maximum variance, not maximum scaling. To identify the scaling dimensions, we developed a dimensionality reduction technique that furnished a set of scaling components (SCs) that were ordered according to the degree of scaling in the data (see Methods).
The SI values for the first few SCs were relatively large, indicating that the optimization process correctly identified the scaling dimensions (Fig. 3c and Supplementary Fig. 2). Because SCs were crossvalidated, the scaling index for SCs of the test data did not follow a strictly decreasing order, although this was the case for the dataset used to determine the SCs (data not shown). Responses projected onto the subspace spanned by the first three SCs traced nearly identical trajectories that evolved at different speeds (Fig. 3c), which is precisely what is expected in the scaling subspace.
Next, we asked how much variance in the neural data the scaling subspace could account for. Ordered SCs explained less variance than the corresponding PCs, suggesting that the scaling dimensions were not identical to PC dimensions (Fig. 3c). To better quantify the relationship between scaling and variance explained, we performed two complementary analyses. First, we examined the relationship between SI and variance explained for each SC. This analysis provided initial evidence that SCs with large SIs explained a relatively large percentage of variance (Fig. 3d). Second, we developed a procedure for quantifying the relationship between scaling and variance without relying on projections onto specific directions, such as PCs or SCs. We used a bootstrap procedure and quantified the relationship between variance explained and SI along 200 random projections in the state space. We then constructed a twodimensional probability distribution of the relationship between variance explained and SI across those random projections (Fig. 3d). This analysis verified that the dimensions with large degrees of scaling also explained a large portion of the variance.
To validate SI as a reliable metric for scaling, we quantified SI for surrogate data created from Gaussian processes. The surrogate data was constructed to statistically match MFC responses in terms of smoothness, starting and terminal firing rates, dimensionality, and the correlation between Short and Long activity profiles, but it was not constrained to exhibit temporal scaling (see Supplementary Note and Supplementary Fig. 3). The surrogate data, despite being matched to the statistics of MFC responses, had smaller SIs than those computed for MFC neurons (Fig. 3e). This verified that a significant portion of variance in MFC resides within a scaling subspace in which activity evolves along invariant trajectories at different speeds.
Finally, we quantified the relationship between speed in the scaling subspace and behavior. Using crossvalidation, we derived the scaling subspace from a subset of shortest and longest trials and asked whether the speed of neural trajectories of the remaining trials in that subspace could predict Tp. Results indicated that longer Tps were associated with slower speeds (Fig. 3f and Supplementary Fig. 4) and that the average speed was inversely proportional to Tp (R ^{2} = 0.87). These results suggest that the brain controls the speed of neural trajectories in order to flexibly produce different time intervals. Notably, this speed control seemed to explain both behavioral variability within each temporal context and flexible switching between the two contexts.
Speed control across cortico–basal ganglia circuits
Having established speed control in MFC as a potential mechanism for temporal flexibility, we asked whether this property was also present downstream of MFC in the basal ganglia. We focused on a region of the caudate that is thought to receive direct input from MFC^{31,32} (Fig. 4a,b). First, we used reversible inactivation to verify the causal involvement of this region in the task (Fig. 4b and Table 1). Afterwards, we recorded from individual neurons (Fig. 4c) and analyzed their responses with respect to the temporal scaling property. Caudate responses, like those in MFC, were complex and heterogeneous and had different profiles for Short and Long trials. At the level of single neurons, the degree of scaling in the caudate was similar to that in MFC (Supplementary Fig. 1). At the population level, analysis of PCs and SCs verified the presence of a scaling subspace in the caudate (Fig. 3e and Supplementary Fig. 5). Finally, the SI values of PCs, as well as an unbiased analysis of responses across random projections in the state space, indicated that dimensions with strong scaling explained a large part of variance in the data (Fig. 4d). These analyses verified that neural signals in the caudate shared the same key properties with MFC and could contribute to subspace speed control.
In addition to receiving inputs from MFC, the basal ganglia also projects back to MFC through the thalamus. The presence of this anatomical substrate raises the possibility that MFC inherits temporal scaling from the basal ganglia via transthalamic projections. To test this possibility, we targeted a region of the thalamus where MFCprojecting thalamocortical neurons were identified antidromically (Fig. 4e and see Methods). Consistent with previous work^{16}, reversible inactivation strongly influenced animals’ timing behavior (Table 1). However, several observations indicated that the function of thalamocortical signals was different from the functions of caudate and MFC signals (Fig. 4g). First, SIs of single thalamic neurons (n _{thalamus} = 846) were significantly smaller across the population compared to the other areas (n _{MFC} = 416 and n _{caudate} = 278, Mann–Whitney–Wilcoxon test, W _{1,260} = 310,733, z = 7.89, P < 0.001 for MFC; W _{1,120} = 189,163, z = 6.98, P < 0.001 for caudate; Supplementary Fig. 1a). Second, scaling in the thalamus was significantly smaller than the C+D+E+S surrogate data that matched the neural data in terms of smoothness (S), endpoints (E), dimensionality (D), and correlation (C; onetailed twosample t test, n = 200, t _{398} = 35.2, P < 0.001; Fig. 3e). Third, scaling was less prominent in the thalamus, as indicated by the relationship between the magnitude of scaling and variance explained along random projections in the state space (Fig. 4h). Fourth, unlike in the caudate and MFC, neural trajectories in the thalamus were not invariant in the space spanned by the first three SCs (Supplementary Fig. 5). This was also evident in the profile of the second PC, which systematically changed in average value, as opposed to scaling. Together, these observations provide strong evidence that thalamic neurons exhibit significantly less scaling than the MFC neurons they project to. Since the output of the basal ganglia to cortex is routed through the thalamus, the weak scaling in thalamocortical neurons implies that scaling may originate within MFC or in other cortical circuits projecting to MFC.
A model for flexible subspace speed control
Since the timescales of MFC response modulations were slower than the intrinsic time constants of single neurons, we assumed that the observed dynamics were the result of networklevel interactions. Motivated by recent advances in understanding the dynamics of cortical population activity using network models^{33,34,35}, we used a recurrent neural network model to investigate the potential underlying mechanisms of speed control (Fig. 5). The model received a context input (Cue) whose magnitude specified the desired interval and a transient pulse (Set) that cued the start of the interval (Fig. 5a). The network was trained so that its output (a weighted linear sum of its units) had to breach a fixed threshold at the desired time^{36}.
The network learned to generate the desired output function (Fig. 5d), and the activity of model neurons emulated the key features observed in MFC: response profiles of individual network units were heterogeneous, complex, and temporally scaled (Fig. 5b). Moreover, the speed of population dynamics directly determined the produced interval (Fig. 5c). These observations were robust regardless of whether the training objective was linear, nonlinear, scaling, or nonscaling (Supplementary Fig. 6). The scaling behavior also persisted when the Cue input was provided transiently (Supplementary Fig. 6). Motivated by the robustness and generality of these results, we reverseengineered the networks to investigate the underlying mechanisms of temporal scaling^{37}.
Temporal scaling could be explained in terms of a pair of inputdependent stable fixed points, F _{init} and F _{terminal}. At the start of the trial, the Cue initialized the state of the network to an inital fixed point, F _{init}. Activation of the Set pulse drove the system away from F _{init}, allowing the system to evolve toward F _{terminal} with a speed that was determined by the magnitude of the Cue input (Fig. 5c,e). Within the network, the input and the recurrent dynamics played complementary roles (Fig. 5c). The input specified the position of the initial and terminal fixed points along a direction, which we refer to as the input subspace. Recurrent dynamics on the other hand, established a recurrent subspace, which determined the neural trajectory between these fixed points. These two subspaces emerged from different components of the network. The input subspace was governed by the direction specified by the input weights. In contrast, the recurrent subspace emerged from the constraints imposed by the recurrent weights. The two subspaces also differed in terms of their relationship to the scaling phenomenon. Within the input subspace, different intervals were associated with changes in the level of activity but did not exhibit scaling. This change in level controlled the speed by setting the position of the neural state along the axis of the input subspace. The recurrent space, on the other hand, did not control the speed but was responsible for the emergence of invariant trajectories and temporal scaling.
The division of labor between these subspaces provides a simple explanation of why scaling and nonscaling signals might coexist within the same network. Nonscaling signals reflect the input that sets the speed, and scaling signals correspond to the evolution of activity with the desired speed. This organization predicts that MFC neurons with weak temporal scaling are likely recipients of relatively strong contextdependent input, possibly derived from signals in upstream thalamic neurons (Fig. 4g), and that neurons with strong temporal scaling are more directly engaged in recurrent interactions. Finally, the modelbased distinction between these two subspaces provides a theoretical basis for analyzing MFC responses within a scaling subspace that corresponds to the recurrent subspace in the model.
Notably, the model allows us to infer that within the nonscaling input subspace, production times should be correlated with the average level—not speed—of neural activity. To test this prediction, we investigated whether Tp could be predicted by the nonscaling component of MFC activity. We inferred the leastscaling direction from our scaling component analysis. SCs specified an orthonormal basis whose axes were ordered according to the level of scaling (Supplementary Fig. 7). Therefore, we used the last SC (SC9) as an estimate of the leastscaling direction and compared Tp to average MFC activity projected onto SC9. As predicted by the model, the average activities of the nonscaling components of MFC were indeed predictive of Tp (Supplementary Fig. 8). This is a compelling result, as it bears out a key prediction about an unsuspected relationship between cortical activity and behavior made by a model that was constrained only to perform the task.
A potential neural mechanisms for speed control
To further investigate the role of input in speed control, we analyzed the eigenvalues of the system near F _{terminal}. In the vicinity of this fixed point, stronger inputs caused the eigenvalues to decrease systematically (Fig. 5f). In a linear dynamical system, such contraction in the eigenvalue spectrum corresponds to a systematic increase in the network’s effective time constants, τ _{eff} (Fig. 5f). From this, we concluded that the action exerted by the input is equivalent to adjusting the system’s effective time constant in a flexible inputdependent manner.
To gain insight into the mechanism that provides such powerful and modular control of time constants, we focused on a simplified model composed of only two mutually inhibitory neurons with a common input (Fig. 6a and Supplementary Note). Previous work has demonstrated that adjustments of the common input in this model could alter its recurrent dynamics to either relax to a single fixed point with a specific time constant or act as an integrator with exceedingly long time constants^{38}. We reasoned that exploring the model’s behavior while between these two regimes might lead us to a mechanistic understanding of how the effective time constant of a network can be flexibly adjusted.
In the presence of balanced input (Cue), the twoneuron model is associated with an energy landscape that engenders a pair of stable fixed points, similarly to the recurrent model (Fig. 6b). We analyzed the phase plane of the model (Fig. 6c) and verified that the input level can be used to create a continuum of τ _{eff}. This is analogous to the recurrent network model in which activity along the input subspace served to control the speed. However, the twoneuron model helped us understand the underlying mechanisms: stronger input drives neurons toward their saturating nonlinearity, where the slopes of activation functions are shallower (Fig. 6d). Shallower slopes correspond to smaller derivatives and larger values of τ _{eff}. In other words, the presence of singleneuron nonlinearities provides a reservoir of slopes that an input can exploit to control the network’s energy gradients (Fig. 6b).
Having established a lowlevel mechanism in the twoneuron model, we asked whether the same mechanism was operative in the recurrent network model. For the recurrent model, we analyzed the operating points of units as a function of the input drive near F _{terminal}. Notably, for stronger inputs, units were systematically driven further toward their saturating nonlinearity (Fig. 5g,h), which is consistent with the mechanism of speed control in the simple network model. These results underscore a simple and powerful mechanism at the level of single neurons for controlling the speed of dynamics independent of the neural trajectory.
Discussion
We found that flexible motor timing was governed by controlling the speed of slow dynamics across populations of MFC and caudate neurons. Speed control also emerged as a natural solution in recurrent network models trained to produce different time intervals. This was achieved by an input that drove the system to the appropriate region of the state space, where recurrent interactions unfolded at desired speeds. In both systems, fluctuations of speed predicted variability within each temporal context, and systematic adjustments of speed provided the means for flexible control of timing. These results suggest that the brain uses a speedcontrol mechanism to deliberately control movement initiation time.
The division of labor conferred by the input and recurrent interactions has broad implications for flexible control of behavior, allowing the same motor and cognitive functions to unfold along the same neural trajectory at different timescales. For example, in decisionmaking tasks, adjustment of a speed command could explain how the brain might flexibly implement different speed–accuracy tradeoffs^{39}. Indeed, if the speed command is controlled by a sensory input, our recurrent network would behave similarly to more detailed network models consisting of excitatory and inhibitory units that approximate temporal integration of sensory information^{40}. However, biophysical models of decision making have not yet been extended to generate the diversity of scaling response profiles that we observed in vivo and in our recurrent model.
The engineered twoneuron model highlights the crucial role of singleneuron nonlinearities; adjustments of speed were governed by the interaction of input with these nonlinearities. This finding suggests that circuits and subcircuits could exploit different inputs and different biophysical properties to adjust speed independently and operate at different timescales. It also predicts that neuromodulatory effects and pharmacological treatments that interfere with the nonlinear response curve of individual neurons could alter the speed of cortical dynamics, as observations from numerous studies of interval timing might suggest^{41}.
The source of the external input that adjusts the speed remains a pertinent and unresolved question. One possibility is that MFC receives this input directly from neurons in other cortical areas, which is consistent with recent observations in the parietal cortex^{42}. Another possibility is that the input has a thalamocortical origin. Thalamic neurons, in turn, may inherit this signal from other cortical and/or subcortical regions. Neuromodulatory signals could also alter cortical dynamics. A number of physiology and pharmacology studies have implicated dopamine in regulating timing behavior^{43, 44}. Cortical dynamics are also known to depend on cellular properties, such as those mediated by NMDA receptors, which are thought to facilitate the generation of stable slow cortical dynamics^{45}.
Another question for future work concerns the exact mechanisms that give rise to the diversity of response profiles in MFC. According to our model, this diversity emerges from recurrent interactions in direct response to an input drive. Alternatively, these activity patterns could be the result of cortical nonlinearities acting upon simpler ramping inputs, which constituted a minority of response profiles in the cortico–basal ganglia circuits we recorded from. Indeed, considering the bidirectional connections between thalamus and cortex, we cannot rule out the possibility that ramping activity in thalamus and/or other cortical areas might contribute to the scaling of more complex response profiles in MFC. Nevertheless, the model seems to provide the most parsimonious account of the data for both cortex and thalamus. The exact details of the signaling pathways, recurrent microcircuitry, and biophysical properties notwithstanding, the mechanisms that we have identified have the potential to explain how the brain flexibly controls the speed of cortical dynamics.
Methods
Methods
Two adult rhesus monkeys (Macaca mulatta, a 6.5kg female and a 9.0kg male, both 5 years old) were trained on a twointerval twoeffector motor timing task. All surgical, behavioral, and experimental procedures conformed to the guidelines of National Institutes of Health and were approved by the Committee of Animal Care at Massachusetts Institute of Technology.
Behavior
The MWorks software package (https://mworks.github.io/) running on a Mac Pro was used to deliver stimuli and to control behavioral contingencies. Visual stimuli were presented on a 23inch (58.4cm) monitor at a refresh rate of 60 Hz. Eye positions were tracked with an infrared camera (Eyelink 1000; SR Research Ltd, Ontario, Canada) and sampled at 1 kHz. A custommade manual button, equipped with a trigger and a force sensor, was used to register button presses.
Motor timing task
Each trial began with the appearance of two fixation cues (FCs), a circle at the center of the screen and a square 0.5° below the circle. The animal had to shift its gaze to the circle, and the square informed the animal to hold its hand gently on the button. On each trial, one FC was colored and the other was white. The colored FC indicated the desired response effector (colored circle for saccade and colored square for button press). The color indicated the desired interval (red for 800 ms and blue for 1,500 ms). We denote these four trial conditions by EL, ES, HL, and HS, where E and H refer to Eye and Hand and S and L to Short (800ms) and Long (1,500ms) intervals. After a delay period (500–1,500 ms, uniform hazard), the saccade target was briefly presented 8° to the left or right of the FC. For buttonpress trials (colored square), the saccadic target was not relevant but was presented so that stimuli were consistent across trials. After another delay (500–1,500 ms, uniform hazard), a 48ms annulus (Set cue) flashed around the FCs cued the animal to start timing. Trials were aborted if the animal made premature eye or hand movements (before Set or long before the desired time). To receive the reward, animals had to initiate a movement with the desired effector (cued by the colored FC) within a small window (‘acceptance window’) around the desired interval (cued by the color of FC). The saccade responses had to land inside a circular window of radius 2.5° centered on the location of the extinguished target and had to be made directly (less than 33 ms after exiting the FC window). Buttonpress responses had to be made with the hand contralateral to the recorded hemifield^{46}. The production interval was measured from the endpoint of Set to the moment the saccade was initiated or the button was triggered. The width of the acceptance window was adjusted dynamically on a trialbytrial basis and independently for the Short and Long conditions using a oneuponedown staircase procedure. As such, animals were rewarded for nearly half of trials (on average, 57% in monkey A and 51% in monkey D) for both temporal contexts. For trials that were rewarded, in addition to reward delivery, the color of the stimulus changed to green and an auditory clicking sound was simultaneously presented. Within the acceptance window, the magnitude of the reward scaled with accuracy.
Electrophysiology
Animals were comfortably seated in a dark and quiet room. Each session began with an approximately 10min warmup period to allow animals to recalibrate their timing and exhibit stable behavior during electrophysiology recordings. Recordings were made through a craniotomy within a recording chamber while the animal’s head was immobilized. Structural MRI scans were used to aid in targeting regions of interest. Single and multiunit responses were recorded using a 24channel laminar probe with 100µm or 200µm interelectrode spacing (Vprobe, Plexon Inc.). Eye position was sampled at 1 kHz, and all behavioral and electrophysiological data were timestamped at 30 kHz and streamed to a data acquisition system (OpenEphys).
The dataset collected for this study included 1,967 single units or multiunits recorded from the MFC, caudate and thalamus of two monkeys (Table 2), in which 69% (1,351/1,967) were tentatively single units. Neurons with firing rates less than 2 spikes per s during the timing epoch were excluded from subsequent analyses.
Reversible inactivation
Injections were made with a microinjection pump (UMP3, World Precision Instruments) and a Hamilton syringe, which was connected to a custom 30 G stainless steel injection cannula via a fused silica injection line (365µm OD, 100µm ID, Polymicro Technologies). In each injection session, we first established the animal’s baseline behavioral performance. Afterwards, we pressureinjected muscimol hydrobromide (5 µg/µL in saline) in the region of interest at a rate of 0.2 µL/min. In the MFC and caudate, a total of 2 µL was injected per session. In pilot inactivation experiments in the thalamus, we noticed that animals stopped performing the task after 2 µL muscimol injection. To ensure animals would perform the task, the total volume of muscimol in the thalamus was reduced to 1.5 µL. The behavioral task was resumed 10 min after the injection was completed. As a control, in separate sessions, sterile saline was injected following the same procedure. The experimental data consisted of unequal test sessions for muscimol and saline, and unequal numbers of trials in the before and after muscimol injection. For statistical comparison, these inequalities may introduce sampling biases. To avoid such biases, we created 50trial minisessions from before and after the injections, in which the trials within a minisession were randomly sampled. The sampling was made without repeats to ensure trials were not counted twice. We quantified the effects of inactivation by comparing mean squared error, bias and variance, \({\rm{MSE}}=\sum {({T}_{p}{T}_{s})}^{2}={{\rm{Bias}}}^{2}+{\rm{Var}}\), before and after the injection for every minisession. The same procedure was used to assess the results of the saline injection experiments.
Antidromic stimulation
We used antidromic stimulation to localize thalamocortical MFCprojecting neurons. Antidromic spikes were recorded on a 24channel electrode (Vprobe, Plexon Inc.) in response to a single biphasic pulse of duration 0.2 ms (current < 500 µA) delivered to MFC via lowimpedance tungsten microelectrodes (100–500 kΩ, Microprobes). The guide tube for the tungsten electrode was used as the return path for the stimulation current. Antidromic activation evoked spikes reliably at a latency ranging from 1.8 to 3 ms, with less than 0.2 ms jitter. The region of interest targeted in the thalamus was within 1 mm of antidromically identified neurons.
Mathematical notation
Throughout the manuscript, we have used lowercase for scalars (\(x\)), bold and lowercase for vectors (\({\boldsymbol{x}}\)), bold and uppercase for matrices (\({\boldsymbol{X}}\)). Brackets were used for indexing vectors and matrices (\({\boldsymbol{x}}[i]\) and \({\boldsymbol{X}}[i,j]\)). Subscripts were used for indexing a set of scalars (\({x}_{i}\)), vectors (\({{\boldsymbol{x}}}_{i}\)), or matrices (\({{\rm{X}}}_{i}\)). Subscripts were also used to show projections onto a subspace. For example, \({{\boldsymbol{x}}}_{PC(1:k)}\) refers to a vector projected onto the first k principal components. Curly brackets were used to indicate a subset of conditions. For example, \({\rm{x}}\{a{\rm{}}={\rm{}}{a}_{0};b{\rm{}}={\rm{}}{b}_{1}\}\) refers to a vector computed for a subset of trials in which both a = a _{0} and b = b _{1} conditions were satisfied. The symbol \(\cup \) was used to indicate data combined across a number of variables. For example \({\bigcup }_{i}^{N}\{{{\boldsymbol{x}}}_{i}^{}\}\) denotes data collected across a union of vectors \({{\boldsymbol{x}}}_{i}\). The symbol <\({\boldsymbol{x}}\)>_{ i } was used to show averaging of a vector x across i. Point functions were shown as lowercase (\(f(.)\)) regardless of whether they were applied to scalars or vectors.
Data analysis
All offline data processing and analyses were performed in Matlab (2016b, MathWorks). Spiking data were bandpassfiltered between 300 Hz and 7 kHz, and spike waveforms were detected at a threshold that was typically set to 3 × the RMS noise. Single units and multiunits were sorted offline using custom software, MKsort (https://github.com/rippleneuro/mksort). The majority of the neurons were recorded in separate behavior sessions.
Estimating firing rates accurately is challenging when rates change dynamically and trials have different durations^{47, 48}, which was the case in our data. Since our focus was on firing rates leading up to the movement, we aligned trials with respect to movement time (Fig. 2c). Additionally, for each condition, we discarded trials with Tp values more than 3 s.d. away from the mean (1.46% of trials). Firing rates were estimated by (i) averaging spike counts per time bin, (ii) using a 40ms Gaussian kernel to compute smooth spiking density functions, and (iii) zscoring to minimize sampling bias due to baseline and amplitude differences across neurons.
To examine the relationship between firing rates and Tp values, we binned trials according to Tp and compared average firing rates for each bin. For the 800ms interval, we used seven bins centered on 740 to 860 ms every 20 ms, and for the 1,500ms interval, we used nine bins centered on 1,300 to 1,620 ms every 40 ms. We denoted the average firing rate of a neuron as a function of time by \(r(t)\), average firing rate for a specific condition c (EL, ES, HL or HS) by \(r(t;c)\), and average firing rate for a specific condition and a specific Tp bin by \(r(t;c,Tp)\). For population analyses, response vectors of individual neurons were organized into rows of a matrix denoted by \({\boldsymbol{r}}(t;c,Tp)\).
To test whether activity profiles could be described by a linear function (for example, ramping activity), we compared 0order to 8thorder polynomial fits to \(r(t)\) using crossvalidation with randomized train and test sets. All neurons that were best explained by a polynomial of order 0 or 1 were considered linear so long as the fit explained at least 50% of variance. We also applied the same procedure allowing up to 200 ms offset from the beginning or end of the timing interval to ensure our results were robust.
Comparing the motor timing models at the level of single and multiunits
To avoid overfitting and facilitate comparison of models with different levels of complexity, all model fitting was performed on the training set and the goodness of fit (R ^{2}) was quantified on the test set. In the clockaccumulator model with a flexible threshold, a linear ramp with fixed slope and different thresholds for different production intervals was fit to the response profile. In the clockaccumulator model with a flexible clock, the threshold was fixed and the ramping rate was adjusted according to the interval. In the clockaccumulator model with both flexible clock and flexible threshold, a linear ramp was adjusted according to the interval and its offset was independently adjusted for each interval. In the oscillation based models, sinusoidal functions or a sum of up to four different sinusoids were fit to activity profiles, in which the frequency, amplitude, and phase for each sinusoid were free parameters. In the population clock, a single polynomial of up to 8th order was fit to the response profiles for both Short and Long contexts. For the temporal scaling model, the response profiles for the Short condition were used to find the bestfitting polynomial, and the temporally scaled versions of the fitted functions were used to test the goodness of fit for Long trials.
Scaling subspace
We used a principal component analysis (PCA) as a first step to compute a lowdimensional and unbiased estimate of data. We found that the first nine principal components (PCs) captured nearly 80% of the variance in the data (Fig. 3b). We therefore computed the scaling components (SCs) from data captured by the first nine PCs, which was computed as follows:
\({{\boldsymbol{r}}}_{PC}(t;c)=\hat{{\boldsymbol{V}}}* {\boldsymbol{r}}(t;c)\) where \(\hat{{\boldsymbol{V}}}=[{v}_{1}^{T};{v}_{1}^{T};...;{v}_{{N}_{PC}}^{T}]\) is the projection matrix and \({v}_{i}\) is the ith PC direction. Therefore, the denoised activity across all conditions and time points \({{\boldsymbol{r}}}_{PC}(t;c)\)is of size \({N}_{pc}\times \left(T\times C\right)\). We computed the corresponding scaled responses using our scaling procedure and denoted the result by \({{\boldsymbol{r}}}_{PC}^{S}(t;c)\). To find the scaling subspace, we solved an optimization problem that minimized the difference between average firing rates associated with different Tp values (for example, \(T{p}_{i}\) and \(T{p}_{j}\)). We denote the corresponding projection by \({U}_{SC}\) and refer to its columns as scaling components (SCs). The resulting projection \({{\boldsymbol{r}}}_{SC}\) can be computed as follows:
We hypothesized that the speed of activity in the scaling subspace predicts Tp. We computed the instantaneous speed in the scaling subspace from projections of responses on to the first three SCs as follows:
For each interval bin, we obtained an unbiased estimate of the relationship between speed and \(Tp\) by resampling trials with replacement within each interval bin. The relationship between the average speed \(S(Tp)\) and production intervals was fitted in the log space by a linear function:
Scaling index for population data
We quantified temporal scaling in single units, principal components (PCs) and scaling components (SCs) using a scaling index (SI) that represented a general measure of the degree of similarity between multiple response profiles associated with different intervals. SI was computed as follows: (i) trials were sorted based on production interval (Tp); (ii) sorted trials were grouped into bins of similar Tp values (as described above); (iii) the first nine PCs and the corresponding SCs for each bin were computed; and (iv) for each PC and SC, the index was computed as the coefficient of determination(R ^{2}) after the PCs and SCs were temporally scaled. This metric, which varies between 0 and 1, quantifies the degree to which each PC/SC undergoes temporal scaling for different Tp values.
We evaluated the degree of scaling among populations in each region of interest by computing the scaling index for each PC and SC in those populations. Additionally, we computed the variance explained by each SC. Finally, to gain an unbiased estimate of the relationship between variance explained and scaling index, we computed these two metrics along randomly selected dimensions within the state space. This analysis revealed the full distributions of variance explained and scaling index and their relationship within the whole state space.
Recurrent network architecture
We constructed a firing rate recurrent neural network (RNN) model with N nonlinear units (N = 200). The network dynamics was governed by the following differential equation:
Variable x(t) is an Ndimensional vector representing the activity of all the units. Variable r(t) represents the firing rates of those units by transforming x through a tanh saturating nonlinearity. The time constant of each neuron was set to τ = 10 ms. This value is different from τ _{eff}, which emerges at the network level. Variable c _{ x } is a vector representing a stationary offset the units receive, and \(\rho (t)\) is a vector representing white noise N(0, 0.01) sampled at each timestep Δt = 1 ms. The recurrent connections in the network are specified by matrix J, whose initial values, following previous work on balanced networks, are drawn from a normal distribution with zero mean and variance \(1/N\). The network receives a twodimensional input u consisting of a context cue u _{ c }(t) and a transient Set pulse u _{ s }(t). The network received these inputs through synaptic weights \({\boldsymbol{B}}=\left[{b}_{c},{b}_{s}\right]\), which were initialized to random values drawn from a uniform distribution with range –1 to 1.
The context input, u _{ c }, represents the intervaldependent context cue input. The value of u _{ c } was set to 0 for 100 ms and then jumped to a graded value proportional to the length of one of 16 desired intervals distributed within a range 500–1,700 ms. The offset of u _{ c } was sampled proportionally from the range 0.1 to 0.6 and was perturbed with Gaussian noise N(0,0.25) at each Δt. Increasing input noise did not qualitatively alter the network training solutions. The transient Set pulse u _{ s }(t) was active for 10 ms with magnitude 0.1 and zero elsewhere. On each training and test trial, the interval between the onset of u _{ c } and u _{ s }(t) was drawn from a uniform distribution with range (100–200 ms).
The network produced a onedimensional output z(t), read out by the summation of linear units with weights w _{ o } and a bias term c _{ z }. The output weights were initialized to zero at the start of training.
Statistics
The Weber fractions across behavioral sessions (Fig. 1c), MSEs before and after inactivation (across minisessions; Table 1 and Figs. 2a and 4b,f), scaling indices obtained from a bootstrap procedure for various brain areas, and surrogate data (Fig. 3e) were assumed to be normally distributed, but this was not formally tested a priori. Depending on assumptions associated with various sessions, onetailed paired or unpaired sample t tests were used. Neurons with extremely low firing rates (less than 2 spk/sec) during the timing epoch were excluded from further analysis. The number of neurons recorded in all three areas in both monkeys and those excluded are reported in Table 2. For singleneuron responses with respect to the seven types of timing models, we used oneway ANOVA to establish that the explanatory power quantified by R ^{2} of various models were significantly different. Then we used post hoc pairedsample t tests to compare temporal scaling model with each alternative model (Fig. 2e). The scaling indices of neurons in different brain areas (Supplementary Fig. 1a) were not normally distributed. For this reason, we used a nonparametric unpaired Mann–Whitney–Wilcoxon test to compare independent samples from pairs of brain areas under examination (thalamus and MFC, thalamus and caudate).
Life Sciences Reporting Summary
Further information on experimental design is available in the Life Sciences Reporting Summary.
Data availability statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
 1.
Stuphorn, V. & Schall, J. D. Executive control of countermanding saccades by the supplementary eye field. Nat. Neurosci. 9, 925–931 (2006).
 2.
Kunimatsu, J. & Tanaka, M. Alteration of the timing of selfinitiated but not reactive saccades by electrical stimulation in the supplementary eye field. Eur. J. Neurosci. 36, 3258–3268 (2012).
 3.
Fried, I. et al. Functional organization of human supplementary motor cortex studied by electrical stimulation. J. Neurosci. 11, 3656–3666 (1991).
 4.
Lewis, P. A., Wing, A. M., Pope, P. A., Praamstra, P. & Miall, R. C. Brain activity correlates differentially with increasing temporal complexity of rhythms during initialisation, synchronisation, and continuation phases of paced finger tapping. Neuropsychologia 42, 1301–1312 (2004).
 5.
Shima, K. & Tanji, J. Neuronal activity in the supplementary and presupplementary motor areas for temporal organization of multiple movements. J. Neurophysiol. 84, 2148–2160 (2000).
 6.
Isoda, M. & Hikosaka, O. Switching from automatic to controlled action by monkey medial frontal cortex. Nat. Neurosci. 10, 240–248 (2007).
 7.
Lu, X., Matsuzawa, M. & Hikosaka, O. A neural correlate of oculomotor sequences in supplementary eye field. Neuron 34, 317–325 (2002).
 8.
Mello, G. B. M., Soares, S. & Paton, J. J. A scalable population code for time in the striatum. Curr. Biol. 25, 1113–1122 (2015).
 9.
Gouvêa, T. S. et al. Striatal dynamics explain duration judgments. eLife 4, e11386 (2015).
 10.
Jin, D. Z., Fujii, N. & Graybiel, A. M. Neural representation of time in corticobasal ganglia circuits. Proc. Natl. Acad. Sci. USA 106, 19156–19161 (2009).
 11.
Matell, M. S., Meck, W. H. & Nicolelis, M. A. L. Interval timing and the encoding of signal duration by ensembles of cortical and striatal neurons. Behav. Neurosci. 117, 760–773 (2003).
 12.
Merchant, H., Harrington, D. L. & Meck, W. H. Neural basis of the perception and estimation of time. Annu. Rev. Neurosci. 36, 313–336 (2013).
 13.
Mauk, M. D. & Buonomano, D. V. The neural basis of temporal processing. Annu. Rev. Neurosci. 27, 307–340 (2004).
 14.
Bartolo, R., Prado, L. & Merchant, H. Information processing in the primate basal ganglia during sensoryguided and internally driven rhythmic tapping. J. Neurosci. 34, 3910–3923 (2014).
 15.
Schultz, W. & Romo, R. Neuronal activity in the monkey striatum during the initiation of movements. Exp. Brain Res. 71, 431–436 (1988).
 16.
Tanaka, M. Inactivation of the central thalamus delays selftimed saccades. Nat. Neurosci. 9, 20–22 (2006).
 17.
Treisman, M. Temporal discrimination and the indifference interval. Implications for a model of the “internal clock”. Psychol. Monogr. 77, 1–31 (1963).
 18.
Killeen, P. R. & Fetterman, J. G. A behavioral theory of timing. Psychol. Rev. 95, 274–295 (1988).
 19.
Matell, M. S. & Meck, W. H. Corticostriatal circuits and interval timing: coincidence detection of oscillatory processes. Brain Res. Cogn. Brain Res. 21, 139–170 (2004).
 20.
Karmarkar, U. R. & Buonomano, D. V. Timing in the absence of clocks: encoding time in neural network states. Neuron 53, 427–438 (2007).
 21.
Buonomano, D. V. & Laje, R. Population clocks: motor timing with neural dynamics. Trends Cogn. Sci. 14, 520–527 (2010).
 22.
Brody, C. D., Hernández, A., Zainos, A. & Romo, R. Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cereb. Cortex 13, 1196–1207 (2003).
 23.
Komura, Y. et al. Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 412, 546–549 (2001).
 24.
Merchant, H., Zarco, W., Pérez, O., Prado, L. & Bartolo, R. Measuring time with different neural chronometers during a synchronizationcontinuation task. Proc. Natl. Acad. Sci. USA 108, 19784–19789 (2011).
 25.
Emmons, E. B. et al. Rodent medial frontal control of temporal processing in the dorsomedial striatum. J. Neurosci. 37, 8718–8733 (2017).
 26.
Gibbon, J. Scalar expectancy theory and Weber’s law in animal timing. Psychol. Rev. 84, 279 (1977).
 27.
Rakitin, B. C. et al. Scalar expectancy theory and peakinterval timing in humans. J. Exp. Psychol. Anim. Behav. Process. 24, 15–33 (1998).
 28.
Creelman, C. D. Human discrimination of auditory duration. J. Acoust. Soc. Am. 34, 582–593 (1962).
 29.
Gibbon, J., Church, R. M. & Meck, W. H. Scalar timing in memory. Ann. NY Acad. Sci. 423, 52–77 (1984).
 30.
Grondin, S. From physical time to the first and second moments of psychological time. Psychol. Bull. 127, 22–44 (2001).
 31.
Hikosaka, O., Sakamoto, M. & Usui, S. Functional properties of monkey caudate neurons. I. Activities related to saccadic eye movements. J. Neurophysiol. 61, 780–798 (1989).
 32.
Parthasarathy, H. B., Schall, J. D. & Graybiel, A. M. Distributed but convergent ordering of corticostriatal projections: analysis of the frontal eye field and the supplementary eye field in the macaque monkey. J. Neurosci. 12, 4468–4488 (1992).
 33.
Sussillo, D., Churchland, M. M., Kaufman, M. T. & Shenoy, K. V. A neural network that finds a naturalistic solution for the production of muscle activity. Nat. Neurosci. 18, 1025–1033 (2015).
 34.
Li, N., Daie, K., Svoboda, K. & Druckmann, S. Robust neuronal dynamics in premotor cortex during motor planning. Nature 532, 459–464 (2016).
 35.
Mante, V., Sussillo, D., Shenoy, K. V. & Newsome, W. T. Contextdependent computation by recurrent dynamics in prefrontal cortex. Nature 503, 78–84 (2013).
 36.
Hanes, D. P. & Schall, J. D. Neural control of voluntary movement initiation. Science 274, 427–430 (1996).
 37.
Sussillo, D. & Barak, O. Opening the black box: lowdimensional dynamics in highdimensional recurrent neural networks. Neural Comput. 25, 626–649 (2013).
 38.
Machens, C. K., Romo, R. & Brody, C. D. Flexible control of mutual inhibition: a neural model of twointerval discrimination. Science 307, 1121–1124 (2005).
 39.
Hanks, T., Kiani, R. & Shadlen, M. N. A neural mechanism of speedaccuracy tradeoff in macaque area LIP. eLife 3, e02260 (2014).
 40.
Wang, X.J. Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36, 955–968 (2002).
 41.
Meck, W. H. Neuropharmacology of timing and time perception. Brain Res. Cogn. Brain Res. 3, 227–242 (1996).
 42.
Jazayeri, M. & Shadlen, M. N. A neural mechanism for sensing and reproducing a time interval. Curr. Biol. 25, 2599–2609 (2015).
 43.
Buhusi, C. V. & Meck, W. H. Differential effects of methamphetamine and haloperidol on the control of an internal clock. Behav. Neurosci. 116, 291–297 (2002).
 44.
Soares, S., Atallah, B. V. & Paton, J. J. Midbrain dopamine neurons control judgment of time. Science 354, 1273–1277 (2016).
 45.
Chaudhuri, R., Knoblauch, K., Gariel, M.A., Kennedy, H. & Wang, X.J. A largescale circuit mechanism for hierarchical dynamical processing in the primate cortex. Neuron 88, 419–431 (2015).
 46.
Fujii, N., Mushiake, H. & Tanji, J. Distribution of eye and armmovementrelated neuronal activity in the SEF and in the SMA and PreSMA of monkeys. J. Neurophysiol. 87, 2158–2166 (2002).
 47.
Namboodiri, V. M. & Hussain Shuler, M. G. Report of interval timing or action? Proc. Natl. Acad. Sci. USA 111, E2239–E2239 (2014).
 48.
Xu, M., Zhang, S.Y., Dan, Y. & Poo, M.M. Representation of interval timing by temporally scalable firing patterns in rat prefrontal cortex. Proc. Natl. Acad. Sci. USA 111, 480–485 (2013).
Acknowledgements
We thank M.S. Fee, J.J. DiCarlo, and R. Desimone for comments on the manuscript, and we thank D. Sussillo for advice on modeling. D.N. was supported by the Rubicon Grant (2015/44614008) from the Netherlands Scientific Organization (NWO). M.J. is supported by the NIH (NINDSNS078127), the Sloan Foundation, the Klingenstein Foundation, the Simons Foundation, the Center for Sensorimotor Neural Engineering, and the McGovern Institute.
Author information
Affiliations
Contributions
J.W. was responsible for all aspects of experiments and analyses and developed the simplified model. D.N. was responsible for the development of the recurrent neural network model. E.A.H. helped with the data collection and analysis. M.J. was responsible for all aspects of the project. All authors helped with the interpretation of data and writing the paper.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Integrated Supplementary Information
Supplementary Figure 1 Scaling index (SI) of single neurons in different brain areas
(a) Histograms on the top show the normalized distribution of the scaling index for individual neurons in the MFC (n = 416 neurons from both animals), caudate (n = 278 neurons), and thalamus (n = 846 neurons) for the Eye (Left) and Hand conditions (Right). The bottom panel shows a comparison of the cumulative probability distribution of scaling index across the three areas. The thalamus shows a predominance of smaller scaling index value (MannWhitneyWilcoxon test, onetailed, W(1260) = 310,733, z = 7.89, *** P < .001 in comparison with MFC, and W(1122) = 189,163, z = 6.98, *** P < .001 in comparison with caudate) (b) Example PSTHs covering a range of SIs in the three brain areas. The SI value and effector condition for each neuron is indicated.
Supplementary Figure 2 The time course of PCs (left) and SCs (right) for MFC data
First 9 PCs over the course of the production interval (abscissa) that explain 80% of variance in the MFC data (n = 281 neurons for Monkey A) in decreasing order of variance explained (left) for Short (warm colors) and Long (cool colors) intervals. First 9 SCs, obtained for the same data, in decreasing order of scaling (right, see Methods).
Supplementary Figure 3 Analysis of scaling with surrogate data
(a) Venn diagram showing the various constraints considered for nonscaling models. All surrogate data was generated from a Gaussian Process (GP) with the same level of temporal smoothness (white rectangle, S) as the data. We considered three additional constraints to make the surrogate data more similar to neural data without an explicit requirement for scaling. One constraint required responses for all production intervals to be at the same level at the time of Set and at the time of response. We refer to this constraint as endpoint matching (red circle, E). Another constraint required that the dimensionality of the surrogate data match the neural data, and additionally the variance explained by each principal component (PC) be matched. We refer to this constraint as dimensionality matching (green circle, D). Finally, we considered a constraint that required the collection of responses for different production intervals to have the same correlation (quantified as R^{2}) as expected from perfect scaling. We refer to this constraint as correlation matching (blue circle, C). We created surrogate data for each constraint and for various combination of constraints, and compared the scaling properties to the original data. Note that each constraint characterized a superset of the scaling hypothesis. (b) Example traces showing the procedure for generating the surrogate data (n = 281 surrogate units) in the C+D+E+S model for 5 randomly selected surrogate units aligned to the time of Set. We first sampled a Short trace (red) from a Gaussian process. The trace in blue corresponds to the perfectly scaled version of the red trace and is not a sample from the surrogate model. The surrogate data were generated using a constrained Gaussian Process (GP) prior as follows: the response for the shortest production intervals (red) was sampled from a GP with the same level of temporal smoothness as the neural data. The corresponding response with perfect scaling was generated by linear scaling (shown in blue). Note that the trace in blue is not a sample from the GP and is therefore, not part of the surrogate data. The gray traces correspond to the surrogate data. To generate the surrogate data, we drew samples from the Gaussian process that satisfied several criteria. First, the starting point as well as the ending point of every gray trace had to be perfectly matched to the starting point and ending point of the perfectly scaled blue trace. Second, across the population of surrogate data, the dimensionality had to match observed neural data. Finally, the correlation between every gray trace and the red trace was the same as the correlation between the red and blue trace. In this way, every sample of GP (gray traces) matched the smoothness, endpoints, dimensionality and correlation as the real data (i.e., C+D+E+S model). (c) Cumulative percentage variance explained by PCs and SCs for the surrogate data generated from the nonscaling C+D+E+S model. (d) The first 9 principal components of population activity (PC, left) and the corresponding 9 scaling components (right, SCs) plotted as the function of time from Set for the nonscaling C+D+E+S model. Note that PCs and SCs are based on the surrogate data (gray traces in panel b) – not the perfectly scaled data (blue traces in panel b).
Supplementary Figure 4 Relationship between MFC population activity and behavior
The speed of neural trajectory in MFC (n = 281 neurons for Monkey A, n = 135 neurons for Monkey D) within the scaling subspace spanned by the first 3 SCs predicted Tp across both Short and Long conditions on a trialbytrial basis. Both ordinate and abscissa follow a logarithmic scale. The case of hand trials for Monkey A was shown in Fig. 5d. Here, all the other conditions are shown.
Supplementary Figure 5 Analysis of scaling at the population level in the caudate (top) and thalamus (bottom)
Left column: Population activity (n = 101 neurons for caudate and n = 481 neurons for thalamus, Moneky A) profiles projected onto the first 3 principal components (PCs). Activity profiles associated with different produced intervals for Short and Long conditions are plotted in different colors (same color scheme used throughout the paper). The state at 700 ms after Set is shown along the trajectories (diamond). Second column from the left: Population activity projected onto the first 3 scaling components (SCs). Activity spanned by the first 3 SCs overlap for different intervals in the caudate but not in the thalamus. Third column from left: Variance explained for individual SCs as a function of scaling index. Right column: The speed of neural trajectory within the scaling subspace spanned by the first 3 SCs. Both ordinate and abscissa follow a logarithmic scale.
Supplementary Figure 6 Alternative recurrent neural network models
(a) A recurrent neural network (RNN) trained to use an intervaldependent Cue input to produce time intervals flexibly. The network was trained using a nonscaling exponential objective with a fixed time constant. (b) The speed of dynamics measured within the space spanned by the first three PCs predicted Tp across both Short and Long conditions. Both ordinate and abscissa follow a logarithmic scale. (c) The response profiles of randomly selected units in the network (a) aligned to the time of Set. (d) A RNN trained to use a brief pulse as the intervaldependent Cue input to produce time intervals flexibly. The network was trained using a linear ramping objective like the network in the main text. (e) The speed of dynamics predicted Tp across both Short and Long conditions with the same format as b). (f) The response profiles of randomly selected units in network (d) aligned to the time of Set.
Supplementary Figure 7 Temporal scaling in the recurrent network mode
(a) The first 9 scaling components (SCs) of the population activity in the RNN with a scaling output function (Fig. 5). The early SCs correspond to the recurrent subspace, and the last SC represents the input subspace. (b) The average speed of population activity in the subspace spanned by the first 3 SCs is predictive of both within context and across context variations in Tp. Both ordinate and abscissa follow a logarithmic scale. (c) The average firing rate of the population activity projected onto the last SC is also predictive of Tp. (d) Cumulative percentage variance explained by PCs (white) and SCs (black). The dashed vertical line corresponds to the 9th component. (eh) Same as ad, for a network that was trained for a nonscaling exponential output objective function (Supplementary Fig. 6d).
Supplementary Figure 8 Nonscaling population activity in MFC
Left: The time course of the SC9 (the least scaling component) across conditions. Right: The average firing rate of population activity (n = 281 neurons for Monkey A) projected onto SC9 (left), also the putative input subspace, increases with produced intervals (Tp). This is consistent with the hypothesis that the average firing rate in the nonscaling subspace controls speed. Based on the recurrent network model, this subspace likely reflects the input drive to MFC. Both ordinate and abscissa are plotted in a logarithmic scale.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–7 and Supplementary Note
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Wang, J., Narain, D., Hosseini, E.A. et al. Flexible timing by temporal scaling of cortical responses. Nat Neurosci 21, 102–110 (2018). https://doi.org/10.1038/s4159301700286
Received:
Accepted:
Published:
Issue Date:
Further reading

Generation of ScaleInvariant Sequential Activity in Linear Recurrent Networks
Neural Computation (2020)

Sequence Memory in the Hippocampal–Entorhinal Region
Journal of Cognitive Neuroscience (2020)

A Network of Topographic Maps in Human Association Cortex Hierarchically Transforms Visual TimingSelective Responses
Current Biology (2020)

Discovering Precise Temporal Patterns in LargeScale Neural Recordings through Robust and Interpretable Time Warping
Neuron (2020)

Coding with transient trajectories in recurrent neural networks
PLOS Computational Biology (2020)