Particle Mobility Analysis Using Deep Learning and the Moment Scaling Spectrum

Quantitative analysis of dynamic processes in living cells using time-lapse microscopy requires not only accurate tracking of every particle in the images, but also reliable extraction of biologically relevant parameters from the resulting trajectories. Whereas many methods exist to perform the tracking task, there is still a lack of robust solutions for subsequent parameter extraction and analysis. Here a novel method is presented to address this need. It uses for the first time a deep learning approach to segment single particle trajectories into consistent tracklets (trajectory segments that exhibit one type of motion) and then performs moment scaling spectrum analysis of the tracklets to estimate the number of mobility classes and their associated parameters, providing rich fundamental knowledge about the behavior of the particles under study. Experiments on in-house datasets as well as publicly available particle tracking data for a wide range of proteins with different dynamic behavior demonstrate the broad applicability of the method.

. Example of PDF (left) and CDF (right) fits on displacement histograms for a, b: simulated two-state diffusion data with = 0.1 2 / and = 1.0 2 / and c, d: simulated threestate data with = 1.0µ 2 / for the fast state, = 0.1µ 2 / for the slow state, and = 0.1, = 0.3 for the immobile state.

Supplementary Note 2: Limitations of MSD-based analysis
Most methods traditionally used in biological applications of single particle tracking are based on mean square displacements (MSD): where = 0, 1, 2, …, and is the data acquisition time interval, is the number of available windows for the given , and and are coordinates. For pure diffusion, exhibiting Brownian motion (BM) in two dimensions (2D), it holds that: with diffusion constant . Because of this known relationship between MSD and , the plot of MSD versus will be a straight line and the diffusion constant can be determined from the slope of the line, which is equal to 4 . Therefore, by fitting a straight line through the MSD measurements of every trajectory, an approximation of the diffusion constant of that trajectory can be calculated. The values of the logarithm of the diffusion constants of each particle trajectory can subsequently be plotted in a histogram to see if there are distinguishable populations (the logarithmic scale is used to magnify the distinction between peaks). This can for instance be done for a BRCA2 dataset ( Figure   S2), resulting in two observable peaks in the MSD histogram. A large drawback of this method is that trajectories that switch between mobility classes are being treated as if they exhibit only one type of motion, leading to an inaccurate estimation of for each mobility class. Additionally, this method does not take into account motion types that are not diffusive. Figure S2. An example of a histogram showing the frequency of diffusion constants that were estimated per BRCA2 trajectory using MSD on a logarithmic scale. There seem to be two populations: one with a higher diffusion constant and one with a lower diffusion constant. The red dotted line indicates the splitting point.

Supplementary Note 3: Long short term memory network
Long short term memory networks (LSTMs) are sophisticated recurrent neural networks that can selectively "remember" some past events and "forget" others over (iteration) time ( Figure S3).
Besides a hidden output state ℎ , LSTMs also maintain a cell memory state that allows the network to keep track of what is important to remember and what to forget. This idea is realized by gates that can selectively let information through. These gates are the forget gate , the input gate and the output gate . By using these gates, LSTMs manage to get only relevant information to pass through, enabling them to learn long-term dependencies.  The slow cluster can be split into separate clusters in a simple way using Gaussian fitting. However, to get accurate results for tracklets that switch between immobile and slow (or even between immobile and fast), a network that was trained with three states needs to be applied.
c: A network that was trained with three-state data yields the same type of pattern, but the clusters are more well-defined. Figure S6. Examples of trajectory classification by the LSTM network on simulated data (with ground truth available). Travelled distances of the particle are shown per time point for a hundred frames, along with the predicted class and the real class. Note: classification mistakes are mainly made at the ends of tracklets, meaning that tracklets are seldom broken up by wrongly classified states. Figure S7. Regions in the moment scaling spectrum. An = 0.5 represents pure diffusion, 0 < < 0.5 represents restricted motion and 0.5 < < 1.0 represents more directed motion.

Supplementary Note 7: Regions in moment scaling spectrum analysis
Some visual examples of different types of trajectories are shown on the right of the spectrum.

Supplementary Note 8: Use of as an invariable measure (irrespective of moment order )
In the main text, clustering of tracklets is done in -space, where indicates the type of motion and is used to distinguish between "faster" and "slower" motion. Intuitively, it might seem strange to use this measure, which characterizes diffusion specifically and is traditionally calculated using the second order moment only. Here it is shown that the diffusion constant is invariant over the moment order and can be used as a measure for subdiffusive (immobile) tracklets as well.
First of all, the diffusion constant can also be calculated from other moments. The moments can be calculated as with displacement, moment order, and PDF( ) = 2 − 2 2 2 because the displacements are Rayleigh distributed, where 2 = 2 is the variance ( diffusion constant, time step). This integral can be solved analytically to obtain = 2 /2 Γ (1 + 2 ).
Here, Γ(n) = (n − 1)! is the gamma function. Finally, this result can be rearranged to obtain a formula for as a function of : When is plotted against for simulated three-state data, it becomes clear that the is practically constant over ( Figure S8). Moreover, the diffusion constant is also constant for the immobile class, demonstrating that in combination with , can indeed be used as a measure to distinguish between fast diffusive, slow diffusive and immobile tracklets. is practically constant over . Figure S9. Effect of tracklet size ( ) on the location in -space (in three-state simulated data).
Smaller tracklets deviate more from the center of their corresponding cluster than larger tracklets.
For easier and more accurate determination of mobility parameters, only tracklets of ≥ 10 are used for clustering.
Supplementary Note 10: Runtimes of the DL-MSS software

Supplementary Note 11: Examples of additional property analysis and visualization
The output of DL-MSS analysis is not limited to the type of results shown in the main text of this paper. This section shows some examples of additional data analysis and visualizations.
Firstly, the results of classification by the deep learning network can be used to calculate transition probabilities between the different classes of mobility ( Figure S10). Secondly, depending on the research question, it may be interesting to know the mean dwell time of the particle in a certain state (Table S2). These dwell times can be derived only from those segments of the trajectories between any two successive points of switching between different motion types (the inner tracklets). This is because it is not known how long a particle was in the given motion state before entering or after leaving the field of view. Therefore, trajectories with less than two switch points, and the trajectory segments up to the first and after the last switch point (the outer tracklets), must be ignored. Even though this implies not all data can be used, the mean dwell times computed from the inner tracklets still provide useful information to compare different datasets.
One thing that stands out when comparing the dwell times of the datasets used in this paper is that the Spot-On datasets have much shorter dwell times, probably due to the fact that the acquisition speed of the datasets (time step of 5 ms) is much faster than our in-house datasets (time step of 30 ms). For higher acquisition speeds, the probability to detect switches in type of mobility is larger, as switches might be "missed" when a lower acquisition speed is used.
Thirdly, it is possible to calculate the fraction of timepoints spent in each state (Table S2). This gives yet another measure that makes it easier to compare different datasets. These type of patterns can be uncovered by studying the distribution of different types of mobility inside the cell ( Figure S11). For instance, tracklets can be colored by class (Figure S11b) or the density of displacements of different mobility types can be represented as a heat map (Figure S11c,d,e). The first case that made this ability clear was a dataset of H2B trajectories that were tracked using settings optimized for BRCA2 tracking. For H2B trajectories, it is expected that nearly all trajectories exclusively exhibit the immobile class. When a scatterplot in -space was made for whole trajectories instead of tracklets (Figure S12a), there appeared an extra cluster that was faster than the immobile cluster (higher ), and even more confined (lower ). The scatterplot for the segmented tracklets ( Figure S12b) showed that even though most tracklets were classified as immobile or slow, there were some unexpected fast tracklets as well. This was a sign that the trajectories in the strange additional clusters mainly consisted of immobile tracklets with only a few steps of faster, confined motion.
Revision of the H2B trajectories obtained from microscopy data indicated that some spots were wrongly linked from one time frame to another creating large displacements that should not be there, a problem that could be solved by adjusting the tracking parameters. After this adjustment, the strange cluster nearly disappeared ( Figure S12c). The same effect can also be simulated by introducing a bigger displacement every few time frames ("big skips"), for example in pure diffusion ( Figure S12d). This gives the same type of extra cluster for whole trajectories as the H2B molecules before adjustment of tracking parameters. Overall, this result indicated that DL-MSS analysis can help identifying defects in tracking.