Leukocyte Tracking Database, a collection of immune cell tracks from intravital 2-photon microscopy videos

Recent advances in intravital video microscopy have allowed the visualization of leukocyte behavior in vivo, revealing unprecedented spatiotemporal dynamics of immune cell interaction. However, state-of-the-art software and methods for automatically measuring cell migration exhibit limitations in tracking the position of leukocytes over time. Challenges arise both from the complex migration patterns of these cells and from the experimental artifacts introduced during image acquisition. Additionally, the development of novel tracking tools is hampered by the lack of a sound ground truth for algorithm validation and benchmarking. Therefore, the objective of this work was to create a database, namely LTDB, with a significant number of manually tracked leukocytes. Broad experimental conditions, sites of imaging, types of immune cells and challenging case studies were included to foster the development of robust computer vision techniques for imaging-based immunological research. Lastly, LTDB represents a step towards the unravelling of biological mechanisms by video data mining in systems biology.

Recent advances in intravital video microscopy have allowed the visualization of leukocyte behavior in vivo, revealing unprecedented spatiotemporal dynamics of immune cell interaction. However, state-of-the-art software and methods for automatically measuring cell migration exhibit limitations in tracking the position of leukocytes over time. Challenges arise both from the complex migration patterns of these cells and from the experimental artifacts introduced during image acquisition. Additionally, the development of novel tracking tools is hampered by the lack of a sound ground truth for algorithm validation and benchmarking. Therefore, the objective of this work was to create a database, namely LTDB, with a significant number of manually tracked leukocytes. Broad experimental conditions, sites of imaging, types of immune cells and challenging case studies were included to foster the development of robust computer vision techniques for imaging-based immunological research. Lastly, LTDB represents a step towards the unravelling of biological mechanisms by video data mining in systems biology.

Background & Summary
Multi-Photon Intravital Video Microscopy (MP-IVM), in combination with image-based systems biology 1 , represent a key methodology for studying the interplay of cells in organs and tissues of living animals 2 . Indeed, recent analyses of leukocyte migration in MP-IVM data, highlighted unprecedented cell-to-cell interaction patterns such as antigen capturing 3 and presentation 4 , hostpathogen interaction 5,6 , tumor immune surveillance 7 and cell activation 8 amongst others. The advantage of MP-IVM with respect to other optical methods relies on the usage of multiple infrared photons. The low energy of the photons allows a deep and point-wise excitation of the sample which reduces light scattering and limits photo-damage. These properties make MP-IVM suitable to capture 4D data with remarkable resolution, depth and prolonged periods of observation 9 . The most common image acquisition and analysis pipeline of MP-IVM data is depicted in (Fig. 1). Initially, an animal having fluorescent cells, is anaesthetized, and prepared for imaging by immobilization and surgical exposition of the organ of interest (Fig. 1a left). Then, 4D data, composed of parallel image planes at different depths, are acquired for several time instants (Fig. 1a right). After acquisition, data are analysed by detecting cells (Fig. 1b left), tracking their position over time (Fig. 1b right) and finally quantifying cell migration 10 . The described pipeline was used to generate all the entries proposed in the current work (Fig. 1d).
Despite the existence of specialized imaging software packages such as Imaris (Bitplane), Volocity (PerkinElmer) and FIJI 11 , the automatic analysis of immune cell migration 10 in MP-IVM data is problematic. Challenges are introduced at each stage of the previously described pipeline and arise both from the complex biomechanical properties of leukocytes and from technical artifacts of in vivo imaging ( Fig. 1, Table 1 and Fig. 2a). More specifically, high plasticity of cell shape, sustained speed and frequent contacts, set a limit on the capacity of detecting and tracking cells for long time periods 12 . Additionally, technical artifacts such as the variation and non-uniform diffraction of the light emitted by fluorescentlytagged cells or the physiological movement of the sample due to peristalsis, breathing or pulsing of blood vessels, further challenge the automatic analysis. Therefore, additional steps such as image pre-processing, tuning of software parameters and manual curation of tracks, are required to improve tracking results. As a consequence, usability of imaging software is reduced 13 , bias introduced and the reproducibility of the results is compromised. An example is provided in (Fig. 2b)  Track length and Track duration were computed for the entry LTDB017a (Data Citation 1). These values exhibited highly significant differences (p 0:0001) between automatically-generated vs. manuallygenerated tracks.
Providing the scientific community with datasets interpreted by experts is essential to foster the development of data science methods. To this end, international cell tracking challenges on public datasets 14,15 allowed to highlight the properties amongst different algorithms. However, the provided datasets did not include leukocytes observed by intravital imaging. For this reason, it is necessary to develop an extended dataset of MP-IVM videos, where a significant number of leukocytes are tracked. Here we present a leukocyte tracking database, namely "LTDB", that includes MP-IVM videos of immune cells, together with their relative tracks which were manually annotated by experts. Each video contains one or more challenges for the automatic analysis (Table 2 (available online only)), and captured the behaviour of one or more cell populations (Table 3) in response to different stimuli (Table 4).
All the videos and tracks are made available as individual files or as a spatio-temporal database ( Fig.  3a) which was optimized for faster access to data and metadata (Fig. 3b).
The expected usage of LTDB is to serve as a ground truth for the validation of tracking algorithms (Fig. 4a). Differences with respect to the ground truth can be evaluated using, for instance, a metric that accounts for complete tracking graph comparison 16 .
LTDB further aims at being a training dataset for supervised machine learning methods. Indeed, in light of the recent application of deep learning for object detection and tracking in highly variable scenarios [17][18][19] , LTDB can provide the large number of images-tracks pairs required for the training of predictive models (Fig. 4b). In this case, broad imaging conditions may support the generalization capabilities of these methods.
Although LTDB was provided to primarily enhance tracking algorithms, the database embeds biomedical knowledge. To this end, data-mining and image-based systems biology methods can be applied to correlate images, tracks and metadata for investigating properties of the immune system in health and disease (Fig. 4c).

Imaging data generation
Experiments were performed by four research groups using three customized two-photon microscopy platforms (Table 5). Either the splenic or the lymph-node surgical models were used for acquisition (Fig.  1a). Videos were acquired from 26 unique experiments, to observe the interplay of neutrophils, B cells, T cells and natural killer cells in innate or adaptive immune responses (Table 4).  Data pre-processing No image processing was applied to the provided videos. RAW images were also used for manual tracking. Cropping of large 4D volumes in space and/or time was performed for the entries of the case study collection to focus on the area of interest.

Manual tracking
Centroids of cells were manually annotated and linked over time, using the "Spots drawing" tool from Imaris (Bitplane). This process was performed by a group of three operators who tracked all the cells

Consensus tracking ground truth generation
Multiple independent annotations and tracks were merged into the consensus ground truth provided along with the dataset using a majority-voting scheme. This process was performed manually by a fourth expert using the "Unify" functionality of Imaris. The Matlab script LTDBCheck.m was used to facilitate   A video belongs to a Collection, depicts one or more Problematic and includes an Image series. The Image entity is double-framed because it is a weak entity, which depends on the Video entity. A Cell has one type and one unique identifier. One Video tracks one (or more) Cell, every Cell being depicted by the Track association at a given timestamp (t) and in a spatial position (x, y, z) of that Video. The VisibleIn association further describes the channel of the video in which a cell is visible. The logical database is derived from the conceptual model and then optimized for read-access. The cells_positions table stores the instantaneous coordinates of each cell and is pre-exported in one or more CSV text files for each video. Imaging data are logically stored as TIFF image series with a specific filename. track matching, detecting common errors and highlighting conflictive situations. Two tracks were said likely to "match" (i.e. referring to the same cell) if their annotations were closer than 10um for at least N time instants. N was defined as the minimum between the track duration and 10. Conflictive situations were detected as tracks matching for only certain time instants but not for the entire track duration. These include a) tracks with an annotation in a far position by mistake, b) a longer track matching with one or multiple shorter tracks, c) two tracks matching for N instants but having different initial and/or final positions (i.e. track switches for closely interacting cells) amongst others. Tracks with a duration shorter than 4 time instants were also inspected manually. Due to the high plasticity of cells these criteria were used only to facilitate the work of the fourth expert who had to manually merge multiple tracks as follows: If at least two operators agreed on the direction of a cell, the track was included in the dataset (i.e. two matching tracks having the same duration and detected in the same frames). If two operators tracked a cell, but the track duration was different, the points annotated only by one operator were evaluated, confirmed or discarded by the fourth expert. When two operators could not agree on the direction of a cell, the following method was applied. If the fourth expert or the Matlab script identified an evident tracking error (i.e. cells not annotated by mistake, unrealistic jumps or broken tracks) the error was corrected and the tracks were merged. For real conflictive situations (i.e. track switching for closely interacting cells) the experts were asked to meet and discuss the most appropriate solution. If still the majority consensus could not be reached, and only in this case, tracks were interrupted. Finally, the position of cell centroids included in the ground truth was not averaged but selected as the centroid closer to the mean. Although this choice may produce less smooth tracks, it avoids to position a centroid outside non-convex cells. These criteria together with the manual merging of tracks and re-evaluation of tracking conflicts, allowed to include the maximum number of tracks for the longest possible period of time.

Animal models
The mouse strains included in this study are specified in Table 6 (available online only).
Prior to imaging, mice were anesthetized with a cocktail of Ketamine (100 mg/Kg) and Xylazine (10 mg/Kg) as previously described 4

Code availability
To facilitate the usage of LTDB, the following Matlab code is provided, under the GPL v3 Open Source licence, at http://www.ltdb.info/downloads/ or via Git-Hub at https://github.com/IRB-LTDB/.
LTDBReadTracks.m Reads the tracks contained in a CSV file.  XTLTDBImportTracks.m Imports tracks as spots in Imaris. LTDB2C2C.m Exports LTDB tracks in the format used for the Cell Tracking Challenge described in 14,15 , mainly for evaluation with the methods proposed in 15 .
LTDBCheck.m Checks for common tracking errors (i.e. annotations deleted by mistakes, broken tracks and overlapping tracks).
EstimateDSMeasures.m Estimates the measures in (Table 2 (available online only)) regarding the dataset complexity.
LTDBCheck.m and EstimateDSMeasures.m make use of the following libraries: Imar-isReader (https://github.com/PeterBeemiller/ImarisReader) to read Imaris files and bwdistsc 20 to efficiently estimate the distance of each voxel from the closest centroid.

Data Records
Data included in this work (Videos and Tracks) are available through figshare (Data Citation 1) Images resulting from MP-IVM are contained in two zip archives with name TIFFS_LTDB001_LTDB020.zip for the videos with ID 001 to 020, and in TIFFS_CS001_CS018.zip for the videos in the case study collection with ID 001 to 018. In these archives, a folder for each video contains 4D images as TIFF files.
Tracks resulting from the consensus tracking ground truth generation, are contained in the archive

GT_TRACKS.zip
A dump of the SQL database used to organize data and metadata is provided in LTDB.sql.
The following supplementary files are available through figshare (Data Citation 1). For a quicker preview, each is available in a H264 encoded MP4 file named oVideoID>.mp4.
A snapshot of all the videos is contained in the archive SNAPSHOTS.zip respectively named Individual tracks produced by different operators are provided in the supplementary archive operator_individual_tracks.zip and named o VideoID>_oTrackID>_ oOpID>. csv. In this case oOpID> is the ID of the operator (OP1, OP2 or OP3).

Technical Validation
Imaging data Imaging data were captured from organs of living animals using either the splenic or the popliteal lymph node surgical models (Fig. 1a and Table 4) which are typical for MP-IVM investigations of the immune system 2 . Cells involved in both innate and adaptive responses were included in the dataset. Videos 12, 13, 14 (Data Citation 1) come from recently published MP-IVM studies 5,7,21 . To represent data generated by multiple laboratories in different experimental settings 22 , LTDB includes videos with different size, resolution, sampling rate and challenges for the automatic analysis (Table 2 (available online only)), acquired by three different microscopy platforms (Table 5). Moreover, cells were labelled with different fluorescent tags and detected by multiple channels ( Table 3).
The following measures were computed to estimate the complexity of each video: signal to noise ratio (SNR), minimum distance between two cells (Dist) and number of cells per time instant. Since the proposed dataset is centroid-based rather than segmentation-based, SNR was estimated by adapting the definitions proposed in 15 with the following heuristic. Let c i,t be the centroid position of cell i at time t. For each voxel v in the current frame, the distance to the closest centroid was computed as d v ¼ minðjjvc i;t jjÞ8i. Then, considering a typical cell diameter of 10um, each voxel v was defined as foreground (FG-inside a cell) or background (BG-outside a cell) according with (Equation 1). This assumption allowed to sample a sufficient number of points in each video to estimate the aforementioned measures. ( Tracks The consensus tracking ground truth provided with LTDB includes 728 unique tracks composed of 44722 instantaneous annotations. On average, each track is composed by 61 annotations. This varying with the track duration. The total observation time included in LTDB amounts to the equivalent of 260 hours for a single cell. Common tracking errors (i.e. cells not annotated by mistake, broken tracks or jumps in the z-axis) as well as conflicts produced by multiple operators were detected by executing the Matlab script LTDBCheck.m provided in the code availability section.
Individual operators produced 1850 tracks (113807 annotations) which were merged into the 728 tracks of the consensus tracking ground truth. The performances of each operator with respect to the consensus ground truth is reported in (Table 7). To this end, the TRA 15 measure was computed. This measure includes a complete comparison of tracks represented as an acyclic oriented graph 16 . In order to estimate this measure, the ground truth and the individual tracks were converted in the format described in 15 and evaluated using the TRAMeasure software provided along. However, that software and methodology matches a cell in the ground-truth with a cell in the track to be evaluated, when they overlap more than 50% in space. Being our dataset centroid-based a difference of 1 voxel would made the matching not possible. Hence, considering the typical cell diameter, we approximated a sphere around each of the centroids. The tolerance radius of the spheres was at maximum of 10um and was truncated in case of two centroids closer than 10um. The script LTDB2CTC.m was used to export the LTDB tracks in the acyclic oriented graph format described in 14,15 .

Usage Notes
The expected use case scenario of LTDB is the evaluation of results produced by a cell tracking algorithm (Fig. 3a). Considering a generic cell tracking algorithm as an input-output system that reads an image sequence and outputs the tracks, LTDB can be used both as a source of images and as a ground truth for comparing the output.
To assess the overall performances of a cell tracking algorithm, we direct the user of LTDB towards the entries LTDB001 to LTDB020. To test the behaviour of an algorithm on specific cases instead, we recommend the user with the videos in the Case Study collection CS001 to CS018 that facilitates manual investigation and debugging having a reduced number of cells.
4D images are provided as uint16 TIFF files. File names were structured as oVi-deoID>_Txxx_Cxxx_Zxxx.tiff where oVideoID> is either(LTDB001 to LTDB020 or CS001 to CS018), the suffix xxx after T, C, Z indicates time instant, channel number and depth level respectively and spans from 000 to 999 maximum. Images with a lower bit depth were stored as uint16 without any scaling. If needed, for normalization the bit-depth of each video can be found in (Table 2 (available online only)).
Tracks are provided in the CSV Format described in (Table 8) and named oVideoID>_o TrackID>_GT.csv. For videos with only one cell population oTrackID> is "a", while for videos Row Table 8. Structure of the CSV track file. The position of the centroids of all the cells tracked in a video was saved in a ASCII CSV file. Columns are delimited by the semicolon separator and rows are terminated by CR LF. The first row reports the identifier of the video, including the eventual suffix "a" or "b". The second to the fifth columns report the voxel size (dx,dy,dz) and the time interval (dt). The second row specifies in which channel (Ch) cells appear in the video. From the third to the last row, the coordinates of cells are saved. The first column represents the unique identifier of a track, not varying for the entire track duration. The second to fifth columns (x, y, z, t) represent the position of the cell with respect to the top-up-left most corner of the z-stack at a specific time point. Coordinates are expressed in μm while the time point is an integer number. with two cell populations tracked it is either "a" or "b". These suffixes correspond to the suffixes used in (Table 2 (available online only)). The synthetic example with ID SQUARE was added to the dataset. This provides a test-case for software having different coordinate systems. A parallelogram of 5 × 5 × 10um is positioned in the first frame close to the origin used for LTDB videos, corresponding to the bottom(x = 0), left(y = 0), deepest (z = 0) corner of the 3D volume. This parallelogram migrates along the y-axis.
In order to evaluate tracking performances we provide a Matlab script LTDB2CTC.m to export LTDB tracks as the acyclic oriented graph representation 16 used in the Cell Tracking Challenge described in 14,15 . This allows the usage of the accurate methodology and software provided by the aforementioned authors to compare computed tracks vs. ground truth.
For detecting cell populations visible in more than one channel (Table 2 (available online only) and Table 3) we encourage the usage of a co-localization method based on supervised machine learning such as Ilastik 23 or Trainable Weka Segmentation 24 .
For discriminative machine learning models, it is worth noticing that all the cells of the videos LTDB001 to LTDB020 which are expected to be visible in the indicated channels were tracked. Other objects such as background, cell debris or additional cell populations were not tracked.
In the context of big-data analysis, (Fig. 3c) LTDB represents a resource to compare the biological properties of tracks (i.e. speed, directionality) amongst different experimental conditions. A review of the possible measures that could be computed from the tracks is provided in 10 .
The SQL database ltdb.sql can be installed optionally and for instance using the MySQL database management system. Queries to retrieve videos of interest (i.e. associated to a specific challenge, type of cell or site of imaging) can be addressed to the locally installed database. Additionally, a web interface was set up to facilitate search, preview and download of videos and it is accessible at http://www.ltdb.info/