Three phase-contrast time-lapse microscopy image datasets were obtained independently. Each dataset consisting of four culture conditions – Control (growth media), 50 ng/mL FGF2, 100 ng/mL BMP2, and 50 ng/mL FGF2 + 100 ng/mL BMP2. Four replicates were imaged per dataset at a frequency of 5 minute intervals over approximately 3.5 days. Each image sequence was compiled and verified using the FiJi distribution of ImageJ 1.51n11,12. Computer-aided cell tracking annotations were produced using our previously published cell tracking system which consists of segmentation6, mitosis detection7, and association8–10 modules. Ground truth annotations were produced by trained personnel and subsequently curated by another annotator with at least five years of experience with mammalian cell culture. Such data can be used to generate cell lineage trees.