Wrinkle force microscopy: a new machine learning based approach to predict cell mechanics from images

Combining experiments with artificial intelligence algorithms, we propose a new machine learning based approach to extract the cellular force distributions from the microscope images. The full process can be divided into three steps. First, we culture the cells on a special substrate allowing to measure both the cellular traction force on the substrate and the corresponding substrate wrinkles simultaneously. The cellular forces are obtained using the traction force microscopy (TFM), at the same time that cell-generated contractile forces wrinkle their underlying substrate. Second, the wrinkle positions are extracted from the microscope images. Third, we train the machine learning system with GAN (generative adversarial network) by using sets of corresponding two images, the traction field and the input images (raw microscope images or extracted wrinkle images), as the training data. The network understands the way to convert the input images of the substrate wrinkles to the traction distribution from the training. After sufficient training, the network is utilized to predict the cellular forces just from the input images. Our system provides a powerful tool to evaluate the cellular forces efficiently because the forces can be predicted just by observing the cells under the microscope, which is a way simpler method compared to the TFM experiment. Additionally, the machine learning based approach presented here has the profound potential for being applied to diverse cellular assays for studying mechanobiology of cells. Significance Statement Cell-generated forces are indispensable determinants of fundamental cell functions such as motility and cell division. As such, quantifying how the forces change upon perturbations to the cells such as gene mutations and drug administration is of profound importance. Here we present a novel machine learning based system that allows for efficient estimations of the forces that are determined only by “observing” microscope images. Given that the cellular traction forces are regulated downstream of diverse signaling pathways, our system – that helps significantly improve the throughput of the measurements – presents a new, high throughput platform for real time analysis of the effects of a massive number of genetic and molecular perturbations on the forces and resulting cell mechanics.


Combining experiments with artificial intelligence algorithms, we
propose a new machine learning based approach to extract the cellular force distributions from the microscope images.The full process can be divided into three steps.First, we culture the cells on a special substrate allowing to measure both the cellular traction force on the substrate and the corresponding substrate wrinkles simultaneously.The cellular forces are obtained using the traction force microscopy (TFM), at the same time that cell-generated contractile forces wrinkle their underlying substrate.Second, the wrinkle positions are extracted from the microscope images.Third, we train the machine learning system with GAN (generative adversarial network) by using sets of corresponding two images, the traction field and the input images (raw microscope images or extracted wrinkle images), as the training data.The network understands the way to convert the input images of the substrate wrinkles to the traction distribution from the training.After sufficient training, the network is utilized to predict the cellular forces just from the input images.Our system provides a powerful tool to evaluate the cellular forces efficiently because the forces can be predicted just by observing the cells under the microscope, which is a way simpler method compared to the TFM experiment.Additionally, the machine learning based approach presented here has the profound potential for being applied to diverse cellular assays for studying mechanobiology of cells.
cell mechanics | traction force microscopy | GAN T here is now growing evidence showing that cells sense mechanical cues in the surrounding microenvironment to regulate their functions such as proliferation, differentiation, apoptosis, and pro-inflammation (1)(2)(3)(4)(5)(6).In response to the mechanical cues, cells often adjust their cytoskeletonal tension and as such many of the mechanical information are translated into a level of inherent cellular traction forces, and in turn into intracellular signals regulating the related functions (3,(7)(8)(9).Traction forces, thus related to various cell functions, are generated by the activity of nonmuscle myosin II and actin filaments that determine cellular contractility (2,(10)(11)(12).Because these proteins work downstream of diverse signaling pathways, it is often difficult to predict how the force may change upon perturbations to particular molecules such as gene mutations and drug administration.Thus, technologies allowing for efficiently evaluating the cellular traction force are expected to appear to enhance comprehensive understanding of the force-related pathways.
We previously developed a wrinkle assay, a modified version of the method originally reported by Harris and colleagues (13,14), in which the silicone substrates are spatially treated with uniform oxygen plasma to allow them to buckle upon the forces exerted by cells (15)(16)(17).As the individual wrinkles are lengthened with the increase in the forces (18), the wrinkle length, detected for example by a machine learning approach (19), can be used as a measure of the relative change in the force caused by perturbations such as specific gene mutations.While this technology is promising in that these experiments are performed easily to potentially enable a highthroughput analysis on the force-related pathways, the interpretation of the wrinkle length was not necessarily straightforward in terms of quantitatively measuring the magnitude and direction of traction forces.
To overcome this limitation in quantification, here we describe a new machine learning system that converts the wrinkle information taken by a microscope into the actual cellular force distributions.For the initial training data, the cellular traction forces are obtained using the traction force microscopy (TFM), and we train the machine learning system with GAN (generative adversarial network) so that the network understands the way to convert the input microscope images to the force distributions from the training data.After sufficient training, the network can be utilized to predict the cellular forces just from the input images.The system would be a powerful tool

Significance Statement
Cell-generated forces are indispensable determinants of fundamental cell functions such as motility and cell division.As such, quantifying how the forces change upon perturbations to the cells such as gene mutations and drug administration is of profound importance.Here we present a novel machine learning based system that allows for efficient estimations of the forces that are determined only by "observing" microscope images.Given that the cellular traction forces are regulated downstream of diverse signaling pathways, our system -that helps significantly improve the throughput of the measurements -presents a new, high throughput platform for real time analysis of the effects of a massive number of genetic and molecular perturbations on the forces and resulting cell mechanics.to evaluate the cellular forces efficiently because the forces can be predicted just by observing the cells, which is a way simpler method compared to the TFM experiment.

Full picture of the system
Our goal is to construct a machine learning system that can predict the cellular force distributions from the microscope image or the extracted substrate wrinkles.The full process can be divided into three steps as shown in Fig. 1.First, we culture the cells (A7r5; embryonic rat vascular smooth muscle cells) on a silicone membrane substrate and measure both the cellular traction force and the substrate wrinkles simultaneously.As shown in Fig. 1(a), the cellular traction forces are obtained using TFM (20,21), and cells generate wrinkles because the surface of PDMS (polydimethyl siloxane) layer is hardened by the plasma irradiation (16,19,(22)(23)(24).Second, the wrinkle positions are extracted from the microscope images as shown in Fig. 1(b) by using our SW-UNet (small world U-Net) (19), which is a convolutional neural network (CNN) that reflects the concept of the small world networks (25,26).Third, the machine learning system utilizing GAN ( 27) is trained to understand a way to convert the microscope image, or the extracted wrinkle image, to the cellular force distributions as shown in Fig. 1(c).After the training, the network can be utilized to predict the cellular forces just from the microscope images.

Simultaneous measurement of wrinkles and traction forces
Before applying the machine learning system, we begin by considering the results of the simultaneous force and wrinkle characterization.Figure 2 summarizes representative results obtained by the experiment and analysis.Due to the pairwise inward pulling generated by cellular traction force (fourth column), the substrates exhibit displacements toward the cell center (third column).As the result of the contraction, the wrinkles emerge mostly underneath the cells (second column).
When the cell size is small, the majority of wrinkles are aligned in a same direction as in Fig. 2(a), while they tend to point in different directions when the cell size is large and the traction is strong as in Fig. 2(c).Figure 3(a) shows the probability distribution function (PDF) of the traction magnitude of N × M samples, where N = 103 is the number of the images and M = 26 × 26 is the number of the force observation points.The average traction is 50.3 ± 57.1 [Pa] (mean ± standard deviation).Figure 3(b) shows that the wrinkle length has a positive correlation with the mean traction of the images, which is in agreement with our previous experimental measurements (22), where the relationship between the wrinkle length and applied force was experimentally investigated using microneedles.The mean traction is simply obtained by averaging the norm of the traction of the image as where m is the index of the observation points.The wrinkle length is measured by counting the number of pixels after skeletonizing the wrinkle images (22).The wrinkle extincts when the mean traction in a image is less than 10 Pa, which is comparable to the noise level or the resolution of the current TFM.In order to analyze the principal direction of the traction, we construct a symmetric stress tensor for each image as

D R A F T
where r = xm − x0 is the relative vector from the image center x0 and n = r/|r| is the normal vector.By diagonalizing the tensor, we obtain the principal direction of the traction φs (shown in Fig. 2, second column with green lines) together with the corresponding principal traction magnitude fp, from the eigenvalue that has the largest norm.At the same time, we obtain the principal direction of the wrinkles φw (also shown in Fig. 2, second column with blue lines) from the 2D-FFT (fast Fourier transform) image of the wrinkles: φw is an angle that is perpendicular to the direction that has a largest power spectrum.Figure 3(c) shows that the traction force is contractile (fp < 0) and is almost linearly related to the wrinkle length (correlation R = −0.82 in a range fp < −5 Pa).Using the relation between the traction and the wrinkles, the wrinkles can be used for one qualitative marker or indicator for rough estimation of the cellular traction magni-tude.Figure 3(d) shows that the two angles φs and φw are perpendicular most of the time.Since the wrinkle direction is perpendicular to that of the force dipoles, the wrinkle would be also practical to qualitatively predict the force directions, as previously done elsewhere (16,18).Therefore, the length and direction of wrinkles provide a qualitative measure of the magnitude and direction of forces exerted by cells on the substrate, respectively.Next, we employ the machine learning based approach to provide a quantitative measure of the forces from the microscope images of wrinkles.

Traction force prediction using GAN
Finally, we train the network and evaluate the performance of the force estimation using our GAN network.Figure 4 compares the predicted force distributions which were estimated by the three different methods.As also shown in Fig. S1, we trained the network with two different input images, the raw microscope images (second column in Fig. 4) and the extracted wrinkle images (third column), to compare the performances.We also evaluated the force distribution using a standard encoder-decoder type CNN and show the results in the fourth column.The figure shows that all the three methods reproduce approximately the same force direction as the ground truth (first column), and the forces are perpendicular to the wrinkles.Figure 5(a) compares the traction of ground truth f true x,y and GAN prediction f predict x,y (input image: microscope images), and it shows that the prediction is highly correlated with the experimental data.The correlation coefficient R is evaluated for all 15 test images and averaged correlations, 0.86-0.88for GAN and 0.83-0.84for CNN as shown in Fig. 5(b), suggest that there are striking agreements.In order to further quantify the error in the force estimation, we introduce two errors: the error in the force magnitude ε f and the force direction ε θ .The error ε f is defined to evaluate the difference in the force magnitude between the ground truth f true and the prediction f predict as: where M = 26 × 26 is the number of observation points, ωm = f true m / f is the weight function and f is the average force in a image which is defined in Eq. (1).Note that we introduce this weight function in order to put weight on the evaluation of large vectors rather than small vectors, which give huge errors even for small differences.We used N = 332 training image sets and 3 test images for the evaluation.The total error is calculated by averaging the error of 15 test images, which are obtained by repeating the evaluation 5 times with randomly selected different test images.Figure 5(c) shows that the error is 33-35% for GAN, and it has better performance compared to the encoder-decoder type simple CNN, which has an error 46%.There is no significant difference by the two input images (microscope images and wrinkle images), and this result indicates that performance of the force estimation would not improve drastically by explicitly teaching the wrinkle position to the machine learning system.Next, we evaluate the angle difference between the predicted force and the ground truth as where θ = arctan(fy/fx) is the force direction.to demonstrate the novel concept of cellular force detection from microscope images.These errors will be minimized by increasing the number of the training data.As demonstrated above, our system succeeded in estimating the force distribution just from the input images with limited levels of errors, in real time.Movies S1-S4 further show the application of the proposed system in providing high throughout, real time measure of the traction force distributions during dynamic cell locomotion.

Discussion
We proposed a new machine learning based system that can predict the cellular force distributions from the microscope images.The full process can be divided into three steps.First, we culture the cells on a plasma-irradiated silocone substrate and measure both the cellular traction force and the substrate wrinkles simultaneously.The cellular traction forces are obtained using the TFM, while cells generate wrinkles on the underlying substrates.Second, the wrinkle positions are extracted from the microscope images by using SW-UNet.Third, we train the GAN system by using sets of corresponding two images, the force distributions and the input images (raw microscope images or extracted wrinkle images), as the training data.The network understands the way to convert the input images to the force distributions from the training.After sufficient training, the network can be utilized to predict the cellular forces just from the input images.Comparing with the TFM experiment (test data), the prediction using our system is highly correlated with the experimental data, with the averaged correlation coefficient of 0.86-0.88 and with 33-35% errors in the force magnitude prediction and angle errors 19-20 • in the force direction.We expect that this error would decrease further by increasing the number of training images.
The system would be a powerful tool to evaluate the cellular forces efficiently because the forces can be predicted just by observing the cells, which is a way simpler method compared to performing the TFM experiment every time needed.
TFM is one of the most used methods to evaluate the cellular forces in mechanobiology study, but as the accuracy of the measurement depends on successful acquisition of the reference positions of the micro-beads that are obtained by removing the cells after each of the experiments in conventional TFM, this method is limited in throughput.The novel GANbased system proposed here overcomes this limitation as it provides the nearly same information, with the high levels of the correlations with the experimental data and the limited levels of the errors, on the cell mechanics only from the still images that are acquired just by plating the cells on the silicone substrate without taking care of the reference as the substrate surface is known to become planar again upon the absence of the cellular forces in a reversible manner.Given that early stages of drug screening require testing a massive number of candidate compounds (23), our system with the potentially high-throughput data analysis capability will be useful particularly in such screening studies.It is important to note that our new system is not the one that essentially competes with TFM, but the huge advantage of the proposed system is focused on its capability to provide data equivalent to the TFM (with a level of the errors) and thereby circumvent performing the TFM that needs considerable technical care.Rather, because the machine learning system depends on the training data, further innovations in TFM such as superresolution imaging (28,29) are potentially introduced to our system to synergetically output more sophisticated data.Thus, our approach presents a versatile framework that integrates the sophisticated experimental techniques and the efficient measurements.

Materials and Methods
Step 1: Simultaneous measurement of traction forces and wrinkles.
Based on our previous studies (16,19,(22)(23)(24), we prepared the substrate that can reversibly generate wrinkles upon application of cellular forces.Firstly, a circular cover glass is treated with oxygen plasma (SEDE-GE, Meiwafosis) to hydrophilize the surface and is desiccated after fluorescent micro-beads (0.2 µm in diameter, carboxylate yellow-green fluorescent beads; Invitrogen) in water solution are distributed on the surface.Secondly, parts A and B of CY 52-276 (Dow Corning Toray) are mixed at a weight ratio of 1.2:1 and poured onto the cover glass to create a PDMS layer with a height of 30-40 µm.Thirdly, the cover glass is placed in a 60°C oven for 20 hours to cure the PDMS.Fourthly, oxygen plasma is applied uniformly along the surface of the PDMS layer to create an oxide layer that works as the substrate for cell culture.Finally, the substrate is coated with 10 µg/mL collagen type I solution for 3 hours.For the TFM measurement, fluorescent micro-beads are attached to the substrate surface as position markers to measure the substrate deformations.The beads need to be firmly adhered to the surface so that cells would not move the beads due to endocytosis.In this work, the covalent bonding between the surface and the beads of 0.001% v/v are performed by following two steps: (i) silane coupling of the substrate surface using 3-Aminopropyltrimethoxysilane and (ii) the covalent bonding formation due to carbodiimide.The beads adhered on the glass surface are monitored to keep the reference position even after removing the cell using 0.25% Trypsin (Trypsin + 1mm mmol/I EDTA-4Na solution; Fuji Wako Pure Chemical Corporation).Traction force microscopy (TFM).The software ImageJ/Fiji and its plugin FTTC (Fourier transform traction cytometry) (30,31) are used to evaluate the force field from the displacement field.The substrate is considered as a soft elastic isotropic material that follows the linear elastic theory.First, the displacement of the substrate surface u is measured by tracking the movement of the fluorescent beads using PIV (particle image velocimetry).Second, the traction field is obtained from the displacement field by solving the governing equation for the elastic halfspace (32,33) given by u(x) = S G(x, y)t(y)dS(y) [5] where t is the traction force, x and y are the positions of the displacement and the traction force, respectively.G is the Green's function that is given by

Cell
where E is the Young's modulus, ν is the Poisson's ratio, r = (rx, ry) = x − y is the relative position vector and r = |r|.The software FFTC solves Eq. ( 5) in the Fourier space, which is given by t where tilde symbols denote the variables in Fourier space, λ is the regularization parameter (33) and I is the unit tensor.In order to evaluate the optimal parameter λ for the Tikhonov regularization, the L-curve criterion (33,34) is applied.Note that E is experimentally determined ( 16) to be 5400 Pa and ν is assumed 0.5 (incompressible) that is a typical value for PDMS material.
Step2: Wrinkle extraction.We use a new method SW-UNet (19), which is a CNN based on U-Net (35) to extract wrinkle patterns from the microscope image as shown in Fig. 1(b).As the training data, we prepare 236 sets of corresponding two images (microscope image and manually labeled wrinkle image).The number of data is increased to 2596 by using the image augmentation techniques.We used NVIDIA Titan RTX to accelerate the training process, and the Adam optimizer is utilized.
Step 3: Prediction of traction force based on GAN-based system.Assume that we have No sets of corresponding images and data; the input images x (microscope images, or extracted wrinkle images) and the force distributions y as shown in Fig. S1.We effectively have the number of training data set 2No because the wrinkle image has only 1D information at each pixel (intensity I(x, y); see also images in Fig. S1) while the force distributions have 2D information (2D force, f (x, y) = {fx(x, y), fy(x, y)}).We designed the network to evaluate the cellular force only for a single axis at one time and focus only on the x-directional force at each evaluation.An input image I i is used as the training data set (I i , fx) and (I i , fy) where I i is an image that rotates I i by 90 degrees.
The force distributions are converted to gray scale images that have intensities where a = 81.GAN structure.Our goal is to convert a physical quantity (wrinkle geometry) to another physical quantity (force distribution).Even though the mechanical formulation between the two quantities is given, it is not necessarily straightforward to solve this inverse problem because of its complex nonlinear dynamics (36,37).Instead, we achieve this purpose by training our machine learning system to understand the underlying mechanical rules.Considering this conversion of the physical quantity as an image "translation", we utilize GAN (generative adversarial network) (38) in this work.GAN mainly consists of two networks, generator G and discriminator D as shown in Fig. S1, and the network is trained by a competition of two networks.Goal of the generator G is to generate fake images (fake force distributions G(x)) from the input images x (microscope images or extracted wrinkle images) and tries to mimic the real images y (real force distributions), while the discriminator D tries to distinguish true and fake images from the group of images.As the training proceeds, the generator learns how to produce fake images that are difficult to be distinguished by the discriminator from the real images, and the discriminator learns the rules to distinguish true/fake images.Once the training is completed, the trained generator G can now be used as the translator to predict the force distribution from the input images x even for test images, which were not included in the training process.
In present work, we design the generator G with a form of encoder-decoder which is based on U-Net (35) but without the copy-and-crop path, and Markovian discriminator (PatchGAN) (39) is utilized as the discriminator D. The generator G converts the input images x to the fake force distribution images G(x).The images of force distribution, y or G(x), and the input images x are concatenated into a single image as the input of discriminator.The two networks G and D are trained based on the labels of real/fake, and we utilize the loss function L that is used in pix2pix ( 27 [9] where E is the expected value, L 1 (G) is the L1 distance between generated images G(x) and ground truths y and λ = 100 is the weight for the L1 term.The first term E[log D] denotes the expected probability that the discriminator categorizes y as the real data, while the second term E[log(1 − D(x, G(x)))] denotes the probability that the discriminator categorizes the generated image G(x) as the fake data.The goal of the generator G is to minimize L while the discriminator D tries to maximize it.
We use the training parameters as follows: 100 training epochs (batch size = 1), ε = 0.0002 learning rate for generator and discriminator, the parameters β 1 = 0.5 and β 2 = 0.9 are used for the Adam optimizer.The whole learning process is again accelerated by Nvidia Titan RTX.

Fig. 1 .
Fig. 1.Overview of the methods and procedures that are utilized in the wrinkle-force microscopy (WFM).(a) Schematic of our experimental setup.A silicone membrane, which can evaluate the cellular force distribution (obtained by TFM) and the surface wrinkles simultaneously, is utilized in this work.(b) The surface wrinkles are extracted from the microscope images by using our machine learning system (SW-UNet).(c) The machine learning system (GAN) is trained to understand the relation between the input images (raw microscope image, or extracted wrinkle images) and corresponding output images (cellular force distribution).(d) After sufficient training, the system can predict the force distributions only from the microscope images.

Fig. 2 .
Fig. 2. Three examples of the simultaneous measurement of wrinkles and traction forces.Each column describes (from left to right) raw image, wrinkles (red lines), displacement field and traction force field.The white scale bar in the first column images is 20 µm.The blue and green lines inside the second column images describe the principal direction of the wrinkle and the traction, respectively.

Fig. 3 .
Fig. 3. Quantitative analysis of the traction forces and wrinkles.(a) Probability distribution function (PDF) of the traction magnitude.(b) The wrinkle length as the function of the mean traction f .Note that the wrinkle length is evaluated by counting the number of pixels after skeletonizing the wrinkle images.(c) The wrinkle length as the function of the principal traction fp.(d) Probability distribution function of the angle differences between the wrinkle direction φw and the traction φs.The figure suggests that the direction of the wrinkles is predominantly perpendicular to the principal direction of the force.

Figure 5 Fig. 4 .
Fig. 4. Prediction of the traction forces from microscope images.Each column shows the result of the traction force predictions by using different methods (from left to right): ground truth, GAN prediction (input: microscope image), GAN prediction (input: extracted wrinkle images), and CNN prediction (input: microscope image).See further examples in Movies S1-S4.

2 GANFig. 5 .
Fig. 5. (a) Comparison of the ground truth f true x,y and the predicted traction f predict x,y .Dashed line shows a condition f predict = f true .(b) Correlation coefficient R between f true x,y and f predict x,y .(c)-(d) Errors of the predicted traction compared to the ground truth data: (c) error in the traction magnitude ε f and (d) the traction direction ε θ .Blue squares are the average value and black circles denote the outliers.Note that m.s.denotes the microscope images.
culture and microscope setup.A7r5 cells were maintained at 37°C in a stage incubator (INUF-IX3W; Tokai Hit) under a humidified 5% CO 2 incubator.An inverted microscope (1X73; Olympus) with a conforcal unit (CSU10; Yokokawa Electric) and oil immersion lens (phase contrast, UPlanFLN 60x/1.25 Oil Iris Ph3, Olympus Corporation) are used to capture the cells and fluorescent beads.During the experiment, DMEM(L)+10% FBS+Penicillin-Streptomycin (Fuji Wako Pure Chemical Corporation) is used as D R A F T the culture medium.
2, b = 50.0,I mid = 255/2 are the coefficients for the conversions, and f d is the components of the force d = x, y.The force distributions in gray scale, which are generated from the test images, can be converted back to the force using this equation.As the training data, we prepared N = 332 sets (83 original images) of corresponding two images.Note that we increased the number of training data by rotating the images, 83 × 4 = 332.