# Computational cytometer based on magnetically modulated coherent imaging and deep learning

## Abstract

Detecting rare cells within blood has numerous applications in disease diagnostics. Existing rare cell detection techniques are typically hindered by their high cost and low throughput. Here, we present a computational cytometer based on magnetically modulated lensless speckle imaging, which introduces oscillatory motion to the magnetic-bead-conjugated rare cells of interest through a periodic magnetic force and uses lensless time-resolved holographic speckle imaging to rapidly detect the target cells in three dimensions (3D). In addition to using cell-specific antibodies to magnetically label target cells, detection specificity is further enhanced through a deep-learning-based classifier that is based on a densely connected pseudo-3D convolutional neural network (P3D CNN), which automatically detects rare cells of interest based on their spatio-temporal features under a controlled magnetic force. To demonstrate the performance of this technique, we built a high-throughput, compact and cost-effective prototype for detecting MCF7 cancer cells spiked in whole blood samples. Through serial dilution experiments, we quantified the limit of detection (LoD) as 10 cells per millilitre of whole blood, which could be further improved through multiplexing parallel imaging channels within the same instrument. This compact, cost-effective and high-throughput computational cytometer can potentially be used for rare cell detection and quantification in bodily fluids for a variety of biomedical applications.

## Introduction

Rare cell detection aims to identify a sufficient number of low-abundance cells within a vast majority of background cells, which typically requires the processing of large volumes of biological sample. The detection and enumeration of these rare cells are vital for disease diagnostics, the evaluation of disease progression and the characterization of immune response1,2,3. For instance, circulating foetal cells present in maternal blood are recognized as a source of foetal genomic DNA, and their isolation is crucial for the implementation of routine prenatal diagnostic testing4. As another example, antigen-specific T cells in peripheral blood play a central role in mediating immune response and the formation of immunological memory, which could lead to the prediction of immune protection and diagnosis of immune-related diseases5. Circulating endothelial cells with a mature phenotype are increased in patients with certain types of cancer and several pathological conditions, indicating their potential as disease markers6. Circulating tumour cells (CTCs) are implicated in various stages of cancer, and have therefore been collected to study their role in the metastatic cascade and to predict patient outcomes from both the disease and treatments received7. To highlight yet another example, haematopoietic stem and progenitor cells, which reside predominantly in bone marrow with low numbers, also found in peripheral blood, possess the unique capacity for self-renewal and multilineage differentiation, and their trafficking in blood may be connected to disease processes8.

The specific and sensitive detection of these rare cells in human blood and other bodily fluids is therefore of great interest. However, millions of events need to be acquired to obtain a sufficient number of these low-abundance cells (e.g., typically <1000 target cells per millilitre of blood9). The direct detection of rare cells from whole blood requires the processing of large amounts of patient sample (e.g., up to a few hundred millilitres10), which is both unrealistic and time consuming. To alleviate this issue, highly specific labelling methods are often used before detection for sample purification/enrichment to facilitate rapid detection and processing5,10. Among these labelling techniques, the use of colloidal magnetic particles as labelling reagents offers benefits in forming stable suspensions, fast reaction kinetics10 and minimum damage to the target cells, with high viability retained11.

Motivated by these important needs and the associated challenges, various technologies have been developed and employed for detecting rare cells in blood. Most of these existing detection methods involve three steps: capture, enrichment and detection12. The capture and enrichment steps use a number of methods, such as barcoded particles13, magnetic beads14, micro-machines15, microfluidic chips16 and density gradient centrifugation12,17. Following the enrichment step, these rare cells can be detected via commonly used techniques, such as immunofluorescence18,19, electrical impedance20 and Raman scattering21 measurements, among others. Notably, commercial products for rare cell detection, such as the CellSearch system22, which automates magnetic labelling, isolation, fluorescence labelling and automated counting, are generally high cost, limiting their adoption worldwide12. Therefore, cost-effective, reliable and high-throughput rare cell detection techniques are urgently needed to improve the early diagnosis of diseases, including cancer, so that earlier treatments can be carried out, helping us to improve patient outcomes while also reducing healthcare costs23,24.

The recent advances in machine learning and, specifically, deep learning have pushed the frontiers of biomedical imaging and image analysis25,26,27,28,29,30,31,32,33,34,35,36,37,38, enabling rapid and accurate pathogen detection39,40,41,42 and computer-assisted diagnostic methods43,44,45,46,47. Powered by deep learning, we demonstrate here that speckle imaging using lensless chip-scale microscopy can be employed for the specific and sensitive detection of rare cells in blood with low cost and high throughput. This novel cell detection and cytometry technique are based on magnetically modulated lensless speckle imaging, which specifically labels rare cells of interest using magnetic particles attached to surface markers of interest and generates periodic and well-controlled motion on target cells by alternating the external magnetic field applied to a large sample volume. The holographic diffraction and the resulting speckle patterns of the moving cells are then captured using a compact and cost-effective on-chip lensless imager (Fig. 1), and are computationally analysed by a deep-learning-based algorithm to rapidly detect and accurately identify the rare cells of interest in a high-throughput manner based on their unique spatio-temporal features. Although previous work has employed the idea of using magnetic modulation for enhancing fluorescence detection48,49, our work is the first of its kind for combining magnetic modulation, lensless imaging and deep learning to create a unique cytometer that does not require additional labelling (e.g., fluorescence) or custom-designed molecular probes.

As shown in Fig. 1, we built a portable prototype of this computational cytometer for rare cell detection. Our magnetically modulated speckle imaging module includes a lensless in-line holographic microscope41,50,51,52,53,54,55,56,57 and two oppositely positioned electromagnets (Fig. 1a, b inset). The lensless microscope contains a laser diode (650 nm wavelength) to illuminate the sample from ~5–10 cm above, and a complementary metal–oxide–semiconductor (CMOS) image sensor is placed ~1 mm below the sample for acquisition of a high-frame-rate video to monitor the spatiotemporal evolution of the sample containing the target cells of interest. Because the light-source-to-sample distance is much greater than the sample-to-image-sensor distance, the optical design has a unit magnification, and the field of view (FOV) of a single image is equal to the active area of the image sensor (which can be 10–30 mm2 using the standard CMOS imagers employed in digital cameras and mobile phones). To increase the screening throughput, target cells are enriched using magnetic separation and loaded inside a capillary tube for imaging (Figs. 1, 2). Magnetic enrichment alone leads to a background of unlabelled cells, bead clusters or weakly labelled cells that are also captured, such that further discrimination of the target cells within this background information is needed to accurately identify and count the rare cells. The imaging module is mounted onto a custom-made linear translation stage, and is translated along the direction of the sample tube to capture a holographic video for each section of the sample tube. During the imaging at each section, the electromagnets are supplied with alternating current with a 180° phase difference to exert an alternating pulling force to the magnetic-bead-conjugated cells in the sample, which causes them to oscillate at the same frequency as the driving current. Extension rods made of permalloy were designed and utilized to enhance the magnetic force at the sample location by ~40-fold (see the Methods section and Fig. S1). The holographic diffraction patterns that are cast by the magnetically modulated target cells are captured using the image sensor and transferred to a laptop computer. A computational motion analysis (CMA) algorithm41 and a densely connected pseudo-3D convolutional neural network structure (P3D CNN)58 then analyse the holographic image sequence that contains the 3D dynamic information from the oscillating cells, which allows rapid and specific detection of the target cells.

## Methods

### Cell preparation

MCF7 cell lines were purchased from ATCC (Manassas, Virginia, USA). Cells were plated with 10 mL of growth media in a T75 flask (Corning Inc., New York, USA) at a concentration of 1 × 105 cells/mL. The growth media was composed of Dulbecco’s Modified Eagle Medium (DMEM, Gibco®, Life Technologies, Carlsbad, California, USA) supplemented with 10% (v/v) foetal bovine serum (FBS, Gibco®, Life Technologies, Carlsbad, California, USA) and 1% penicillin–streptomycin (Sigma-Aldrich Co., St. Louis, Missouri, USA). Cells were grown in a humidified incubator at 37 °C in a 5% CO2 environment. Cells were harvested by treating them with 0.25% trypsin-EDTA (Gibco®, Life Technologies, Carlsbad, California, USA) for 3 min 2 to 3 days after seeding, depending on confluency. Then, the cells were pelleted by centrifuging for 3 min at 1200 RPM and resuspended in the growth media to a final concentration of 1 × 106 cells/mL.

### Sample preparation

Rare cell dilution: The MCF7 cells were serially diluted in Dulbecco’s phosphate-buffered saline (DPBS, Sigma-Aldrich Co., St. Louis, Missouri, USA) at different concentrations (2 × 104 cells/mL, 2 × 103 cells/mL, and 2 × 102 cells/mL). The dilution of MCF7 cells in whole blood was prepared by mixing the cell solution with whole blood at a ratio of 1:19 (v/v). Most of the experiments were performed by mixing 200 μL of cell solution with 3.8 mL of whole blood. Healthy human whole blood (from anonymous and existing samples) was obtained from the UCLA Blood and Platelet Center.

Bead washing: CELLection Epithelial Enrich Dynabeads (Invitrogen, Carlsbad, California, USA) were first resuspended in DPBS and vortexed for 30 s. A magnet (DX08B-N52, K&J Magnetics, Inc., Pipersville, Pennsylvania, USA) was then used to separate the Dynabeads, and the supernatant was discarded. This process was repeated three times, and the Dynabeads were resuspended in DPBS at the initial volume.

Rare cell separation: The washed Dynabeads were then added to the MCF7-spiked whole blood sample at a concentration of 2.5 μL beads per 1.0 mL of blood sample. The mixture was incubated for 30 min with gentle tilting and rotation. A magnet was placed under the vial for 5 min, and the supernatant was discarded after that. To this solution, we added 1 mL of cold DPBS buffer and mixed it gently by tilting from side to side. This magnetic separation procedure was repeated five times. After the final step, the sample was resuspended in 0.7 mL of DPBS and gently mixed with 2.5 mL of 400 cP methyl cellulose solution (Sigma-Aldrich Co., St. Louis, Missouri, USA) using a pipette. The sample was incubated for 5 min to reduce the number of bubbles before it was loaded into a glass capillary tube (Part # BRT 2-4-50; cross-section inner dimension of 2 mm × 4 mm; \$11.80 per foot; Friedrich & Dimmock, Inc., Millville, New Jersey, USA). The ends of the capillary tube were sealed with parafilm before the tube was mounted onto our computational cytometer for imaging and cell screening.

### Design of the computational cytometer based on magnetically modulated lensless speckle imaging

As shown in Fig. 1, our device hardware consists of an imaging module and a linear translation module. The imaging module, i.e., the scanning head in Fig. 1, contains a laser diode (650 nm wavelength, AML-N056-650001-01, Arima Lasers Corp., Taoyuan, Taiwan, China) for illumination, which has an output power of ~1 mW. The sample is loaded inside a capillary tube with a rectangular cross-section, which is placed ~7.6 cm below the light source. A CMOS image sensor (acA3800-14um, Basler, Ahrensburg, Germany) with a pixel size of 1.67 μm, which is placed below the glass tube with a narrow gap (~1 mm), is used to capture the holographic speckle patterns generated by the liquid sample. To induce oscillatory motion to the labelled cells in the sample, two electromagnets (Part #XRN-XP30 × 22, Xuan Rui Ning Co., Ltd., Leqing, Zhejiang Province, China) with custom-machined permalloy extensions are placed on either side of the glass tube. An alternating driving current (square wave) is supplied to either of the electromagnets, with a 180° phase shift between them, which creates alternative pulling force to the magnetic particles within the sample. The low level of the driving current is 0, and the high level of the driving current is ~500 mA. The frequency is 1 Hz, which was experimentally optimized to maximize the signal corresponding to the magnetic-bead-conjugated cancer cells.

The linear translation stage is custom-built using off-the-shelf components. A bipolar stepper motor (No. 324, Adafruit Industries LLC., New York, USA) with two timing pulleys and a timing belt is used to provide mechanical actuation, and the imaging module is guided by a pair of linear motion sliders and linear motion shafts on either side of the scanning head. 3D-printed plastic is used to construct the housing for the scanning head, and laser-cut acrylic is used to create the outer shell of the device.

### Image acquisition

After the sample is loaded into the capillary tube and placed onto our computational cytometer, the image acquisition procedure begins. The linear translation stage moves the scanning head to a series of discrete positions along the glass tube. At each position, the stage stops allowing the CMOS image sensor to capture a sequence of 120 holograms at a frame rate of 26.7 fps before moving onto the next position. The image data are saved to a solid-state drive (SSD) for storage and further processing.

Because the FOV corresponding to the edges (i.e., top and bottom rows) of the image sensor is subject to a highly unbalanced magnetic force field due to the closeness to one of the electromagnets, only the central 1374 rows of the image sensor’s pixels are used to capture the image sequence, where the magnetic force from the two electromagnets are relatively balanced.

Because the CMOS image sensor temperature quickly rises when it is turned on, it tends to cause undesired flow inside the glass tube due to convection. Therefore, a scanning pattern is engineered to reduce the local heating of the sample: if we denote 1, 2, …, 32 as the indices of the spatially adjacent scanning positions, the scanning pattern follows 1, 9, 17, 25, 2, 10, 18, 26, …. This scanning pattern ensures that a given part of the sample cools down before the scanning head moves back to its neighbourhood. The power to the image sensor is also cut-off during the transition between the two successive scanning positions, which is implemented by inserting a MOSFET-based switch into the power line of the USB cable.

### Computational detection and localization of cell candidates and deep-learning-based classification

The image processing procedure (Fig. 4) can be divided into two parts: (1) a preliminary screening step, which applies computational drift correction and MCF7 candidate detection to the entire FOV to locate target cell candidates in 2D, and (2) a classification step, which refocuses the holographic image sequence to each individual MCF7 candidate in its local area, generates an in-focus amplitude and phase video for each candidate, and classifies the corresponding video with a trained deep neural network. This procedure is further detailed below.

### Computational drift correction

The sample fluid in the glass capillary tube often drifts slowly throughout the duration of the image acquisition, which is due to, e.g., the imperfect sealing at the ends of the tube and the convection due to the heat of the image sensor. Because the detection and classification of the target cells are largely based on their periodic motion, the drifting problem should be corrected. Since our sample is embedded within a viscous methyl cellulose, minimal turbulent flow is observed and the drifting motion within our imaged FOV is almost purely translational. We used a phase correlation method73 to estimate the relative translation between each frame in the sequence with respect to a reference frame (chosen to be the middle frame in the holographic image sequence) and used 2D bilinear interpolation to remove the drift between frames. As shown in Fig. S2, this drift correction step successfully removed many false positive detections in the CMA step due to the background drift.

### Detection of target cell candidates

The detection of the target cell candidates plays a key role in automatically analysing the sample, as it greatly narrows down the search space for the rare cells of interest and allows the subsequent deep-learning-based classification to be applied to a limited number of holographic videos. In the preliminary screening stage, the lateral locations of the MCF7 candidates are detected. Each frame of the raw hologram sequence is propagated to various axial distances throughout the sample volume using a high-pass-filtered angular spectrum propagation kernel, which can be written as:

$${\mathbf{B}}_{i}\left( {z_j} \right) = {{HP}}\left[ {\mathcal{P}}\left( {{\mathbf{A}}}_{i},z_{j} \right) \right]$$
(1)

where HP(·) denotes the high-pass filter (see Supplementary Information for details), $${\mathcal{P}}$$ (·) denotes angular spectrum propagation59, Ai denotes the ith frame of the raw hologram sequence after the drift correction, and zj denotes the jth propagation (axial) distance. The selected propagation distances ranged from 800 μm to 5000 μm with a step size of 100 μm to ensure coverage of all possible MCF7 candidates within the sample tube. A zoomed-in image of Bi(zj) corresponding to an example region is shown in Fig. 4e.

Next, for every given propagation distance, a CMA algorithm is applied to reveal the oscillatory motion of the target cells within the sample, which focuses on periodic changes in the recorded frames:

$${\mathbf{C}}\left( {z_j} \right) = \frac{1}{{N_{\mathrm{F}} - N}}\mathop {\sum}\limits_{i = 1}^{N_{\mathrm{F}} - N} {\left( {\frac{1}{2}\left| {{\mathbf{B}}_{i}\left( {z_j} \right) - {\mathbf{B}}_{i + N/2}\left( {z_j} \right)} \right| +\, \frac{1}{2}\left| {{\mathbf{B}}_{i + N/2}\left( {z_j} \right) - {\mathbf{B}}_{i + N}\left( {z_j} \right)} \right| - \left| {{\mathbf{B}}_{i}\left( {z_j} \right) - {\mathbf{B}}_{i + N}\left( {z_j} \right)} \right|} \right)}$$
(2)

where C(z) and B(z) are shorthand notations for C(x, y; z) and B(x, y; z), respectively, NF is the total number of recorded frames (in our case, NF = 120), and N is chosen such that the time difference between the ith frame and the (i + N)th frame is equal to the period of the alternating magnetic field. Therefore, the first two terms inside the summation in Eq. (2) represent half-period movements at the jth propagation distance, and the last term represents the whole-period movement. Ideally, for objects that oscillate periodically with the alternating magnetic force field, the first two terms should be relatively large, and the last term should be relatively small. For randomly moving objects, the three terms in the summation approximately cancel each other out. As a result, C(x, y; z) is a 3D contrast map that has high values corresponding to the locations of periodic motion that matches the frequency of the external magnetic field. An example of C is shown in Fig. 4f.

To simplify segmentation, a maximum intensity projection along the axial direction (i.e., z) is applied to flatten the 3D image stack into a 2D image, which can be written as:

$${\mathbf{D}}\left( {x,y} \right) = \mathop {{{\mathrm{max}}}}\limits_z \left[ {{\mathbf{C}}\left( {x,y;z_1} \right),{\mathbf{C}}\left( {x,y;z_2} \right),...,{\mathbf{C}}\left( {x,y;z_{N_{\mathrm{H}}}} \right)} \right]$$
(3)

where x and y are the lateral indices and NH is the total number of axial positions (in our case, NH = 43). An example of D is shown in Fig. 4c, with a zoomed-in image shown in Fig. 4g. Thresholding-based segmentation was applied to the calculated 2D image D, and the resulting centroids are used as the lateral positions of the MCF7 candidates.

### Autofocusing and video generation

After the preliminary screening, which identifies the lateral centroids of potential target cell candidates, the subsequent processing is applied to each MCF7 candidate only within their local area. Autofocusing62,63 was first performed to locate the MCF7 candidate in the axial direction. Because C(x, y; zj) should have a higher value when approaching the in-focus position of each MCF7 candidate, the approximate axial position was obtained by maximizing (as a function of zj) the sum of the pixel values of C(x, y; zj) (j= 1, 2, …, NH) in a local neighbourhood around each individual MCF7 candidate. We chose to use a local neighbourhood size of 40 × 40 pixels (i.e., 66.8 μm × 66.8 μm). This process can be written as follows:

$$\hat z_k = \mathop {{{\mathrm{arg}}\,{\mathrm{max}}}}\limits_{z_j = 1,2,...,N_H} \mathop {\sum}\limits_{x,y = - 19}^{20} {{\mathbf{C}}\left( {x_k + x,y_k + y;z_j} \right)}$$
(4)

where $$\widehat z_k$$ is the resulting in-focus position for the kth potential target cell candidate and xk and yk are the lateral centroid coordinates of the kth potential target cell candidate.

The same criterion to find the focus plane can be applied again with finer axial resolution to obtain a more accurate estimation of the axial distance for each MCF7 candidate. We used a step size of 10 μm in this refined autofocusing step. Two examples of this process are shown in Fig. 4h. Alternatively, the Tamura coefficient62,63 could also be used as the autofocusing criterion to determine the in-focus plane.

Finally, the in-focus amplitude and phase video corresponding to each MCF7 candidate was generated by digitally propagating every frame of the drift-corrected hologram sequence to the candidate’s in-focus plane. The final video has 120 frames at 26.67 fps with both the amplitude and phase channels, and each frame has a size of 64 × 64 pixels (pixel size = 1.67 μm). Two examples corresponding to two cell candidates are shown in Fig. 4i.

### Target cell detection using densely connected P3D CNN

Each video of the MCF7 candidate was fed into a classification neural network (Fig. 6), which outputs the probability of having an MCF7 cell in the corresponding video (Fig. 4j, k). We designed a novel structure for the classification neural network, named densely connected P3D CNN, which is inspired by the Pseudo-3D Residual Network58 and the Densely Connected Convolutional Network60. The original P3D CNN58 used a mixture of three different designs of the P3D blocks to gain structural diversity, which resulted in a better performance. In this work, we introduced a densely connected structure to the P3D CNN structure by adding dense (skip) connections inside the spatio-temporal convolution block (dashed black arrows in Fig. 6 inset) to unify the three different P3D blocks. This allowed a simpler network design that was easier to implement for our task.

The detailed structure of the densely connected P3D CNN is shown in Fig. 6. The network contains five densely connected spatio-temporal convolutional blocks. As shown in the inset of Fig. 6, each block consists of a 1 × 3 × 3 spatial convolutional layer (Convs), a 3 × 1 × 1 temporal convolutional layer (Convt), followed by a max pooling layer (Max). Each spatial (or temporal) convolutional layer is a composition of three consecutive operations: batch normalization, a rectified linear unit (ReLU) and a spatial (or temporal) convolution (with stride = 1 and output channel number equal to the growth rate k= 8). In each block, we introduced skip connections between the input and output of the Convs layer as well as the Convt layer by concatenating () the input and the output in the channel dimensions. For a given input tensor mp, the densely connected spatio-temporal convolutional block maps it to the output tensor mp+1, which is given by:

$$m_{p + 1} = {\mathrm{Max}}\left[ {{\mathrm{Conv}}_t\left( {{\mathrm{Conv}}_s(m_p) \oplus m_p} \right) \oplus \left( {{\mathrm{Conv}}_s(m_p) \oplus m_p} \right)} \right]$$
(5)

For example, consider an input video with a size of c × t × h × w, where c, t, h and w denote the number of channels, number of frames (time), height and width of each frame (space), respectively. Here, c = 2, t = 120, and h = w = 64. We first pass the video through a 1 × 7 × 7 spatial convolutional layer (stride = 2) and a 9 × 1 × 1 temporal convolution layer (stride = 3) sequentially. The output channel numbers of the layers are included in Fig. 6 in each box. Then, the data go through five dense blocks, where between the 2nd and 3rd dense blocks, we add an additional 3 × 1 × 1 (stride = 1) convolutional filter with no padding to ensure that the time and space dimensions are equal. A fully connected (FC) layer with a 0.5 dropout rate and a softmax layer are introduced, which output the class probability (target rare cell or not) for the corresponding input video. Finally, a decision threshold is applied to the class probability output to generate the final positive/negative classification, where the decision threshold is tuned based on the training/validation data to reduce the FPR (detailed in the next sub-section “Network training and validation”).

### Network training and validation

We performed ten experiments (i.e., ten samples) to create the training/validation data sets for our classifier and then used the trained classifier to perform blind testing on additional serial dilution experimental data (Fig. 5), which had no overlap with the training/validation data. Among the ten experiments for constructing the training/validation data set, five were negative controls, and the other 5 were spiked whole-blood samples at an MCF7 concentration of 103 mL−1. When manually labelling the video clips to create the training/validation data set, we noticed that some videos were difficult to label, where the annotators could not make a confident distinction. Therefore, to ensure an optimal labelling accuracy, our negative training data came from only the five negative control experiments, where all the candidate videos from those experiments were used to construct the negative data set. The positive training data were manually labelled by two human annotators using five experiments spiked at 103 mL−1, where only the video clips that were labelled as positive with high confidence by both annotators were selected to enter the positive training data set, while all the others were discarded.

Next, the training/validation data sets were randomly partitioned into a training set and a validation set with no overlap between the two. The training set contained 1713 positive videos and 11324 negative videos. The validation set contained 788 positive videos and 3622 negative videos. The training data set was further augmented by randomly mirroring and rotating the frames by 90°, 180° and 270°. The convolutional layer weights were initialized using a truncated normal distribution, while the weights for the FC layer were initialized to zero. Trainable parameters were optimized using an adaptive moment estimation (Adam) optimizer with a learning rate of 10−4, and a batch size of 240. The network converged after ~800–1000 epochs. The network structure and hyperparameters were first optimized to achieve high sensitivity and specificity for the validation set. At a default decision threshold of 0.5, a sensitivity and specificity of 78.4% and 99.4%, respectively, were achieved for the validation set; a sensitivity and specificity of 77.3% and 99.5%, respectively, were achieved for the training set. After this initial step, because our rare cell detection application requires the classifier to have a very low FPR, we further tuned the decision threshold of our classifier to avoid false positives. For this, the training and validation data sets were combined to increase the total number of examples, and we gradually increased the decision threshold (for positive classification) from 0.5 while monitoring the FPR for the combined training/validation data set. We found that a decision threshold of 0.99999 was able to eliminate all false-positive detections in the combined training/validation data set. We further raised the decision threshold to 0.999999 to account for potential overfitting of the network to the training/validation data and further reduced the risk of false-positive detections.

At a decision threshold of 0.999999, as expected, the TPR dropped down to 10.5% (refer to Fig. S3, which reports the receiver operating characteristic (ROC) curve based on the validation data set, with an area under the curve of 0.9678). This low TPR results in underdetection of the target cells, as also evident in our serial dilution results (Fig. 5). The selection of the decision threshold is dependent on the specific application of interest and should be tuned based on the expected abundance of target cells and the desired LoD. For the application considered in this work, because the expected number of target cells at the lowest concentration (i.e., 10 mL−1) is extremely low, the decision threshold was tuned to a high level to suppress false positives, which in turn resulted in a very low TPR. However, for less demanding cell detection or cytometry applications where the desired LoD is not as stringent, the decision threshold may be relaxed to a lower level, which also allows the TPR to be higher.

### Computation time

Using our current computer code, which is not optimized, it takes ~80 s to preprocess the data within one FOV (corresponding to a volume of 14.7 mm2 × 2 mm) for extracting the MCF7 cell candidates, corresponding to the preliminary screening step in Fig. 4. For each detected cell candidate, it takes ~5.5 s to generate the input video for network classification. The network inference time for each input video is <0.01 s. Based on these numbers, if there are, e.g., ~1500 cell candidates per experiment, the total processing time using the current computer code would be ~3.0 h. However, we should note that the data-processing time depends on various factors, including the computer hardware configuration, the cell concentration in the sample, the programming language and whether the code is optimized for the hardware. In our work, although we used relatively high-performance hardware (an Intel Core i7 CPU, 64 GB of RAM and an Nvidia GeForce GTX 1080Ti GPU) and used some of the GPU functions provided by MATLAB (MathWorks, Natick, MA, USA), we did not extensively optimize our code for improved speed. A careful optimization of the GPU code should bring a significant speedup in our computation time.

### COMSOL simulation of the magnetic force field generated by the electromagnet and the permalloy extension

Because of space constraints, the electromagnet could not be placed sufficiently close to the imaging area, which caused the magnetic force to be low. We used a custom-machined extension rod made of permalloy74 (relative permeability μr ~ 100,000) to relay the force field and enhance the relative magnetic force on target cells by ~40 times. To simulate the magnetic force field distribution near an electromagnet with and without the permalloy extension, a finite element method (FEM) simulation was conducted using COMSOL Multiphysics (version 5.3, COMSOL AB, Stockholm, Sweden). A 3D model was developed using the magnetic field interface provided in the COMSOL AC/DC physics package. A stationary study was constructed based on the geometry of a commercially available electromagnet, where the core was modelled with a silicon steel cylinder (radius = 3 mm, height = 10 mm), and the coil was modelled with a surface current of 10 A/m on the side of the core running in the azimuthal direction. The permalloy extension was modelled using Permendur. A thick layer of air was added as a coaxial cylinder with a radius of 10 mm and a height of 30 mm. The magnetic flux density inside the simulation space was simulated using the magnetic field module. Then, a coefficient form PDE module in the mathematics library was used to derive the relative magnetic force field. The magnetic force that is received by superparamagnetic beads is given by:

$${\mathbf{F}} = \frac{{V\chi }}{{\mu _0}}\left( {{\mathbf{B}} \cdot \nabla } \right){\mathbf{B}}$$
(6)

where V is the volume of the magnetic particle, χ is the magnetic susceptibility, μ0 is the magnetic permeability in a vacuum and B is the magnetic flux density.

Our simulation results are shown in Fig. S1. The results in Fig. S1b indicate that the relative magnetic force rapidly reduces as a function of the distance from the electromagnet. However, by using a permalloy extension, the relative magnetic force at the sample location is enhanced by ~40 times.

## References

1. 1.

De Biasi, S. et al. Rare cells: focus on detection and clinical relevance. In Single Cell Analysis (eds Robinson, J. & Cossarizza, A.) 35–98 (Springer, Singapore, 2017).

2. 2.

Jindal, A. et al. Discovery of rare cells from voluminous single cell expression data. Nat. Commun. 9, 4719 (2018).

3. 3.

Arvaniti, E. & Claassen, M. Sensitive detection of rare disease-associated cell subsets via representation learning. Nat. Commun. 8, 14825 (2017).

4. 4.

Rezaei, M. et al. A reappraisal of circulating fetal cell noninvasive prenatal testing. Trends Biotechnol. 37, 632–644 (2019).

5. 5.

Bacher, P. & Scheffold, A. Flow-cytometric analysis of rare antigen-specific T cells. Cytom. Part A: J. Int. Soc. Anal. Cytol. 83A, 692–701 (2013).

6. 6.

Bertolini, F. et al. The multifaceted circulating endothelial cell in cancer: towards marker and target identification. Nat. Rev. Cancer 6, 835–845 (2006).

7. 7.

Lang, J. M., Casavant, B. P. & Beebe, D. J. Circulating tumor cells: getting more from less. Sci. Transl. Med. 4, 141ps13 (2012).

8. 8.

Massberg, S. et al. Immunosurveillance by hematopoietic progenitor cells trafficking through blood, lymph, and peripheral tissues. Cell 131, 994–1008 (2007).

9. 9.

Dharmasiri, U. et al. Microsystems for the capture of low-abundance cells. Annu. Rev. Anal. Chem. 3, 409–431 (2010).

10. 10.

Zborowski, M. & Chalmers, J. J. Rare cell separation and analysis by magnetic sorting. Anal. Chem. 83, 8050–8056 (2011).

11. 11.

Chen, Y. C. et al. Rare cell isolation and analysis in microfluidics. Lab a Chip 14, 626–645 (2014).

12. 12.

Shen, Z. Y., Wu, A. G. & Chen, X. Y. Current detection technologies for circulating tumor cells. Chem. Soc. Rev. 46, 2038–2056 (2017).

13. 13.

Zheng, F. Y. et al. Aptamer-functionalized barcode particles for the capture and detection of multiple types of circulating tumor cells. Adv. Mater. 26, 7333–7338 (2014).

14. 14.

Talasaz, A. H. et al. Isolating highly enriched populations of circulating epithelial cells and other rare cells from blood using a magnetic sweeper device. Proc. Natl Acad. Sci. USA 106, 3970–3975 (2009).

15. 15.

Balasubramanian, S. et al. Micromachine-enabled capture and isolation of cancer cells in complex media. Angew. Chem. Int. Ed. 50, 4161–4164 (2011).

16. 16.

Nagrath, S. et al. Isolation of rare circulating tumour cells in cancer patients by microchip technology. Nature 450, 1235–1239 (2007).

17. 17.

Park, J. M. et al. Highly efficient assay of circulating tumor cells by selective sedimentation with a density gradient medium and microfiltration from whole blood. Anal. Chem. 84, 7400–7407 (2012).

18. 18.

Xiong, K. et al. Biomimetic immuno-magnetosomes for high-performance enrichment of circulating tumor cells. Adv. Mater. 28, 7929–7935 (2016).

19. 19.

Lu, N. N. et al. Biotin-triggered decomposable immunomagnetic beads for capture and release of circulating tumor cells. ACS Appl. Mater. Interfaces 7, 8817–8826 (2015).

20. 20.

Han, S. I. & Han, K. H. Electrical detection method for circulating tumor cells using graphene nanoplates. Anal. Chem. 87, 10585–10592 (2015).

21. 21.

Sha, M. Y. et al. Surface-enhanced Raman scattering tags for rapid and homogeneous detection of circulating tumor cells in the presence of human whole blood. J. Am. Chem. Soc. 130, 17214–17215 (2008).

22. 22.

Wang, L. H. et al. Promise and limits of the CellSearch platform for evaluating pharmacodynamics in circulating tumor cells. Semin. Oncol. 43, 464–475 (2016).

23. 23.

Weller, D. et al. The Aarhus statement: improving design and reporting of studies on early cancer diagnosis. Br. J. Cancer 106, 1262–1267 (2012).

24. 24.

Etzioni, R. et al. The case for early detection. Nat. Rev. Cancer 3, 243–252 (2003).

25. 25.

Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).

26. 26.

Rivenson, Y. et al. Deep learning enhanced mobile-phone microscopy. ACS Photonics 5, 2354–2364 (2018).

27. 27.

Rivenson, Y. et al. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light.: Sci. Appl. 7, 17141 (2018).

28. 28.

Wu, Y. C. et al. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery. Optica 5, 704–710 (2018).

29. 29.

Liu, T. R. et al. Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 3926 (2019).

30. 30.

Rivenson, Y. et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light.: Sci. Appl. 8, 23 (2019).

31. 31.

Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3, 466–477 (2019).

32. 32.

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).

33. 33.

Wu, Y. C. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light.: Sci. Appl. 8, 25 (2019).

34. 34.

Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

35. 35.

Sinha, A. et al. Lensless computational imaging through deep learning. Optica 4, 1117–1125 (2017).

36. 36.

Strack, R. AI transforms image reconstruction. Nat. Methods 15, 309 (2018).

37. 37.

Li, Y. Z., Xue, Y. J. & Tian, L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica 5, 1181–1190 (2018).

38. 38.

Zhu, B. et al. Image reconstruction by domain-transform manifold learning. Nature 555, 487–492 (2018).

39. 39.

Gӧrӧcs, Z. et al. A deep learning-enabled portable imaging flow cytometer for cost-effective, high-throughput, and label-free analysis of natural water samples. Light.: Sci. Appl. 7, 66 (2018).

40. 40.

Wu, Y. C. et al. Label-free Bioaerosol sensing using mobile microscopy and deep learning. ACS Photonics 5, 4617–4627 (2018).

41. 41.

Zhang, Y. B. et al. Motility-based label-free detection of parasites in bodily fluids using holographic speckle analysis and deep learning. Light.: Sci. Appl. 7, 108 (2018).

42. 42.

Kim, G. et al. Rapid and label-free identification of individual bacterial pathogens exploiting three-dimensional quantitative phase imaging and deep learning. bioRxiv. https://doi.org/10.1101/596486. (2019).

43. 43.

Lakhani, P. & Sundaram, B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284, 574–582 (2017).

44. 44.

Ehteshami Bejnordi, B. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).

45. 45.

Lindsey, R. et al. Deep neural network improves fracture detection by clinicians. Proc. Natl Acad. Sci. USA 115, 11591–11596 (2018).

46. 46.

De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342–1350 (2018).

47. 47.

Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44–56 (2019).

48. 48.

Anker, J. N., Behrend, C. & Kopelman, R. Aspherical magnetically modulated optical nanoprobes (MagMOONs). J. Appl. Phys. 93, 6698–6700 (2003).

49. 49.

Anker, J. N. et al. Characterization and applications of modulated optical nanoprobes (MOONs). MRS Online Proc. Libr. 790, 4.1 (2003).

50. 50.

Greenbaum, A. et al. Wide-field computational imaging of pathology slides using lens-free on-chip microscopy. Sci. Transl. Med. 6, 267ra175 (2014).

51. 51.

Feizi, A. et al. Rapid, portable and cost-effective yeast cell viability and concentration analysis using lensfree on-chip microscopy and machine learning. Lab a Chip 16, 4350–4358 (2016).

52. 52.

Luo, W. et al. Synthetic aperture-based on-chip microscopy. Light.: Sci. Appl. 4, e261 (2015).

53. 53.

Greenbaum, A. et al. Increased space-bandwidth product in pixel super-resolved lensfree on-chip microscopy. Sci. Rep. 3, 1717 (2013).

54. 54.

Su, T. W., Xue, L. & Ozcan, A. High-throughput lensfree 3D tracking of human sperms reveals rare statistics of helical trajectories. Proc. Natl Acad. Sci. USA 109, 16018–16022 (2012).

55. 55.

Greenbaum, A. & Ozcan, A. Maskless imaging of dense samples using pixel super-resolution based multi-height lensfree on-chip microscopy. Opt. Express 20, 3129–3143 (2012).

56. 56.

Luo, W. et al. Pixel super-resolution using wavelength scanning. Light.: Sci. Appl. 5, e16060 (2016).

57. 57.

Zhang, Y. B. et al. Wide-field imaging of birefringent synovial fluid crystals using lens-free polarized microscopy for gout diagnosis. Sci. Rep. 6, 28793 (2016).

58. 58.

Qiu, Z. F., Yao, T. & Mei, T. Learning spatio-temporal representation with pseudo-3d residual networks. In Proc. 2017 IEEE International Conference on Computer Vision. 5533–5541 (IEEE, Venice, Italy, 2017).

59. 59.

Goodman, J. W. Introduction to Fourier Optics, 3rd edn. (Roberts and Company Publishers, Englewood, Colorado, 2005).

60. 60.

Huang, G. et al. Densely connected convolutional networks. In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition. 4700–4708 (IEEE, Honolulu, HI, 2017).

61. 61.

Ye, H. et al. Evaluating two-stream CNN for video classification. In Proc. 5th ACM on International Conference on Multimedia Retrieval. 435–442, https://dl.acm.org/citation.cfm?id=2749406 (ACM, Shanghai, China, 2015).

62. 62.

Zhang, Y. B. et al. Edge sparsity criterion for robust holographic autofocusing. Opt. Lett. 42, 3824–3827 (2017).

63. 63.

Tamamitsu, M. et al. Comparison of Gini index and Tamura coefficient for holographic autofocusing based on the edge sparsity of the complex optical wavefront. arXiv preprint arXiv:1708.08055 (2017).

64. 64.

Hajian-Tilaki, K. Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Casp. J. Intern. Med. 4, 627–635 (2013).

65. 65.

Krause, S. et al. The microenvironment determines the breast cancer cells’ phenotype: organization of MCF7 cells in 3D cultures. BMC Cancer 10, 263 (2010).

66. 66.

Fatsis-Kavalopoulos, N. et al. Formation of precisely composed cancer cell clusters using a cell assembly generator (CAGE) for studying paracrine signaling at single-cell resolution. Lab a Chip 19, 1071–1081 (2019).

67. 67.

Zhao, M. X. et al. An automated high-throughput counting method for screening circulating tumor cells in peripheral blood. Anal. Chem. 85, 2465–2471 (2013).

68. 68.

Wu, X. X. et al. Improved SERS-active nanoparticles with various shapes for CTC detection without enrichment process with supersensitivity and high specificity. ACS Appl. Mater. Interfaces 8, 19928–19938 (2016).

69. 69.

Balsam, J., Bruck, H. A. & Rasooly, A. Cell streak imaging cytometry for rare cell detection. Biosens. Bioelectron. 64, 154–160 (2015).

70. 70.

Issadore, D. et al. Ultrasensitive clinical enumeration of rare cells ex vivo using a micro-hall detector. Sci. Transl. Med. 4, 141ra92 (2012).

71. 71.

Liu, W. et al. Rare cell chemiluminescence detection based on aptamer-specific capture in microfluidic channels. Biosens. Bioelectron. 28, 438–442 (2011).

72. 72.

Gao, T. et al. DNA-oriented shaping of cell features for the detection of rare disseminated tumor cells. Anal. Chem. 91, 1126–1132 (2019).

73. 73.

Reddy, B. S. & Chatterji, B. N. An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process. 5, 1266–1271 (1996).

74. 74.

Jiles, D. Introduction to Magnetism and Magnetic Materials. 2nd edn. (Boca Raton: Chapman and Hall, 1998).

## Acknowledgements

The Ozcan Research Group at UCLA acknowledges the support of the Koç Group, NSF Engineering Research Center (ERC, PATHS-UP), the Army Research Office (ARO; W911NF-13-1-0419 and W911NF-13-1-0197), the ARO Life Sciences Division, the National Science Foundation (NSF) CBET Division Biophotonics Program, the NSF INSPIRE Award, NSF Partnerships for Innovation: Building Innovation Capacity (PFI:BIC) Program, the National Institutes of Health (NIH, R21EB023115), the Howard Hughes Medical Institute (HHMI), the Vodafone Americas Foundation, the Mary Kay Foundation, and the Steven & Alexandra Cohen Foundation. We also thank Yunzhe Qiu for his help with the deep neural networks.

## Author information

Authors

### Contributions

Y.Z. and A.O. designed and conceived this project; Y.Z., B.B., Z.D., X.L. and S.Y. designed and built the prototype; M.O., J.K., D.K. and D.K. cultured the MCF7 cells; Y.Z., M.O., A.R., J.K., D.K., K.T., D.K., H.C.K. and C.C. prepared the samples; Y.Z., A.R., T.L. and A.F. performed the experiments; Y.Z. and T.L. developed the image-processing algorithm; Y.Z., T.L. and B.B. processed the data; Y.Z. and B.B. built the deep neural network; T.L. and B.B. labelled the training data for the neural network; Y.L. and A.G. performed the COMSOL simulations; A.O., D.D.C. and O.B.G. supervised the project; and all authors contributed to and commented on the paper.

### Corresponding author

Correspondence to Aydogan Ozcan.

## Ethics declarations

### Conflict of interest

Y.Z., A.R., D.D.C. and A.O. have a pending patent application on the presented computational cytometer.

## Rights and permissions

Reprints and Permissions

Zhang, Y., Ouyang, M., Ray, A. et al. Computational cytometer based on magnetically modulated coherent imaging and deep learning. Light Sci Appl 8, 91 (2019). https://doi.org/10.1038/s41377-019-0203-5

• Revised:

• Accepted:

• Published:

• ### Automatic detection and characterization of quantitative phase images of thalassemic red blood cells using a mask region-based convolutional neural network

• Yang-Hsien Lin
• , Ken Y.-K. Liao
•  & Kung-Bin Sung

Journal of Biomedical Optics (2020)

• ### A neural network approach for real-time particle/cell characterization in microfluidic impedance cytometry

• , John S. McGrath
• , Riccardo Reale
• , Paolo Bisegna
• , Nathan S. Swami
•  & Frederica Caselli

Analytical and Bioanalytical Chemistry (2020)

• ### Optobiology: live cells in optics and photonics

• Lisa Miccio
• , Pasquale Memmolo
• , Francesco Merola
• , Martina Mugnano
•  & Pietro Ferraro

Journal of Physics: Photonics (2020)

• ### Detection of Bacterial and Viral Pathogens Using Photonic Point-of-Care Devices

• Peuli Nath
• , Alamgir Kabir
• , Somaiyeh Khoubafarin Doust
• , Zachary Joseph Kreais
•  & Aniruddha Ray

Diagnostics (2020)

• ### Perspectives on liquid biopsy for label‐free detection of “circulating tumor cells” through intelligent lab‐on‐chips

• Lisa Miccio
• , Flora Cimmino
• , Ivana Kurelac
• , Massimiliano M. Villone
• , Vittorio Bianco
• , Pasquale Memmolo
• , Francesco Merola
• , Martina Mugnano
• , Mario Capasso
• , Achille Iolascon
• , Pier Luca Maffettone
•  & Pietro Ferraro

View (2020)