Protocol | Published:

A practical guide to intelligent image-activated cell sorting

Abstract

Intelligent image-activated cell sorting (iIACS) is a machine-intelligence technology that performs real-time intelligent image-based sorting of single cells with high throughput. iIACS extends beyond the capabilities of fluorescence-activated cell sorting (FACS) from fluorescence intensity profiles of cells to multidimensional images, thereby enabling high-content sorting of cells or cell clusters with unique spatial chemical and morphological traits. Therefore, iIACS serves as an integral part of holistic single-cell analysis by enabling direct links between population-level analysis (flow cytometry), cell-level analysis (microscopy), and gene-level analysis (sequencing). Specifically, iIACS is based on a seamless integration of high-throughput cell microscopy (e.g., multicolor fluorescence imaging, bright-field imaging), cell focusing, cell sorting, and deep learning on a hybrid software–hardware data management infrastructure, enabling real-time automated operation for data acquisition, data processing, intelligent decision making, and actuation. Here, we provide a practical guide to iIACS that describes how to design, build, characterize, and use an iIACS machine. The guide includes the consideration of several important design parameters, such as throughput, sensitivity, dynamic range, image quality, sort purity, and sort yield; the development and integration of optical, microfluidic, electrical, computational, and mechanical components; and the characterization and practical usage of the integrated system. Assuming that all components are readily available, a team of several researchers experienced in optics, electronics, digital signal processing, microfluidics, mechatronics, and flow cytometry can complete this protocol in ~3 months.

Introduction

iIACS1 is a machine-intelligence technology that can be used to perform real-time continuous image-based sorting of single cells with high throughput. iIACS is based on a seamless integration of high-throughput optical microscopy (e.g., multicolor fluorescence imaging, bright-field imaging)2,3, cell focusing4,5,6, cell sorting7, and deep learning8,9,10,11 on a hybrid software–hardware data management infrastructure, enabling real-time automated operation for data acquisition, data processing, intelligent decision making, and actuation. iIACS extends beyond the capability of FACS12,13,14,15,16,17,18,19 from fluorescence intensity profiles of cells to multidimensional images. This enables sorting of cells or cell clusters with unique spatial, biochemical, and morphological traits that are difficult to discern when compressing these spatial data into a 1D FACS signal. Sorting decisions can be made by gating on image features, which allows substantial flexibility and the potential for identifying new cell populations, just as extended multicolor data have enabled discovery of new immune cell subsets with FACS12,13,14,15,16,17,18,19. The large amount of image data collected using iIACS can also be used for training and implementing an artificial neural network such as a deep convolutional neural network (CNN)8,10,11, which can act to rapidly sort out a known cell population with unique morphological features in new samples. By virtue of these unique features, iIACS is expected to address one of the most fundamental biological questions that cannot be answered by FACS: how are the composition, structure, and morphology of cells linked to their function? Therefore, iIACS serves as an integral part of holistic single-cell analysis by enabling direct links between population-level analysis (flow cytometry), cell-level analysis (microscopy), and gene-level analysis (sequencing) of sorted cells. Given the ability to access the new dimension of image data, we anticipate that iIACS will serve as an essential tool moving forward to unlock a diverse range of biological, pharmaceutical, and medical applications.

Development of the approach

The realization of iIACS was made possible by using a radically different data management infrastructure to overcome the fundamental trade-off between speed and accuracy, a constraint that limits the performance of virtually all sensing technologies20,21. Specifically, the trade-off relation exists between the volume and complexity of image data (which are correlated to accuracy) and the data transfer and image-processing speed (which are correlated to response time). In fact, FACS12,13,14,15,16,17,18,19can handle only low-resolution data (e.g., ~20 light scattering and fluorescence signals without spatial information) for real-time data processing and actuation (i.e., sorting). On the other hand, digital analysis of high-resolution data (i.e., images) in image-based high-content screening22,23,24,25,26,27,28,29,30,31and imaging flow cytometry32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 is too slow to perform real-time actuation because of the limited speed of data transfer and image processing. A typical strategy for increasing the data-processing throughput is to process the data in parallel by using multiple computers, but this is generally limited to ‘off-line’ operation, resulting in a long turnaround time (more than several seconds), and does not allow real-time actuation. We developed iIACS to overcome this trade-off relation by integrating high-throughput optical microscopy2,3, cell focusing4,5,6, and cell sorting7 on a hybrid software–hardware data-management infrastructure that runs a telecom-grade 10-Gbps all-Internet-protocol (IP) architecture59,60,61. Key challenges that were overcome include the ability to rapidly process image data without long data transfer times, the ability to predict the location of flowing cells for sorting with high accuracy to account for longer sort decision times than with traditional FACS, and the ability to send data between multiple components of the system with minimal delay. Consequently, this optical–microfluidic–electrical–computational–mechanical system enables high flexibility; high scalability; and automated operation for data acquisition, data processing, decision making, and actuation, all within 32 ms, even with deep-learning algorithms.

Overview of the procedure

In this protocol, we provide a practical guide to iIACS that describes how to design, build, characterize, and use an iIACS machine. Specifically, as shown in Fig. 1, the protocol provides recipe-like instructions about how to design an iIACS machine (Steps 1–12); construct essential elements of the iIACS machine, including a liquid pump (Steps 15–20 and 118–120), a microfluidic chip (Steps 21–39), a cell focuser (Steps 40–46 and 123–128), a microscope (Steps 47–71 and 148–165), a speed meter (Steps 72–82 and 135–147), monitoring optics (Steps 83–88, 121, and 122), an image processor (Steps 89–107 and 190–197), a neural network (Steps 180–189), and a cell sorter (Steps 40–46 and 129–134); integrate these components into the iIACS machine (Steps 112–117); characterize its performance (Steps 204–218); and use it for high-content sorting experiments in various settings (Steps 219–226). The protocol is designed for any potential team of researchers with interdisciplinary knowledge interested in performing iIACS for high-throughput intelligent image-based isolation of cells from heterogeneous cell populations. Note that although the original iIACS machine was composed of several unique components such as a frequency-division-multiplexed (FDM) fluorescence microscope2and a dual-membrane push–pull cell sorter7, iIACS is not limited to this specific configuration and can, in principle, be constructed by utilizing other elements that have similar specifications (e.g., other types of high-speed microscopes42,49,52, computers, and sorting modules).

Fig. 1: Overview of the procedure.
figure1

As a practical guide to iIACS, this protocol provides recipe-like instructions on how to design, build, characterize, and use an iIACS machine. Image adapted from Nitta et al.1.

Applications of the method

As with all transitions of data to a new geometric dimension, we anticipate the applications of iIACS will be manifold, with many uses difficult to even predict. By virtue of its ability to provide spatial resolution, iIACS is expected to address the fundamental biological question of how the spatial architectures of cells are connected to their physiological functions. For example, the localization of transcription factors to the nucleus or cytoplasm is known to cause dramatic differences in cellular behavior62. Cellular morphology, such as size and shape, has a strong influence on intracellular signaling, cellular growth, and cellular differentiation63. Nuclear shape, nucleus-to-cytoplasm ratio, and aggregation are an important set of signatures for cancer detection in cytology22,23,64,65,66,67,68,69,70. The localization of Ca2+-binding proteins in chloroplasts for photosynthesis varies with environmental CO2 conditions71,72. Abnormal morphological traits of the budding yeast Saccharomyces cerevisiae are used as a gene feature for high-dimensional morphological phenotyping and many functional genomic applications73,74,75. The cytoskeletal organization of adherent cells and localization of intracellular organelles appear to be connected to function76,77 and could be accessed using iIACS in combination with a microfluidics-compatible adherent cell microcarrier technology78. A range of other morphological features, such as chromosome number, compaction, and location; RNA localization; and lipid droplet distribution are probably linked to unique cellular functions but have not yet been fully investigated because of the lack of appropriate tools. Beyond single cells, iIACS should be compatible with cell clusters and organoids, in which the spatial structure of multiple interacting cell types can define function79, and analysis and growth of purified subsets can enable new understanding. For example, investigation of immune cells forming linkages or ‘synapses’ with antigen-presenting cells or cells targeted for killing can benefit from an understanding of how the geometry of the synapse is linked to function80,81. Sorting of rare Caenorhabditis elegans or Drosophila melanogaster embryo mutants across development on the basis of cellular localization can uncover developmental defects and genetic underpinnings of tissue morphogenesis82,83. Among these potential applications, we demonstrated real-time high-content sorting of microalgal and blood cells on the basis of intracellular protein localization and cell–cell interaction from large heterogeneous populations for studying photosynthesis and atherothrombosis, respectively1. As evidenced by these intriguing applications, iIACS holds promise for helping researchers tackle various biological problems.

Advantages and limitations of the method

The primary advantage of iIACS is its ability to sort target cells on the basis of intelligently analyzed multicolor fluorescence and bright-field images of numerous single cells within a short period of time. Here, the information content provided by iIACS (i.e., the number of pixels multiplied by the number of colors) is ~1,000 times greater than that by FACS (i.e., ~20 colors without spatial resolution). Consequently, it substantially speeds up highly inefficient, time-consuming, labor-intensive microscopy-based cell identification and isolation processes by several orders of magnitude, opening up a window onto a new class of biological, pharmaceutical, and medical applications. In fact, iIACS was used to demonstrate high-content sorting of ~2,000 Chlamydomonas reinhardtii mutants with aberrant protein localization out of 0.2 million cells within only 40 min1, which would take ~6 months by conventional methods. Likewise, it also enabled high-content sorting of 255 platelet aggregates from ~6,000 human blood cells within only 1 min1, which would take ~1 d by conventional methods. Currently, iIACS is optimized for analyzing individual cells and is not able to handle larger biological objects such as cell spheroids, organoids, tissue fragments, or whole organisms, but with proper modifications, iIACS should be available to those objects. There is also a current limitation to sort at rates of ~100 events (e.g., single cells, cell clusters, cell debris) per second (e.p.s.), which is sufficient for many applications but may be a bottleneck when trying to accumulate a large number of rare cells that are present in a sample (e.g., at frequencies <0.01%). In comparison, the throughput of conventional FACS machines is 5,000–10,000 e.p.s. This limitation is not based on the optical imaging approach, microfluidic focusing, or sorting throughput, but rather on the computational analysis pipeline, which we envision will continue to improve as computational power continues to advance. Another limitation is that there are currently no commercial iIACS systems available (although there is a startup expected to commercialize the technology within a few years), meaning that its implementation and operation require a relatively specialized skill set (see ‘Level of expertise needed to implement the protocol’).

Comparison with other methods

iIACS is unique in the fact that it unites microscopy, cell sorting, and deep learning and currently has no similar counterparts. It may, however, be compared with FACS12,13,14,15,16,17,18,19, image-based high-content screening22,23,24,25,26,27,28,29,30,31, imaging flow cytometry32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58, and imaging-based cell picking84,85,86,87,88,89,90,91,92,93. As mentioned above, FACS can handle only low-resolution information (e.g., ~20 light scattering and fluorescence signals without spatial information) for real-time data processing and sorting and cannot provide high-resolution information such as optical images. This means that FACS is not capable of addressing how the spatial distribution of the biomolecules within the cells is connected to their physiological functions. Image-based high-content screening and imaging flow cytometry are too slow in digital image analysis to perform real-time sorting because of the limited speed of data transfer and image processing. Imaging-based cell picking uses an image sensor to monitor the activity of numerous cells (>10,000) and a robotic arm to pick up target cells. If the number of target cells is very small, the technology is effective, but if it is large (>10% of the total population), then the technology is not effective because the response and actuation speed of the robotic arm is low (>10 s per cell, corresponding to ~0.1 e.p.s.).

Level of expertise needed to implement the protocol

iIACS requires expertise in the areas of optics, electronics, digital signal processing, microfluidics, mechatronics, and flow cytometry (Table 1). On the basis of our experience, a graduate student or postdoctoral researcher well trained in optics can set up an FDM microscope and speed meter. A graduate student or postdoctoral researcher well trained in microfluidics can fabricate and install a liquid pump, cell focuser, and cell sorter. A graduate student or postdoctoral researcher well trained in digital signal processing and machine learning can develop and run an image processor. A senior researcher with interdisciplinary expertise to oversee the system design, construction, and characterization is required as a leader. A team of such researchers can integrate these components into an iIACS machine and characterize its performance. A few graduate students or postdoctoral researchers can operate the iIACS machine for demonstrating applications. Because the required expertise is interdisciplinary, frequent communication is necessary for successful construction and operation of an iIACS machine.

Table 1 Level of expertise needed to implement the protocol

Experimental design

System design (Steps 1–12)

The iIACS machine consists of a liquid pump, a cell focuser, a microscope, a speed meter, an image processor, and a cell sorter. Although the actual implementation of these elements is flexible, with many degrees of freedom, our recently developed iIACS machine1 (Figs. 2 and 3) is built on a single microfluidic chip as a platform into which the cell focusing and sorting implementation and the optical interrogation of the microscope and speed meter are all integrated. This is because the single microfluidic chip provides higher cell viability and less contamination in comparison with capillary-based systems. For designing an iIACS machine, it is important to consider several important factors such as throughput, sensitivity, dynamic range, image quality (e.g., spatial resolution, number of pixels, number of colors), sort purity, and sort yield, but throughput is typically the primary determinant that subsequently constrains the other parameters and, hence, needs to be considered first. Once the desired specifications are set, each component needs to be designed and constructed before integrating all the elements into a system.

Fig. 2: Schematic of the iIACS machine.
figure2

The iIACS machine consists of (i) a liquid pump (not shown in the figure), (ii) a cell focuser, (iii) a microscope, (iv) a speed meter, (v) an image processor, and (vi) a cell sorter. The corresponding human operation for the same process (although several orders of magnitude slower than the iIACS machine) is shown at the bottom. BF, bright field; PD, photodetector. Image adapted from Nitta et al.1.

Fig. 3: The iIACS machine.
figure3

The inset shows an enlarged picture of the optics–microfluidics integration unit.

The functionality of our iIACS machine is described as follows (Figs. 2 and 3). First, a sample tube that contains suspended cells is injected into the microfluidic chip by a liquid pump composed of pressure vessels, electro-pneumatic regulators, and liquid flow meters that provides the optimum flow rate ratio between the sample and horizontal and vertical sheath flows for 3D hydrodynamic focusing (Figs. 46). Second, a two-step 3D on-chip hydrodynamic focuser that employs two sheath flows confines flowing cells both horizontally and vertically so that a highly stable 3D focused cell flow at a flow speed of 1 m/s is generated for blur-free image acquisition with the FDM microscope2 (Figs. 5 and 7). Third, the FDM microscope is used to capture an image of each single cell flowing in the microchannel (Figs. 8 and 9). Specifically, a linear array of multiple intensity-modulated excitation beams with different modulation frequencies is focused across the microchannel to produce a linear array of focal spots in a direction orthogonal to the flow. Each flowing cell is scanned in the flow direction by the beam array to acquire a series of 1D bright-field and multi-color fluorescence spatial profiles of the cell with multiple single-pixel photodetectors, from which 2D images of the cell are constructed by stacking the 1D spatial profiles obtained by demodulating the detected photodetector signals at the modulation frequencies. Fourth, a speed meter is used to detect forward-scattered light from each flowing cell at the FDM microscope to predict the optimum sort timing by estimating the sort latency at the sort point (Fig. 10). The speed measurement and sort timing prediction are important elements of the iIACS machine for sorting with high purity; these elements are not commonly used in FACS, because with FACS it takes only a few hundred microseconds for a cell to flow from the optical interrogation point to the sort point; the sort latency of the iIACS machine is >30 ms (~100 times longer than that of FACS) because of its requirement for digitally analyzing much larger amounts of data (images) on an image processor (Fig. 11). Fifth, as the iIACS machine requires a long computation time of >30 ms, a 3D on-chip acoustic cell focuser is used to keep flowing cells at the center of the microchannel from the FDM microscope to the sort point (~32 mm) (Figs. 5 and 7). This acoustic focusing is implemented by exciting both the horizontal and vertical acoustic resonance modes of the square-shaped cross-section of the microchannel with piezoelectric transducers in order to prevent flowing cells from shifting laterally due to inertial lift and gravitational forces, which would otherwise lead to fluctuations in the cell arrival time at the sort point. Sixth, an on-chip dual-membrane push–pull cell sorter is used to produce a local flow in the direction orthogonal to the flow to rapidly kick target cells from the central stream of cells by deforming dual glass membranes with piezoelectric actuators7 (Fig. 12). With the actuation off, cells flow into the central branch of the three-branch microchannel junction, which is connected to a waste tube. With the actuation on at the right timing, cells flow into either the upper or lower branch of the microchannel, which leads to a collection tube. Finally, collection and waste tubes that contain the sorted and unsorted cells, respectively, are removed from the iIACS machine.

Fig. 4: Schematic of the liquid pump.
figure4

The liquid pump is composed of pressure vessels, electro-pneumatic regulators, and liquid flow meters and provides the optimum flow rate ratio between the sample, horizontal sheath, and vertical sheath flows for 3D hydrodynamic focusing. DAC, digital-to-analog converter; GPIO, general-purpose input/output; SPI, serial peripheral interface.

Fig. 5: Schematic of the microfluidic chip.
figure5

A single microfluidic chip performs three functions: (bottom left) two-step 3D hydrodynamic focusing for producing a single stream of cells for continuous blur-free image acquisition with the FDM microscope, (bottom center) acoustic focusing for maintaining the single stream during the computation of the image processor, and (bottom right) multichannel sorting for isolating target cells from the cell stream. Image adapted from Nitta et al.1.

Fig. 6: Construction of the microfluidic chip.
figure6

a, Top view of the microfluidic chip. b, Fabrication process of the microfluidic chip. First, grating-shaped channels are etched by the reactive-ion etching (RIE) process on 200-µm-thick borosilicate glass substrates for the cover and base layers (Step 23). Then, 100-µm-deep × 400-µm-wide microchannels are fabricated on the cover and base layers by the wet-etching procedure, using masks made by Cr and Au sputtering and OFPR spin-coating (Steps 26 and 27). A 200-µm-thick silicon (Si) substrate for the microchannel layer is anodically bonded to the base layer (Step 31). Microchannel patterns, including microchannels, inlets, outlets, and dual-membrane pumps, are fabricated by the DRIE process using photographic masks made with SU-8 (Steps 32 and 33). The inlets and outlets are formed on the cover layer by sandblasting, using masks made with SCM250 (Steps 34 and 35). The cover layer and microchannel layer are anodically bonded (Step 37). Finally, a packaged microfluidic chip is obtained. Scale bar, 1 cm. c, Photographs of the fabrication process. Step 23: form a grating pattern. Step 24: sputter Cr and Au layers. Step 25: spin-coat positive photoresist. Step 27: pattern microchannels. Step 28: remove the etching masks. Step 31: bond the patterned base layer and the Si substrate. Step 32: spin-coat negative photoresist. Step 33: etch the Si layer. Step 34: laminate a sheet-type photoresist. Step 35: form the inlets and outlets. HF, hydrofluoric acid.

Fig. 7: Schematic and characterization of the cell focuser.
figure7

a, Schematic of the two-step 3D hydrodynamic focuser (main) and focusing efficiency as a function of sheath-to-sample volume ratios (inset). b, Focusing efficiency of the hydrodynamic focuser. The normalized speed is defined as the speed of a flowing event normalized by the speed of the fastest event. The flow rate ratio of sample:sheath1:sheath2 is indicated in the key. c, Sort latency versus normalized speed. The latency for the predicted sort time is defined by subtracting the actual arrival time of an event from its predicted sort time. d, Combined focusing efficiency of the hydrodynamic and acoustic focusers. b,c, and d adapted from Nitta et al.1.

Fig. 8: Schematic of the FDM microscope.
figure8

Left inset: schematic of the generation of the intensity-modulated excitation beam array by interfering deflected beams from the AODs (top) and the relation between the optical frequency shifts at the AODs and intensity-modulation frequencies (bottom). Middle inset: schematic of the excitation beam irradiation. Right inset: image construction process. The APD signals are digitized and sent to the IC node of the image processor. The fluorescence signals can be detected in the direction of the transmitted illumination or in the opposite direction, depending on the overall design of the iIACS machine. AP, anamorphic prism pair; HBS, half beam splitter; HWP, half-wave plate; ND, neutral density.

Fig. 9: Characterization of the FDM microscope.
figure9

a, Sensitivity, dynamic range, and linearity of the fluorescence detection channel FL1. Pink, orange, lime, green, blue, and purple histograms show a distribution of the signal intensities of the 6-peak fluorescent calibration particles. b, Sensitivity, dynamic range, and linearity of the fluorescence detection channel FL2. Pink, orange, lime, green, blue, and purple histograms show a distribution of the signal intensities of the 6-peak fluorescent calibration particles. c, Fluorochrome separation using standard compensation particles for FITC, PE, and PE-Cy5 and blank particles. Green, yellow, red, and blue dots represent the distribution of FITC, PE, PE-Cy5, and blank particles, respectively. d, Spatial resolution and calibration using size standard particles. Purple, blue, green, orange, and red histograms indicate particles of 2.0–2.4 µm, 3.0–3.4 µm, 5.0–5.9 µm, 7.0–7.9 µm, and 8.0–12.9 µm, respectively. a.u., arbitrary units. Image adapted from Nitta et al.1.

Fig. 10: Schematic and characterization of the speed meter.
figure10

a, Schematic of the speed meter. OI1 and OI3 are located adjacent to OI2, whereas OI4 is located slightly upstream of the cell sorter. b, Time-course plots of signals from the photodetectors at OI1 (magenta), OI3 (green), and OI4 (blue). Each pulse corresponds to the passage of a particle. The top, middle, and bottom plots show detected particles within a time range of 0–200 ms, 50–100 ms, and 56.5–60.5 ms, respectively. The three sequential peaks highlighted in the middle plot indicate that the latency between OI1 and OI4 is ~32.5 ms. The OI1 and OI3 peaks are enlarged in the bottom plot, indicating that the latency between OI1 and OI3 is ~0.36 ms. c, Latency between OI1 and OI3 versus the latency between OI1 and OI4, with a linear regression using 6-µm particles. The inset shows a histogram of events in regression residual. Image adapted from Nitta et al.1.

Fig. 11: Design of the image processor.
figure11

a, Schematic of the image processor. b, Flow chart of the implemented algorithm for the classic image analysis. c, Flow chart of the implemented algorithm with the deep CNN for the intelligent image analysis. BF, bright-field; ch, channel; CID, cell identifier; DAC, digital-to-analog converter; DRAM, dynamic random-access memory; Ether, Ethernet; FPGA, field-programmable gate array; GPU, graphics processing unit; HDD, hard disk drive; PCIe, peripheral component interconnect express; PD, photodetector; SM, speed meter; SSD, solid-state drive. a adapted from Nitta et al.1.

Fig. 12: Schematic and characterization of the cell sorter.
figure12

a, Schematic of the dual-membrane push–pull cell sorter. b, Demonstration of switching the local flow by using the on-chip dual-membrane pumps. Scale bar, 200 µm. a adapted from Nitta et al.1.

Liquid pump (Steps 15–20 and 118–120)

In our iIACS machine1, a liquid pump is used to control the flows into the microfluidic chip. Specifically, the liquid pump is composed of an air compressor, pressure vessels, electro-pneumatic regulators, gas and liquid valves, liquid flow meters, and a controller (Fig. 4); it can be divided into three parts: a pump for feeding the sample and two pumps for independently feeding sheath fluids for horizontal and vertical hydrodynamic focusing. These three pumps maintain the optimum flow rate ratio between the sample, horizontal sheath, and vertical sheath flows for 3D hydrodynamic focusing. The ratio is regulated by a feedback system, which starts with three liquid flow meters that measure the flow rates of the three liquid flows, followed by the controller (composed of a microcontroller and a computer) to evaluate the flow rates and control three electro-pneumatic regulators. As the primary purpose of the feedback system is to compensate for slow flow rate drifts due to temperature and water level fluctuations, subsecond time resolution is sufficient for the feedback control. Rapid fluctuations of the flow rates, such as those caused by pulsation from pumps or liquid dripping, should be carefully eliminated by selecting appropriate fluidic devices. The electric valves are useful for switching between fluidic operation modes, including run, stop, wash, and backflush, all of which are essential for practical cell sorting experiments.

Microfluidic chip (Steps 21–39)

In our iIACS machine1, a single microfluidic chip is used to conduct three functions (Fig. 5): (i) two-step 3D hydrodynamic focusing for producing a single stream of cells for continuous blur-free image acquisition with the FDM microscope, (ii) acoustic focusing for maintaining the single stream during the computation of the image processor, and (iii) multichannel sorting for isolating target cells from the cell stream. As shown in Figs. 5 and 6, the microfluidic chip consists of three layers: a base layer, a microchannel layer, and a cover layer (AutoCAD designs are available as Supplementary Data 1 of this protocol). To form rigid microchannels that avoid unwanted potential instability due to flow-induced deformation of microchannels in the chip, 200-µm-thick borosilicate glass substrates are used for the base and cover layers, whereas a 200-µm-thick silicon substrate is used for the microchannel layer. The use of such thin glass substrates is important for using a pair of objective lenses with high numerical apertures but for avoiding their direct contact with the microfluidic chip. The cross-sectional dimensions of the central microchannel, including the four optical interrogation points (OI1–OI4) and the sort point, are 200 µm × 200 µm. The details of the fabrication process are shown in Fig. 6.

Cell focuser (Steps 40–46 and 123–128)

The function of the hydrodynamic focuser is to tightly focus randomly injected cells into a single continuous stream at the center of the microchannel before arriving at OI1, so that the cells can be imaged at the same lateral position in the same focal plane of the FDM microscope (Figs. 5 and 7). In our iIACS machine1, a two-step 3D hydrodynamic focusing technique is used for this purpose. Specifically, the hydrodynamic focuser uses a sheath flow to confine flowing cells vertically and another sheath flow introduced perpendicularly to confine them horizontally. For stable 3D hydrodynamic focusing, the liquid pump provides the optimum flow rate ratio between the sample, the horizontal sheath (sheath1), and the vertical sheath (sheath2) flows. In our iIACS machine1, a flow rate ratio of sample/sheath1/sheath2 = 1:16:40 is used to focus cells to the center of the microchannel (Fig. 7). The tight focusing is also important for minimizing the effect of the parabolic flow speed distribution, ensuring a predictable sort latency.

The function of the acoustic focuser is to maintain the single stream of cells at the center of the microchannel all the way between the optical interrogation point of the FDM microscope and the sort point (Fig. 5). In our iIACS machine1, this distance is ~32 mm, which is unusually long in the field of flow cytometry. Because the flow speed distribution in a microchannel is parabolically shaped, acoustic focusing serves to prevent flowing cells from drifting laterally and arriving at OI4 inconsistently because of inertial lift and gravitational forces (this is especially evident for large cells). The acoustic focuser consists of a pair of piezoelectric transducers and a 0.57-mm-thick 20 mm × 20-mm element glued to both of the glass substrates of the microfluidic chip with an epoxy resin. Acoustic focusing is accomplished by exciting both vertical and horizontal resonance modes of the 200 × 200-µm cross-section of the microchannel by actuating the piezoelectric transducers with a sinusoidal driving signal at 3.66 MHz and 142 Vpp (peak-to-peak voltage), which is provided by a function generator via a high-voltage amplifier.

Microscope (Steps 47–71)

Among all the elements of the iIACS machine, the microscope is one of the most important parts. The requirement for the microscope is its ability to obtain high-resolution multicolor fluorescence images of cells flowing at a high speed (>1 m/s) in a continuous manner without motion artifacts. In addition to the fluorescence imaging capability, bright-field imaging is also important to providing morphological information of cells. In our iIACS machine1, the FDM microscope is used to meet the requirements, as standard charge-coupled device (CCD) and complementary metal-oxide semiconductor (CMOS) image sensors fail to do. The FDM microscope is based on integration of a broadband, spatially distributed, dual-frequency comb or spatial dual-comb into frequency-division multiplexing with single-pixel photodetection (Fig. 8).

Although the FDM microscope has several variants, the configuration of the FDM microscope used in our iIACS machine1 is described as follows (Fig. 8). A 488-nm continuous-wave laser beam is converted by a pair of acousto-optic deflectors (AODs) excited by multitone electrical signals into multiple intensity-modulated excitation beams with distinct modulation frequencies and propagation directions. By using a beam expander (a pair of achromatic doublet lenses), these beams are expanded and imaged on the back aperture of an objective lens to form a linear array of focal spots across the microchannel so that cells pass through the beam array perpendicularly. Consequently, each cell is scanned by the beam array to produce a series of 1D spatial profiles of the transmitted excitation beams and fluorescence signals from the cell. The transmitted excitation beams and two-color fluorescence signals (508–580 nm (FL1) and >580 nm (FL2)) are separately detected by three avalanche photodetectors (APDs). The frequency range of the excitation beam array is 4–200 MHz. The signal waveform of each APD is digitized at a sampling rate of 1.25 gigasamples (GS)/s. The 1D spatial profiles contained in the digitized signal waveforms are separated in the frequency domain by taking their Fourier transform, as each beam spot has a distinct modulation frequency. Finally, three-color images of each cell are obtained by stacking the 1D spatial profiles in the flow direction. As a result, the FDM microscope obtains bright-field and fluorescence images of cells flowing at 1 m/s with a pixel size of 0.25 µm (in the flow direction) × 0.84 µm (in the spot array direction) and a 1D field of view of 42 µm in the beam array direction. The total signal-processing time for individual events differs, depending on the cell–cell interval in the flow.

Speed meter (Steps 72–82 and 135–147)

A key requirement of iIACS is the isolation of the target cells of interest from the single stream of cells by measuring the flow speed of each cell with the speed meter and using the information to precisely predict the sort timing (Fig. 10). This situation is unique and different from FACS, in which it typically takes only a few hundred microseconds for a cell to flow from the optical interrogation point to the sort point. In iIACS, the long computation time of the image processor (>30 ms) is needed to analyze images whose data size is much larger than that in FACS. Consequently, the sort latency of iIACS (defined as the travel time of cells between the optical interrogation point and the sort point) is >100 times longer than that of FACS. In our iIACS machine1, the speed meter is implemented by measuring forward-scattered light from each passing cell at multiple optical interrogation spots. Specifically, as shown in Fig. 10, the iIACS machine has two optical interrogation spots (OI1 and OI3) adjacent to the FDM microscope (OI2) and one optical interrogation spot slightly upstream of the sorter (OI4). The size of each optical interrogation beam is ~6 µm × 80-µm in the flow direction and its perpendicular direction, respectively. When a cell passes through each optical interrogation spot, it scatters the incident optical beam so that the forward-scattered light is detected by an APD through an annular aperture obstruction target. Figure 10 shows an example of the scattering signals detected at OI1, OI3, and OI4. The scattering signals at these spots are digitized and sent to the image processor to calculate the flow speed and predict the sort timing at the sort point. When the signal intensity at OI1 exceeds a predetermined threshold, the image processor generates a trigger signal for the FDM microscope to initiate bright-field and fluorescence imaging and for the image processor to initiate the speed calculation to predict the sort latency.

Image processor (Steps 89–107 and 190–197)

The brain of iIACS that connects and controls all the sensors and actuators is a real-time intelligent image processor composed of multiple field-programmable gate arrays (FPGAs), CPUs, and graphics processing units (GPUs) on an all-IP network59,60,61—a communication network that uses the IP to send and receive signals between multiple computers, multiple sensors, and multiple actuators, and that is an essential element of the Internet of Things. This communication network acts as a platform upon which the FPGAs, CPUs, and GPUs digitally process different types of data (three APD signals from the FDM microscope and three photodiode signals from the speed meter) and communicate with each other at a high data transfer rate of 10 Gbps, offering high flexibility, high scalability, and real-time automated operation for intelligent image processing and decision making (Fig. 11). Specifically, in our iIACS machine1, the CPUs and GPUs perform image construction of events on the basis of three-color image acquisition (represented by the image construction (IC) node in Fig. 11) and image analysis of the events (represented by the image analysis (IA) nodes in Fig. 11), which includes multidimensional feature extraction, classification, and sort decision making, whereas the FPGA (represented by the time management (TM) node in Fig. 11) precisely determines the optimum sort timing on the basis of signals from the speed meter, predicts the latency at the sorter with a high precision (200 µs; Fig. 10c), and generates a trigger signal for the sort driver (another FPGA located outside the network), followed by amplifiers and piezoelectric actuators for the dual-membrane pumps, if a target event is identified. The TM node is on an FPGA board equipped with a 1-Gbps Ethernet port. The IC node is equipped with two multi-core CPUs, 10-Gbps Ethernet ports, and a 4-channel high-speed digitizer operating at a sampling rate of 1.25 GS/s, all of which are connected by a peripheral component interconnect express (PCIe) bus. The outputs of the APDs in the FDM microscope are connected to Ch0–Ch2 of the digitizer, and the serial digital signal from the TM node is connected to Ch3. In contrast to FACS, which uses only FPGA-based hardware for real-time signal processing, the combination of the FPGA-based hardware and the CPU–GPU-based software in the iIACS machine offers much higher flexibility and scalability for scaling and deploying on-demand network functions based on complex digital algorithms for image analysis, machine learning, and cell sorting. Selection of digital algorithms is highly flexible within the range of the specified processing time, thereby rendering previously developed image-processing algorithms (including machine learning) for microscopy and imaging flow cytometry platforms23,94,95 readily available to the iIACS machine. In the above embodiment, the all-IP network is a general-purpose interconnection of a 10-Gbps Ethernet and a 1-Gbps Ethernet and makes it easy to change the system configuration with high scalability simply by changing the computing resources, such as the number of CPUs on the IC/IA nodes, depending on the required processing load, such as the event rate of the iIACS machine. This infrastructure allows expansion of the system with high flexibility by connecting additional/other imaging methods and/or sensors to the all-IP network.

The signal-processing flow in the image processor is described as follows. When an object (e.g., a cell, a cell cluster, debris) passes through the optical interrogation points OI1 and OI3, its forward-scattering signals are detected by the photodiodes and digitized and analyzed by the TM node. Specifically, when the OI1 signal exceeds a specified threshold value, the TM node recognizes it as an event, assigns a cell ID number to the event, and gives a time stamp as a passage time. When two or more consecutive events occur within a threshold interval, the latter event(s) is aborted. After a predetermined duration (typically 160 µs), the cell ID signal and trigger signal are sent to the IC node as two-channel electrical serial signals. After the OI3 signal is detected, the TM node calculates the speed and predicts the sort time. All the calculations about each event are temporarily stored in memory on the FPGA until the corresponding sort time. In parallel, the cell ID and event flow speed value are sent to one of the IA nodes via a user datagram protocol (UDP) communication. When the digitizer in the IC node receives the trigger signal, the IC node begins the acquisition of the FDM signals. On the basis of the algorithms discussed above, three-color FDM images are produced with a typical field of view of 42 μm × 26.25 μm and an image size of 50 pixels × 105 pixels with 16-bit depth. The number of the 1D spatial profiles in the flow direction varies, depending on the type of events. The cell ID is received by Ch3 of the digitizer. The image data and cell ID are encapsulated in UDP packets and sent to one of the IA nodes. The IA node begins image analysis when it receives all the UDP packets of the images and cell IDs from the IC node and those of the speed information from the TM node. If some packets are not delivered within a specified time, the event is aborted. The image analysis algorithm is coded with Open Source Computer Vision Library (OpenCV) on C++. First, the pixel size is normalized to a square, leading to an image size of 168 pixels × 105 pixels. A sort decision is made on the basis of classic image analysis or deep CNN, depending on the complexity and need of events. It is also possible to implement other image analysis algorithms, as long as the total signal-processing time fits within the sort latency. Finally, the sort/unsort decision signal is sent from the IA node to the TM node with the cell ID via the UDP communication. When the TM node receives the sort decision, the sort trigger signal is transmitted to the sort driver at the sort time of the corresponding ID. In the sort driver, a predetermined signal waveform is generated and sent through a digital-to-analog converter to actuate the dual-membrane push–pull cell sorter.

Neural network (Steps 180–189)

The deep CNN is a type of deep neural network that is a flexible machine learning method for handling large datasets8,10,11. Because the deep CNN requires a substantial level of computation, it has previously been considered unsuitable for real-time processing. In our iIACS machine1, both a six-layer deep CNN and an eight-layer deep CNN (composed of convolution layers and fully connected layers accompanied by max-pooling and dropout connections) were constructed on TensorFlow96 and Keras97 software frameworks accelerated with the GPU, and used for classifying calibration particles and blood cells, respectively. >2,000 images of each cell type were manually classified to construct class labels as the ground truths, some of which were used for training the deep CNNs, whereas the others were used for validating them. The trained CNNs were implemented on the IA node in the real-time intelligent image processor (examples of the data and implemented CNN codes are available from the website http://www.goda.chem.s.u-tokyo.ac.jp/intelligentIACS/software.zip). The processing time of the image analysis using the deep CNNs combined with data transfer and image construction for each event was <32 ms (Fig. 11).

Cell sorter (Steps 40–46 and 129–134)

The function of the dual-membrane push–pull cell sorter is to rapidly isolate target cells from the single stream of cells by producing a perpendicular local flow at the sort point with a pair of piezoelectrically actuated dual-membrane pumps (Fig. 12). Each external piezoelectric actuator is placed on the corresponding glass membrane fabricated as a part of the microfluidic chip. Because the glass membrane is deformed by the motion of the piezoelectric actuator, a secondary local flow is generated across the microchannel in a direction perpendicular to the cell flow. When the membrane pumps are turned on in an antiphase manner, the local flow diverts a target cell and surrounding fluid out of the cell stream into either the upper or lower microchannel at the three-way junction. When the membrane pumps are turned off, cells flow directly toward the central microchannel. The upper and lower microchannels are connected at the downstream end to share one outlet. This configuration is effective for high-throughput sorting without the need for initializing the dual-membrane pumps. The membrane pumps are connected via a bubble-releasing channel to remove bubbles in the membrane pumps. A ramp voltage signal with an amplitude of 80 V and a rise time of 200 µs is used for the actuation. Although we used a three-way junction in our iIACS machine1, the number of exit ports can be increased to five, seven, or even a larger number, provided that the membrane pumps have sufficient lateral resolution to divert target cells from the central flowing stream of cells98.

Integration (Steps 112–117)

It is of great importance to precisely integrate all the elements into a single system in order to ensure the best performance. Specifically, the most complicated part of the system—requiring special attention for installation—is the optic–microfluidics integration, that is, the integration of the FDM microscope, speed meter, and microfluidic chip that hosts the two cell focusers and cell sorter. In our iIACS machine1, this is accomplished by building and installing an optics–microfluidics integration unit, which consists of three objective lenses, a microfluidic chip holder, and the chip itself (Fig. 13) (design available as Supplementary Data 2 of this protocol). Two objective lenses, A1 and A2, are aligned to image the cells flowing at the center of the microchannel, whereas the other objective lens, lens B, is used for monitoring the sorting process with a high-speed CMOS camera. The three optical interrogation beams of the speed meter (OI1, OI3, OI4) and the FDM microscope share these objective lenses. The microfluidic chip is sandwiched between two holding plates of the chip holder with tube connectors and piezoelectric actuators, and is then inserted into a slot in the optics–microfluidics integration unit (Fig. 13).

Fig. 13: Schematic of the optics–microfluidics integration unit.
figure13

a, Schematic of the unit. It mainly consists of three objective lenses and the microfluidic chip integrated with a chip holder. b, Schematic of the chip holder. The microfluidic chip is sandwiched between two holding plates of the chip holder with tube connectors and piezoelectric actuators. c, Procedure of placing the chip unit into the slot for the chip unit. d, Installation of the integration unit with details of the optical interrogation (OI1–OI4) and monitoring with the high-speed CMOS camera. The fluorescence signals can be detected in the direction of the transmitted illumination or in the opposite direction, depending on the overall design of the iIACS machine. a,d adapted from Nitta et al.1.

Monitoring optics (Steps 83–88)

iIACS requires precise control of flowing cells. To characterize the iIACS machine, it is important to have the ability to monitor the behavior of flowing cells in the microchannel. In our iIACS machine1, the monitoring optics consists of mirrors, lenses, and a high-speed CMOS camera. Specifically, the monitoring optics can be used to monitor the behavior of flowing cells in the microchannel at either the optical interrogation or sort points by switching optical paths. The objective lenses are shared with the FDM microscope and/or the speed meter at the optical interrogation and sort points. The positions of the laser spots of the FDM microscope and speed meter can also be verified with the high-speed CMOS camera. Although not essential, the monitoring optics is an important part of the iIACS machine for aligning the FDM microscope and speed meter with ease.

Characterization (Steps 148–165 and 204–218)

It is critical to characterize the basic performance and specifications of the iIACS machine with calibration particles and cell lines before conducting sorting experiments for practical applications (Figs. 1417). Just like FACS17,19, iIACS is evaluated on the basis of (i) throughput, (ii) sensitivity, (iii) dynamic range, and (iv) linearity. Throughput is the number of detectable or sortable events (e.g., single cells, cell clusters, cell debris) that flow through the iIACS machine per unit of time. Sensitivity is a detection threshold or an ability to resolve event populations, typically on the basis of molecules of equivalent soluble fluorophores (MESFs). Dynamic range is the ratio of the largest possible fluorescence signal to the smallest possible fluorescence signal. Linearity is the proportionality of the output to the number of photons. In addition to these parameters, iIACS is also evaluated on the basis of imaging-related parameters unique to iIACS, including (v) spatial resolution, (vi) the number of pixels, and (vii) the field of view of the microscope. Furthermore, (viii) recovery, (ix) purity, and (x) yield are used to evaluate sorting experiments and may vary, depending on the experiment and sample type, as they are indicators of the sorting experiments, not the iIACS machine. Recovery is the ratio between the number of target events in the collection tube and the number of target events that are imaged, analyzed, and recognized as sort events. Purity is the ratio of the number of target events in the collection tube and the total number of events in the collection tube. Yield is the ratio of the number of target events in the collection tube and the number of target events that flow through the iIACS machine (equivalent to the total number of target events in the collection and waste tubes, assuming that there is no lost event). In general, parameters (i–vii) are evaluated with calibration particles and cell lines, whereas additional parameters (viii–x) are evaluated in individual sorting experiments with practical cell samples.

Fig. 14: Throughput performance of the iIACS machine.
figure14

a, Time course of consecutive events imaged with the FDM microscope under three event-rate conditions (169.6, 85.18, and 30.55 e.p.s.). b, Histograms of events in the event-interval time under the three conditions (dots), with prediction lines calculated on the basis of Poisson distributions. c, Histograms of events in the event-interval time with 169.6 e.p.s. with (red) and without (gray) using the digital circuit that aborts events with an interval shorter than a specified duration. d, Event rate (green), actual throughput (blue), and abort rate (red) evaluated with the 169.6-e.p.s. condition under different event-interval threshold values. As the threshold value is increased, the error rate is reduced and the event rate is decreased. Consequently, the actual throughput, which is the product of the event rate and the abort rate, has a peak value at a threshold value of ~1 ms. c and d adapted from Nitta et al.1.

Fig. 15: Sorting performance of the iIACS machine.
figure15

a,b, Processing time of the image construction and data transfer combined (dark blue) and the image analysis (light blue) for each event using 6-µm particles, based on the classic algorithm (a) and deep CNN (b), on which deep learning was implemented. The measured events are rank-ordered in the total processing time. c, Deep CNN in a six-layer structure with four convolution layers and two fully connected layers for two training classes: 3-μm particles and 6-μm particles. d, Trajectories of a 6-µm particle (unsorted) and 3-µm particle (sorted) flowing at 1 m/s, recorded by a high-speed CMOS camera at 20,000 frames/s. Scale bars, 100 µm. e,f, Enrichment of 3-µm particles from a 1:1 mixture of 3-µm and 6-µm particles with the classic algorithm and deep CNN; 3-µm and 6-µm particles are shown in purple and green, respectively. In both demonstrations, a high throughput of ~100 e.p.s. and a high purity of ~99% were achieved. Scale bars, 1 mm (main); 50 µm (insets). Image adapted from Nitta et al.1.

Fig. 16: Images of various types of cells obtained by the iIACS machine.
figure16

a, Microalgal cells. From top to bottom: Chlorella sorokiniana, Chlamydomonas reinhardtii, Haematococcus lacustris, Gloeomonas anomalipyrenoides, and Euglena gracilis. Ch0: bright-field imaging. Ch1: staining with SYTO16 for C. sorokiniana, C. reinhardtii, H. lacustris, and G. anomalipyrenoides, and with BODIPY for E. gracilis. Ch2: autofluorescence mainly from chlorophyll. b, Human cells. From top to bottom: H1975 (adenocarcinoma) cell, erythrocyte, leukocyte, single platelet, and platelet aggregate. Ch0: bright-field imaging. Ch1: staining with SYTO16 for the H1975 cell, erythrocyte, and leukocyte, and dark-field imaging for the single platelet and platelet aggregate. Ch2: staining with anti-EpCAM PE for the H1975 cell, erythrocyte, and leukocyte, and with anti-CD61 PE for the single platelet and platelet aggregate. The flow speed of all the cells is 1 m/s (required to achieve ~100 e.p.s.). The high quality of the images indicates that the iIACS machine is capable of identifying the intracellular chemical distribution and morphological features of various cell types ranging from 3 to 30 µm in cell diameter. Scale bars, 10 μm. The blood samples were collected from a 38-year-old healthy male volunteer. The study related to the platelets, erythrocytes, and leukocytes was approved by the Institutional Ethics Committee of Faculty of Medicine, the University of Tokyo (no. 11049-5) and was conducted according to the Declaration of Helsinki. All volunteers provided written informed consent. Image adapted from Nitta et al.1.

Fig. 17: Flowchart of how to operate the iIACS machine.
figure17

The user is required to follow each step to perform iIACS and obtain reliable results.

In our recent work, we took a series of steps to evaluate these parameters and hence to characterize the iIACS machine1. First, we measured the sensitivity, dynamic range, and linearity of the fluorescence detection channels using 6-peak fluorescent particles to verify the ability of the iIACS machine to distinguish fluorescent particles with different fluorescence intensities in terms of MESFs. We also conducted fluorochrome separation using standard calibration particles for fluorescein isothiocyanate (FITC), phycoerythrin (PE), and PE cyanine 5 (PE-Cy5), as well as blank particles (Fig. 9). In addition, we evaluated and calibrated the spatial resolution of the FDM microscope (Fig. 9). Second, we tested various cell types, ranging from 3 to 30 µm in cell diameter, to evaluate the image quality of the microscope (Fig. 16). For cells whose size is outside this range, the microfluidic chip design needs to be modified. Third, we verified that all the functions on the image processor, including the image construction, image analysis, and data transfer, were conducted within a total of 32 ms for both a classic algorithm (Fig. 15) and a deep CNN on which deep learning was implemented. Fourth, we evaluated the event rate and event density of the machine to verify that events obey Poisson distributions. The event rate, actual throughput, and abort rate were checked at 169.6 e.p.s. under different event interval threshold values. As predicted, as the threshold value was increased, the error rate was reduced and the event rate was decreased. As a result, the actual throughput, which is the product of the event rate and the non-abort rate, was found to peak at a threshold value of ~1 ms, which indicates a throughput of ~100 e.p.s. (Fig. 14). Finally, to evaluate sorting experiments, we constructed a centrifugation-based cell counting device. Specifically, the sorted liquid in the collection tube was loaded onto a custom glass-bottom chamber with a volume of 1 mL and a viewing area of 7 mm in diameter—fabricated by bonding a glass substrate and a cylindrical structure made of polydimethylsiloxane (PDMS)—and centrifuged at 300g for 10 min to collect cells on the glass substrate. To avoid non-specific binding of cells to the PDMS, the chamber was incubated with blocking solution (1% (wt/vol) BSA in PBS solution, sterilized with a 0.22-µm filter) before loading the sample. After the centrifugation, the whole viewing area was scanned with a commercially available fluorescence microscope equipped with a digital CMOS camera and a 20× objective lens, followed by image analysis to enumerate cells with NIS-Elements AR.

Materials

Biological materials

  • Cellular samples, such as primary cells of interest including human blood, or cultured cells, e.g., H1975 cells (ATCC, cat. no. CRL-5908)

    Caution

    For primary cells derived from human donors, informed consent must be obtained from the donors. The protocol must be performed in compliance with appropriate national laws and institutional regulatory board guidelines. An appropriate ethics review board must approve the protocol for the experimental design. The experiments described in this protocol for detecting human blood cells were approved by the International Ethics Committee of the Faculty of Medicine, University of Tokyo.

Reagents

Microfluidic chip

  • OFPR photoresist (Tokyo Ohka Kogyo, cat. no. OFPR-800 15cp)

    Caution

    OFPR photoresist is highly flammable and toxic. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • SCM250 photoresist (Nikko-Materials, cat. no. SCM250)

  • OFPR developer and SCM250 developer (Tokyo Ohka Kogyo, cat. no. NMD-3)

  • Chromium (Cr) etchant (Nihon Kagaku Sangyo, cat. no. N14B)

  • Gold (Au) etchant (Kanto Chemical, cat. no. AURUM-302)

  • SU-8 photoresist (Nihon Kayaku, cat. nos. SU-8 3010 and SU-8 3025)

    Caution

    SU-8 is highly flammable and toxic. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • SU-8 developer (Tokyo Ohka Kogyo, cat. no. PM thinner)

    Caution

    This liquid is highly flammable and toxic. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • Isopropanol (Sigma-Aldrich, cat. no. 278475)

    Caution

    Isopropanol is highly flammable. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • Acetone (Kanto Chemical, cat. no. 01026-80)

    Caution

    Acetone is highly flammable and toxic. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • Pressurized nitrogen gas (Jyotou Gas, >99.99 vol%)

  • Sulfur hexafluoride gas (TOMOE SHOKAI, cat. no. SF6 5N)

  • Octafluorocyclobutane gas (TOMOE SHOKAI, cat. no. C4F8 5N)

  • Milli-Q ultrapurified water (Merck, cat. no. Direct-Q UV 8)

  • Sulfuric acid (Kanto Chemical, cat. no. 37390-00)

    Caution

    Sulfuric acid is highly corrosive. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • Hydrogen peroxide (Kanto Chemical, cat. no. 18084-00)

    Caution

    Hydrogen peroxide is highly corrosive. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • Hydrofluoric acid (Morita Chemical Industries, cat. no. 18083-00)

    Caution

    Hydrofluoric acid is extremely toxic and highly corrosive. Use a fume hood and wear appropriate personal protective equipment when handling it.

Cell focuser

  • Epoxy resin (3M, cat. no. 7004)

Centrifuge-based cell counter

  • Polydimethylsiloxane (PDMS; Dow Corning, cat. no. Sylgard 184)

  • BSA (Sigma-Aldrich, cat. no. A9576)

  • Hexane (FujiFilm Wako Pure Chemical, cat. no. 084-03421)

    Caution

    Hexane is highly flammable and toxic. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • PBS (D-PBS(−) without Ca and Mg, liquid; Nacalai Tesque, cat. no. 14249)

  • Acetone (FujiFilm Wako Pure Chemical, cat. no. 014-08681)

    Caution

    Acetone is highly flammable and toxic. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • Methanol (FujiFilm Wako Pure Chemical, cat. no. 138-06473)

    Caution

    Methanol is highly flammable. Use a fume hood and wear appropriate personal protective equipment when handling it.

  • Isopropanol (FujiFilm Wako Pure Chemical, cat. no. 165-09161)

    Caution

    Isopropanol is highly flammable. Use a fume hood and wear appropriate personal protective equipment when handling it.

iIACS maintenance and characterization

  • Sheath fluid (Beckman Coulter, cat. no. 8599600)

  • Clean solution (Becton Dickinson, cat. no. 340345)

  • Rinse solution (Becton Dickinson, cat. no. 340346)

  • Standard 6-peak particle set (Spherotech, cat. no. RCP-30-5)

  • Fluorescent calibration particle set (Spherotech, cat. no. ECFP-K1)

  • Standard calibration particle set (Spherotech, cat. no. PPS-6K)

  • 1-µm Particle set (Thermo Fisher Scientific, cat. no. F8820)

Equipment

Liquid pump

  • Pressure vessel (Unicontrols, model no. TM5SR)

  • Electro-pneumatic regulator (Asahi Enterprise, model no. 641AB203)

  • Liquid flow meter (Sensirion, model nos. FlowMeterKit SLI-2000 and FlowMeterKit SLI-0430)

  • Diaphragm isolation valve (Takasago Electric, model no. MTV-3R-1/4-28UNF)

  • Pinch valve (Takasago Electric, model no. PS-0815NC)

  • Round-bottom polystyrene tube (Falcon, cat. no. 2-1919-04)

  • Aqueous-solution filter (GE Healthcare Whatman, cat. no. Polycap AS 36)

  • Microcontroller board (Arduino 1.6.11;Arduino, model no. Mega 2560 R3)

  • Computer with Visual Studio 2015, 8 GB or more RAM, Windows 10

Microfluidic chip

  • Silicon wafer (6-inch diameter, 200-µm thickness, both sides polished; Matsuzaki Seisakusyo)

  • Borosilicate glass wafer (6-inch diameter, 200-µm thickness; Eikoh)

  • Dry-etching machine (Tateyama Machine, model no. TEP-01)

  • Deep reactive-etching machine (Samco, model no. RIE-800)

  • Sputterer (Canon Anelva, model no. E-200S)

  • Laser lithography machine (Heidelberg Instruments, model no. DWL66FS)

  • Mask aligner (SÜSS MicroTec, model no. MA-6)

  • Spin coater (Mikasa, model no. MS-B100)

  • Hot plate (AS ONE, model no. HP-2SA)

  • Dicing saw (DISCO, model no. DAD522)

  • Anodic bonding machine (Nanometric Technology, model no. APA-01)

  • Sandblasting machine (Elfo-tec, model no. ELP-1TR)

FDM microscope

  • Arbitrary waveform generator (Signatec, model no. PXDAC4800-DC)

  • Digitizer (Spectrum, model no. M4i.2212-x8)

  • Computer with LabVIEW 2016, 128 GB or more RAM, Windows 10

  • BNC cable (RS Components, cat. no. 122-2145)

  • SMA cable (RS Components, cat. no. 498-7585)

  • BNC-SMA adaptor (RS Components, cat. no. 888-4673)

  • Electric wire (RS Components, cat. no. 677-3243)

  • Amplifier (Mini Circuits, model no. ZHL-1-2W-S+)

  • DC power supply for amplifier (Kikusui Electronics, model no. PMX35-3A)

  • Optical power meter (Thorlabs, model no. PM100D, S121C)

Optical components for the FDM microscope

  • Laser (Coherent, model no. Genesis CX 488-2000 STM)

  • Mirror (Thorlabs, model no. BB1-E02)

  • Achromatic doublet lenses (Thorlabs, model nos. AC254-100-A-ML, AC254-125-A-ML, AC254-150-A-ML, AC254-200-A-ML, AC254-300-A-ML, and AC254-500-A-ML)

  • Wave plate (Thorlabs, model no. WPH10M-488)

  • Kinematic mirror mount (Thorlabs, model no. KM100)

  • Rotation mount (Thorlabs, model no. RSP1/M)

  • Prism holder (Optosigma, model no. KKD-25PH)

  • Polarizing beam splitter (Thorlabs, model nos. PBS101, PBS201)

  • Cylindrical lens (Thorlabs, model nos. LK1662L1-A, LK1277L1-A)

  • Acousto-optic deflector (AOD; Brimrose, model no. TED-300-200-488)

  • Translation stage (Optosigma, model no. TSD-251S)

  • Rotation stage (Optosigma, model no. KSP-256)

  • Anamorphic prism pair (Thorlabs, model nos. PS879-A, PS875-A)

  • Non-polarizing beam splitter (Optosigma, model no. NPCH-15-4880)

  • Kinematic mount (Thorlabs, model no. KMS/M)

  • Right-angle prism mirror (Thorlabs, model no. MRA15-E02)

  • Mounts for anamorphic prism pair (Thorlabs, model no. FMP1)

  • Kinematic mount (Edmund Optics, model no. 58-874)

  • Dichroic mirror (Semrock, model nos. ff509-fdi01, ff580-fdi01)

  • Notch filter (Semrock, model no. NF03-488E)

  • Lens holder (Thorlabs, model no. LM1XY/M)

  • Aspheric lens (Thorlabs, model no. A240TM-A)

  • Lens mount (Thorlabs, model no. S1TM12)

  • Avalanche photodetector (APD; Thorlabs, model nos. APD430A2/M, APD430A/M)

  • Beam shutter and controller (Thorlabs, model no. SH1/M, SC10)

  • Post (Thorlabs, model nos. TR20/M-JP, TR30/M-JP, TR40/M-JP, TR50/M-JP, TR75/M-JP, TR100/M-JP, TR150/M-JP)

  • Post holder (Thorlabs, model nos. UPH30/M, UPH50/M, UPH75/M, UPH100/M, UPH150/M)

  • Angle clamp (Thorlabs, model no.RA180/M)

  • Acousto-optic deflector attachment (custom made, see Steps 57 and 61)

  • Neutral-density filter (Thorlabs, model no. NE05A, NE10A, NE20A, NE30A)

  • High-pass filter (Thorlabs, model no. EF513)

  • Objective lens (20×, numerical aperture = 0.75; Leica, model no. HC PL APO CS2)

  • Objective lens mount (custom made, see Fig. 13a)

  • Flip mount (Thorlabs, model no. TRF90/M)

  • Iris (Thorlabs, model no. SM1D25)

Optical components for the speed meter and monitoring optics

  • Laser (Thorlabs, model no. LDM850)

  • Dichroic mirror (Thorlabs, model nos. DMSP805, DMSP805R)

  • Mirror (Thorlabs, model nos. BB1-E03, BB2-E02, POLARIS-19S50/M)

  • Neutral-density filter (Thorlabs, model no. NDC-50C-4M-B)

  • Cylindrical lenses (Thorlabs, model nos. LJ1653L1-B, LK1431L1-B, LJ1558L2-B, LJ1629L2-B, LJ1363L1, LJ1141L1)

  • Beam splitter cube (Thorlabs, model nos. CCM5-BS017/M, BS020)

  • Plano-convex spherical lens (Thorlabs, model nos. LA1213-B, LA1433-B)

  • Aspheric lens (Thorlabs, model no. A260TM-A)

  • Dichroic mirror (Semrock, model no. Di02-R635-25×36)

  • Annular aperture obstruction target (Thorlabs, model no. R1DF500)

  • APD (Thorlabs, model no. APD410A/M)

  • Objective lens (10×/0.30; Olympus, model no. UPlanFLN)

  • Illumination light (Schott, model no. KL 1600 LED)

  • Condenser lens (Thorlabs, model no. ACL2018U-A)

  • Achromatic doublet lens (Thorlabs, model no. AC254-400-B-ML)

  • Tube lens (Thorlabs, model no. TTL200)

Speed meter and monitoring optics

  • Oscilloscope (Pico Technology, model no. 2406B)

  • High-speed CMOS camera (Vision Research, model no. Phantom v1211)

  • BNC cables (RS Components, cat. no. 122-2144)

Optic–microfluidics integration unit, cell focuser, and cell sorter

  • High-voltage amplifier (NF, model nos. HSA4101, HSA4012)

  • Function generator (NF, model no. WF1974)

  • Piezoelectric actuator (Tokin, cat. no. AE0203D04DF)

  • Piezoelectric transducer (0.57-mm-thick, 20 mm × 20-mm element; Fuji Ceramics, 3.66Z20*20S-SYX)

  • O-ring (Musashi Oil Seal, cat. no. SS-005-1A)

  • Tube connecter (SMC, cat. no. M-3AU-2)

  • Parallel pins (KS Sangyo, cat. nos. 2005-H7, 2010-H7)

  • Spring (Tokyo Hatsujyo Manufacturing, cat. no. JA-28)

  • Steel ball attachment (Tsubaki Nakashima, cat. no. SBM-SUJ-2)

  • Mechanical stage (Optosigma, model nos. TSD-251S, TSD-251SR, TSD-253L, TSD-253RL, TSD-401C, TSD-401CDMUU, OSMS40-5ZF)

Image processor

  • FPGA board (Xilinx, model no. KC705, SP601)

  • Analog-to-digital converter board (ADC; Analog Devices, model no. AD9259-50EBZ)

  • Digital-to-analog converter (DAC; Analog Devices, model no. AD5248)

  • Computer with GPU, 10-Gbps network interfaces, 32-GB or more RAM, and Ubuntu Linux 16.04LTS

  • Network switch (NetGear, model no. XS716T-100AJS)

  • Category 6A Ethernet cables (RS Components, cat. no. 775-4864)

  • SMA cable (RS Components, cat. no. 144-0880)

Centrifuge-based cell counter

  • Glass-bottom chamber (single-well glass-base dish, well 8.0Φ, no. 1-S; Iwaki, cat. no. 3971-101)

  • Centrifuge (Thermo Scientific, model no. Sorvall Legend X1)

  • Fluorescence microscope (Nikon Instruments, model no. Ti2)

  • Excitation light source (Lumencor, model no. SOLA SE II)

  • Digital CMOS camera (Hamamatsu Photonics, model no. ORCA Flash 4.0 V3)

  • Filter set (Semrock, model nos. CFW-BP01-Clinical-NTE, GFP-A-Basic-NTE, TRITC-A-Basic-NTE, Cy5-A-Basic-NTE)

  • Objective lens (Nikon Instruments, model no. S Plan Fluor ELWD 20× DIC N1)

  • Coverslip (thickness no. 1, 0.12–0.17 mm, 25-mm diameter; Matsunami, cat. no. C025001)

  • Sonicator (AS ONE, model no. VS-D100)

  • Air plasma (Meiwafosis, model no. SEDE-PFA)

  • Desiccator (AS ONE, model no. SD-3)

  • Syringe filter (Sartorius, model no. 16532)

Software

Procedure

System design

Timing 30–60 d

  1. 1

    Determine the maximum target throughput of cell sorting.

    Critical step

    Consider modifying the maximum target throughput if it is not practical according to the specifications of the microscope (Step 4), real-time intelligent image processor (Step 8), and microfluidic chip (Step 9).

  2. 2

    Calculate the appropriate flow speed and sort window in the time domain to achieve the maximum target throughput.

    Critical step

    Consider the purity and yield of sorting at the calculated flow speed. The arrival of flowing cells with or without hydrodynamic focusing is governed by the Poisson process, resulting in inconsistent intervals between consecutive cells19. In a Poisson process, the time interval T between successive events will possess a cumulative probability distribution function of the form P(T<t)=1–eNt, where P is the probability of an event and N is the throughput. If the yield or purity is 100%, the throughput is N = 100 e.p.s., and the sort window is t = 500 µs, then the purity or yield is 1–P(T<t)95%. This means that 5% of events would have a shorter interval than the sort window. Consider target yield and purity and tune the parameters that are related to abort (e.g., high-purity mode (in which events with shorter intervals are aborted) or high-yield mode (in which all positive events are sorted)).

  3. 3

    Design the structure of the iIACS machine. Consider the design of the iIACS machine shown in Fig. 2 as an exemplary model. The details of the entire structure of our iIACS machine1 are described in the ‘System design’ section of the ‘Experimental design’ section. In our iIACS machine1, an optics table was used as a platform upon which to install the iIACS machine. Optics tables are useful for constructing tabletop instruments because they are flexible and it is easy to do trial-and-error tests with them, although it is also possible to fabricate a home-made platform because of the compactness of the whole structure.

  4. 4

    Choose an appropriate type of the microscope on the basis of the target cells and the target throughput for iIACS.

    Critical step

    Consider the image quality of cells that flow at a high flow speed. A high-speed microscope is required to acquire blur-free images.

    Critical step

    Consider modifying the target throughput if the flow speed is too high to allow acquisition of blur-free images using the microscope.

  5. 5

    Design the optical setup of the microscope, including the IC node. Consider the schematic of the microscope shown in Fig. 8 as an exemplary model. The details of the microscope are described in the ‘Microscope’ section of ‘Experimental design’ section.

  6. 6

    Design the optical setup of the speed meter for measuring the speed of flowing cells in the microchannel and the monitoring optics for visualizing the sort area via the high-speed CMOS camera. Consider the schematic of the speed meter shown in Fig. 10a as an exemplary model. The details of the speed meter and monitoring optics are described in the ‘Speed meter’ and ‘Monitoring optics’ sections of ‘Experimental design’ section, respectively.

  7. 7

    Design the real-time intelligent image processor on the all-IP network. Consider the schematic of the image processor shown in Fig. 11 as an exemplary model. The details of the image processor are described in the ‘Image processor’ section of the ‘Experimental design’ section.

    Critical step

    Important design issues for the all-IP network system architecture of the iIACS machine are as follows: use a UDP instead of a transmission control protocol (TCP)/IP because of the real-time nature of the system; use FPGA to interface between the Ethernet and instrumentation hardware instead of general-purpose operating systems such as Windows and Linux because the general-purpose operating systems have jitters that are mainly due to IP filtering; arrange network switches and devices carefully to avoid packet losses caused by packet conflicts.

  8. 8

    Estimate the total image-processing time of the IC and IA nodes, sort decision making, and data transfer.

  9. 9

    Design the microfluidic chip on the basis of the requirements of the microscope and real-time intelligent image processor. The microscope requires adjustment of the cell focusing performance (need to flow cells in the field of view and the depth of focus) and flow speed (need to flow cells in the range where images can be taken), whereas the real-time intelligent processor requires enough latency from the image timing of an event to its sort timing. Moreover, a microchannel with a square-shaped cross-section is suitable for acoustic focusing. For designing the various elements of the microfluidic chip, such as the hydrodynamic focuser and the dual-membrane push–pull cell sorter, a numerical simulation of computational fluid dynamics is useful. Consider the schematic of the microfluidic chip shown in Fig. 5 as an exemplary model. The details of the microfluidic chip are described in the ‘Microfluidic chip’, ‘Cell focuser’, and ‘Cell sorter’ sections of ‘Experimental design’ section.

    Critical step

    Consider modifying the target throughput to reduce the flow speed if it is not practical to achieve enough sort latency to fit within the total image-processing time estimated in Step 8.

  10. 10

    Design the optics–microfluidics integration unit that holds the microfluidic chip and objective lenses. Consider the schematic of the optics–microfluidics integration unit shown in Fig. 13 as an exemplary model. The details of the optics–microfluidics integration unit are described in the ‘Integration’ section of ‘Experimental design’ section.

    Critical step

    The objective lenses can be fixed onto the optics table directly, but using an optics–microfluidics integration unit would be effective in designing an overall layout free of mechanical interference.

  11. 11

    Design the liquid pump that controls the flow in the microchannels. Consider the schematic of the liquid pump shown in Fig. 4 as an exemplary model, in which the liquid pump consists of a control board and components (e.g., valves, flow sensors) to control the flow. The details of the liquid pump are described in the ‘Liquid pump’ section in the ‘Experimental design’ section.

    Critical step

    Commercially available syringe pumps can be used, but a home-made liquid pump makes operations easier.

  12. 12

    Design the configurations and layout of the iIACS machine, which consists of the microscope, the speed meter, the optics–microfluidics integration unit, the liquid pump, and computers for the real-time intelligent image processor, on the basis of the space available in the lab. Consider the iIACS machine shown in Fig. 3 as an exemplary model of the system layout.

    Critical step

    Draw the configurations and layout using a 3D design software program such as SolidWorks. It is possible to download precise models of all the components (Thorlabs provides free models for many optical components: https://www.thorlabs.com/). Reserve enough space to install and tune the components.

    Critical step

    Modify the optics–microfluidics integration unit design to fit the optical setup of the microscope if needed.

Preparation of components

Timing 30–60 d

  1. 13

    Determine the specifications of all the necessary components, which are listed in the Materials section.

    Critical step

    Make sure that the specifications of the components meet the requirements for constructing the iIACS machine. In particular, the frequency bandwidth of the AODs, the laser power, the bandwidth of the network, and the total image-processing time are important. Modify the system design if needed.

  2. 14

    Purchase and prepare the components. The waiting time may be weeks long.

Construction of the liquid pump

Timing 7–14 d

  1. 15

    Download and install Microsoft Visual Studio 2015 and Arduino IDE to a computer.

  2. 16

    Develop programs to control valves and pressures for flowing liquids. The diagram of the liquid pump and flow pipeline configurations used in our iIACS machine are shown in Fig. 4.

    Critical step

    Use two control modes: a manual control mode and an automatic control mode. In the manual control mode, the operator can directly set pressure values to flow liquids, which means that the liquid pump is an open-loop system. In the automatic control mode, the operator can set the flow speed and sample-stream diameter (or flow rate ratios between the sample flow, vertical sheath flow, and horizontal sheath flow). According to the settings, the pressure levels are controlled in a feedback control system based on proportional–integral–differential (PID) control.

  3. 17

    Assemble the components (i.e., valves, tubing, liquid flow meters, pressure vessels, electro-pneumatic regulators, control board) to create the liquid pump.

  4. 18

    Install the liquid pump near the optics table.

  5. 19

    Verify the performance of the liquid pump.

  6. 20

    Place the sheath solutions (e.g., DI water, commercially available sheath fluid for conventional FACS) in the two pressure vessels and flow the solutions into the aqueous solution filters and tubing.

    Critical step

    Make sure that bubbles in the aqueous solution filters are removed. Remaining bubbles can, otherwise, destabilize the flow speed and cell focusing.

    troubleshooting

Construction of the microfluidic chip

Timing 7–14 d

Critical

Steps 23–34 and 36–38 should be performed in a clean-room environment with a grade of class 10,000 or higher.

Critical

Refer to Fig. 6 for a schematic overview of Steps 23–38.

  1. 21

    Draw a mask design of the microchannel for photolithography using a computer-aided design software such as AutoCAD. The AutoCAD design files for the microfluidic chip design used in our setup are available in Supplementary Data 1.

    Critical step

    Design the positions of the inlets and outlets of the microchannels such that there is enough space for the inlet and outlet tubing, objective lenses, and the piezoelectric actuators of the membrane pumps, in order to avoid mechanical interference.

  2. 22

    Submit the design to a vendor (e.g., Toyo Precision Parts Mfg.: http://www.toyo-ppm.co.jp/eng/) to print the mask.

  3. 23

    Spin-coat (3,000 r.p.m. for 30 s at room temperature (22–26 °C)) and pattern negative photoresist SU-8 (cat. no. SU-8 3010) as the etching mask of the reactive-ion etching (RIE) process. The pattern includes a grating pattern to prevent breaking of the layers due to the local stress concentration in a bonding process. The processing guidelines for the negative photoresist are available online (http://www.microchem.com/Prod-SU83000.htm). Form a grating pattern on the 200-µm-thick borosilicate glass wafer of the base and cover layers using the RIE process.

  4. 24

    Sputter Cr and Au layers to be 5-nm thick and 15-nm thick, respectively, on the borosilicate glass wafer, which is the surface opposite the surface with the grating pattern.

    Critical step

    Perform Step 24 under a low-pressure condition (<1 Pa) to make sure that the sputtered metal layers have high purity.

  5. 25

    Spin-coat (3,000 r.p.m. for 30 s at room temperature) positive photoresist OFPR to be 10-µm thick on the sputtered Au layer and pattern the spin-coated photoresist. The processing guidelines for the positive photoresist are available from its vendor (https://www.tok.co.jp/eng/products/semiconductor/list).

  6. 26

    Pattern the sputtered Cr and Au layers through the patterned photoresist layer, using an Au etchant and a Cr etchant.

  7. 27

    Pattern microchannels on the base and cover layers, using hydrofluoric acid with a depth and width of 100 and 400 µm, respectively.

    Caution

    Perform Step 27 in a fume hood with appropriate personal protective equipment.

    troubleshooting

  8. 28

    Remove the etching masks, which are composed of positive photoresist, Cr, and Au, using appropriate reagents: acetone, the Cr etchant, and the Au etchant, respectively.

    Caution

    Perform Step 28 in a fume hood with appropriate personal protective equipment.

    Pause point

    The fabricated cover layers can be stored at room temperature in a dust-free environment for no more than 3 months until further use.

  9. 29

    Cut a 6-inch 200-µm-thick silicon (Si) wafer into a 43-mm × 94-mm rectangular shape by using an automatic dicing saw.

  10. 30

    Clean the patterned base layer and the Si substrate with piranha solution.

    Caution

    Piranha solution is a mixture of sulfuric acid and hydrogen peroxide (H2SO4/H2O2 = 2:1), is a highly reactive solution, becomes extremely hot during the preparation, and can melt plastics. Perform the cleaning process with piranha solution with appropriate personal protective equipment. Use glass containers. For the preparation of piranha solution, add the peroxide to the acid slowly.

  11. 31

    Bond the patterned base layer and the Si substrate by anodic bonding.

  12. 32

    Spin-coat (3,000 r.p.m. for 30 s at room temperature) and pattern negative photoresist SU-8 (cat. no. SU-8 3025) as the etching mask of the deep reactive-ion etching (DRIE) process. The pattern includes the microchannels, inlets, outlets, and dual-membrane pumps. The processing guidelines for the negative photoresist are available online (http://www.microchem.com/Prod-SU83000.htm).

  13. 33

    Etch the Si layer by the DRIE process.

    Critical step

    Use the pulse-time-modulated plasma-etching method to achieve notch-free etching profiles. This etching method can suppress the charging phenomenon on the base layer that causes appearance of unwanted notches. Typically, the duration and the period of the pulses are set to be 1 ms and 3 ms, respectively.

    Pause point

    The fabricated microchannel can be stored at room temperature in a dust-free environment for no more than 3 months until further use.

  14. 34

    Laminate a 250-µm-thick sheet-type negative photoresist SCM250 to the back side of the wet etched surface on a 60 °C hot plate, and pattern the etching mask for the sandblasting process of the cover layer.

  15. 35

    Form the inlets and outlets by sandblasting the cover layer through the etching mask.

    Caution

    Perform Step 35 with appropriate personal protective equipment.

  16. 36

    Clean the microchannel and the cover layer with piranha solution.

    Caution

    Piranha is a mixture of sulfuric acid and hydrogen peroxide (H2SO4/H2O2 = 2:1), is a highly reactive solution, becomes extremely hot during the preparation, and can melt plastics. Perform the cleaning process with piranha solution with appropriate personal protective equipment. Use glass containers. For the preparation of piranha solution, add the peroxide to the acid slowly.

  17. 37

    Package the microchannel and cover layer, using anodic bonding.

  18. 38

    Cut the packaged microfluidic chip into 20 mm × 90-mm pieces with an automatic dicing saw.

  19. 39

    Inspect the microfluidic chip under a microscope.

    Pause point

    The developed microfluidic chip can be stored at room temperature in a dust-free environment for no more than 6 months until further use.

Construction of the cell sorter and hydrodynamic and acoustic cell focusers

Timing 1–3 d

  1. 40

    Draw the design of the chip holder (Fig. 13b) with the aid of a computer-aided design software program such as SolidWorks (see Supplementary Data 2 for our example design). The chip holder is one of the components of the optics–microfluidics integration unit. Consider the schematic of the hydrodynamic focuser shown in Fig. 7 as an exemplary model.

  2. 41

    Submit the chip holder design to a machine shop to fabricate the house-designed components of the chip holder.

  3. 42

    Assemble the house-designed components into the chip holder.

  4. 43

    Set the developed microfluidic chip (from Step 39) into the chip holder.

  5. 44

    Construct the hydrodynamic cell focuser by connecting tubings.

  6. 45

    Construct the cell sorter by integrating two piezoelectric actuators into the chip holder.

  7. 46

    Construct the acoustic cell focuser by gluing piezoelectric transducers to both sides of the microfluidic chip with epoxy resin.

Construction of the FDM microscope

Timing 1–3 d

  1. 47

    Prepare a computer, an arbitrary waveform generator, a digitizer, an oscilloscope, and a function generator.

  2. 48

    Install the arbitrary waveform generator and the digitizer on the computer.

  3. 49

    Install LabVIEW on the computer. Copy sample LabVIEW programs for the arbitrary waveform generator and the digitizer from recording media provided by the manufacturers.

  4. 50

    Confirm that the digitizer and the waveform generator work properly.

  5. 51

    Develop a waveform-generation program using LabVIEW. Use the built-in function ‘Multitone Generation.vi’ to create the waveform data. Use a sample program provided by the manufacturer for the configuration and operation of the arbitrary waveform generator.

    Critical step

    Frequency components of a desired multitone waveform should be multiples of fw/Nw and fd/Nd, where fw, Nw, fd, and Nd represent the sampling frequency of the arbitrary waveform generator, the number of data points in the output waveform dataset of the arbitrary waveform generator, the sampling frequency of the digitizer, and the number of data points acquired on the digitizer, respectively.

  6. 52

    Install a 488-nm laser on the optics table. Turn on the laser.

    Caution

    The laser output is strong enough to damage eyes. Do not look into the laser. Wear laser safety glasses to protect your eyes whenever the laser is on.

  7. 53

    Adjust the beam size of the laser output, using beam expanders (pairs of achromatic doublet lenses) and anamorphic prism pairs such that it fits within the active apertures of two AODs.

  8. 54

    Install a half-wave plate and a polarizing beam splitter to split the laser beam into the transmission beam and the reflection beam. Adjust the rotation angle of the half-wave plate along the incident beam direction such that the powers of the transmission and reflection beams are identical.

  9. 55

    Insert a half-wave plate into the path of the transmission beam. Adjust the rotation angle along the incident beam direction such that the polarization of the incident beam (horizontal polarization) is rotated by 90°.

  10. 56

    Place the two AODs at the designed positions.

    Critical step

    Make sure that the AODs are placed in correct directions such that the deflected beams from the AODs are combined with correct pairing of optical frequencies.

  11. 57

    Place at least two steering mirrors to direct the transmission beam to one of the AODs (AOD1) that is mounted on a rotation stage and a translation stage by a custom-made attachment.

  12. 58

    Connect the outputs of the arbitrary waveform generator, amplifiers, and AODs. Connect a DC power supply and the amplifiers. Turn on the DC power supply.

  13. 59

    Generate driving signals to the AODs, using the waveform-generation program.

  14. 60

    Adjust the position and tilt angle of AOD1 to maximize the power of the first-order deflection beam array.

  15. 61

    Place at least two steering mirrors to direct the reflection beam to the other AOD (AOD2) that is mounted on a rotation stage and a translation stage by a custom-made attachment.

  16. 62

    Adjust the position and tilt angle of AOD2 to maximize the power of the first-order deflection beam array.

  17. 63

    Place a non-polarizing beam splitter at a designed position. Combine the deflected beams from the AODs, using the non-polarizing beam splitter to form a Mach–Zehnder interferometer. Adjust the position and angle of the non-polarizing beam splitter such that the deflected beams spatially overlap at a distant location (tens of centimeters or longer) from the non-polarizing beam splitter. As a result, two combined beam arrays are generated from the non-polarizing beam splitter. They are called excitation beam arrays hereafter.

  18. 64

    Detect one of the excitation beam arrays using an APD. Attenuate the power of the beam array using neutral-density filters to avoid saturation of the APD. Monitor the power spectrum of the APD output, using the digitizer and a spectrum analysis program. Maximize the power of the desired frequency components and minimize unwanted cross-talk by adjusting the positions and angles of the AODs, non-polarizing beam splitter, and mirrors.

    Critical step

    Record the data of an optimized power spectrum. The data can be used as a reference for aligning the FDM microscope.

  19. 65

    Build relay lens systems to direct the other excitation beam array to the place where the microfluidic chip is set. Place a dichroic mirror (DM1) that reflects the excitation beam array and transmits fluorescent light in the middle of the beam path.

    Critical step

    Adjust the beam size, using the relay lens systems and anamorphic prism pairs such that the beam array fills the back aperture of an objective lens A2 to tightly focus the beam array onto the microfluidic chip.

  20. 66

    Place a focusing lens and an APD for detecting bright-field image signals. Adjust the position of the APD or the focusing lens such that all the beam array is incident on the active area of the APD.

    Critical step

    The gain of the APD for bright-field imaging should be minimized to maximize the signal-to-noise ratio. A higher gain results in increased excess noise on the APD and hence reduced signal-to-noise ratio.

  21. 67

    Place a mirror at the position of the microfluidic chip. Adjust the position of the mirror such that the excitation beam array is focused onto the mirror.

  22. 68

    Place a focusing lens and an APD for fluorescence detection behind the dichroic mirror. Detect the reflected beam array using the APD (a small portion of the beam array transmits through DM1). Adjust the position of the APD or the focusing lens such that all the beam arrays are incident on the active area of the APD.

  23. 69

    Place a dichroic mirror (DM2) that splits longer and shorter wavelength components of the fluorescent light in the beam path between the focusing lens and DM1.

  24. 70

    Place a focusing lens and an APD for fluorescence detection in the reflection beam path of DM2. Detect the reflected beam array using the APD (a major portion of the beam array reflects DM2). Adjust the position of the APD or the focusing lens such that all the beam arrays are incident on the active area of the APD.

  25. 71

    Remove the mirror placed at the position of the microfluidic chip.

Construction of the speed meter

Timing 1–3 d

  1. 72

    Install a diode laser on a kinematic rotation mount.

    Caution

    The laser output is strong enough to damage eyes. Do not look into the laser. Wear laser safety glasses to protect your eyes whenever the laser is on.

    Critical step

    Make sure that the emission wavelength of the laser does not overlap with that for fluorescence excitation and emission.

  2. 73

    Place a pair of cylindrical lenses to reduce the beam diameter in the vertical direction.

  3. 74

    Adjust the rotation angle of the laser mount and the distance between the two cylindrical lenses such that the long axis of the elliptical laser beam profile is horizontal, i.e., parallel to the flow direction.

  4. 75

    Place two beam splitters to split the laser beam for the optical interrogations at OI1, OI3, and OI4.

  5. 76

    Place pairs of concave lenses in the beam paths for OI1 and OI3 so that the divergence of the two beams becomes identical.

  6. 77

    Direct the two beams for OI1 and OI3 such that they co-propagate with an angle of 2° and spatially overlap at the back aperture of the objective lens A2. Direct the beam for OI4 so that it has an elliptical focus at the sort point.

  7. 78

    Install dichroic mirrors before and after the objective lenses to combine and separate the beams for the speed meter (OI1 and OI3) and the excitation beam array (OI2).

  8. 79

    Place a cylindrical lens to focus the two beams in the vertical direction. Make sure that both beams are focused in the vertical direction and overlap at the back aperture of the objective lens A2.

  9. 80

    Place a cylindrical lens, flat mirrors, lenses, and APDs to couple the output beams to the APDs. Make sure that the beam profiles are spherical when they are coupled to the APDs.

  10. 81

    Install annular aperture obstruction targets in front of the APDs so that they block the unscattered beams, but permit the scattered light to reach the APDs.

  11. 82

    Install an oscilloscope and connect its channels to the outputs of the APDs.

Construction of the monitoring optics

Timing 1–3 h

  1. 83

    Install two LED light sources and a high-speed CMOS camera on the optics table.

  2. 84

    Place and align mirrors and lenses to collimate and direct each LED light into each objective lens (A2 or B2) for the FDM microscope and speed meter. Each objective lens focuses each LED light at the sort point and the optical interrogation point in the microchannel.

  3. 85

    Collect the transmitted LED lights via the objective lenses, which are shared with the FDM microscope and speed meter.

  4. 86

    Place mirrors to direct the transmitted LED lights toward the high-speed CMOS camera.

  5. 87

    Place tube lenses to focus and project the transmitted LED lights onto the active area of the high-speed CMOS camera.

  6. 88

    Insert a flipper mirror to change images of the sorting point and the optical interrogation point that are projected on the high-speed CMOS camera.

Construction of the IC node in the image processor

Timing 5–7 d

  1. 89

    Develop an image acquisition program for the IC node, using LabVIEW. Use a sample program provided by the manufacturer for the configuration and operation of the digitizer. The front panel of the program should have monitor outputs of the raw waveform data collected from the digitizer and its power spectrum, which is generated by a built-in function of fast Fourier transformation. In addition, the front panel of the program should include monitor outputs of constructed images.

  2. 90

    Add an ID number readout function and a function for image data transfer via Ethernet with UDP to the image acquisition program.

    Critical step

    UDP is preferred for the data transfer protocol. Disable the monitor outputs of constructed images when the data transfer is activated in order to maximize the data-processing speed.

  3. 91

    Verify the whole operation of the image acquisition program using a test input signal generated by either the arbitrary waveform generator or the function generator.

Construction of the IA node in the image processor

Timing 30–60 d

  1. 92

    Prepare a computer equipped with high-speed multicore CPUs, GPUs (if needed), and 10-Gbps Ethernet network interfaces.

    Critical step

    Make sure that the memory and the number of the CPU cores are enough to handle the designed throughput.

  2. 93

    Install an image-processing library such as OpenCV and a deep-learning software program(s) such as PyTorch99, Keras97, TensorFlow96, or Chainer100.

  3. 94

    Develop a real-time image-processing program. The typical program used in our setup is available in Supplementary Data 3.

    Critical step

    Implement a multi-thread execution function of the image analysis and sorting decision so that the total image-processing times satisfy the sort latency limitation. To check actual total image-processing times of a sort experiment, implement a recording function of the total image-processing times.

  4. 95

    Verify that all operations work properly. Debug the program if needed.

Construction of the TM node and sort driver in the image processor

Timing 30–60 d

  1. 96

    Prepare an ADC board and an FPGA board for the TM node. If the types of connectors on the digitizer and FPGA boards are different, an interposer board to adapt their connections should also be prepared.

  2. 97

    Prepare another set of a digital-to-analog converter board and an FPGA board for the sort driver.

  3. 98

    Prepare a computer to program and control the TM node and sort driver.

  4. 99

    Connect all the components to build the TM node and sort driver. Typically, digital signals such as sort trigger, image trigger, and image ID signals are transmitted by SMA cables between the TM node and the sort driver. Connect the USB ports of the computer to the USB (universal asynchronous receiver–transmitter (UART)) ports of the FPGA boards via USB cables.

  5. 100

    Download and install an FPGA development software program on the computer. The software and its version are designated by the FPGA boards.

  6. 101

    Develop a signal-processing program for the FPGA board in the TM node. The typical program used in our setup is available in Supplementary Data 4. The functions of the event detection, cell ID assignment, speed calculation, sort timing calculation, network transmission, and sort trigger signal generation should be implemented.

    Critical step

    Timing parameters such as the sort latency and the distance between the optical interrogation points should be variable. Appropriate timing parameters need to be set up on the computer.

  7. 102

    Develop a waveform-generation program for the FPGA board in the sort driver. The waveform should be designed for sort actuations via the electrical amplifiers. A typical waveform is square shaped with a rise/fall time.

    Critical step

    Waveform parameters such as the amplitude, rise time, and fall time should be variable to optimize the operation of the dual-membrane push–pull cell sorter.

  8. 103

    Develop a control program to perform parameter setting for the FPGA boards. Typically, parameters and control commands are transmitted from the computer to the FPGA boards through the USB (UART) port. The parameter settings are stored in the registers in the programmed FPGA boards.

  9. 104

    Synthesize and implement the developed programs via the FPGA development software program.

  10. 105

    Download the developed programs into configuration devices on the FPGA boards. Use the FPGA development software program to transfer and store the configuration data in flash memory devices on the FPGA boards.

  11. 106

    Connect a function generator to the ADC board and provide artificial event signals.

  12. 107

    Verify that the whole system works properly.

    Critical step

    Make sure that the parameters of the artificial event signals (e.g., the width of the pulse signals, the frequency of the signal events, the time interval between successive events) meet the requirements of sorting experiments.

Construction of the optics–microfluidics integration unit

Timing 7–14 d

  1. 108

    Design the optics–microfluidics integration unit with the aid of a computer-aided design software program such as SolidWorks. The design of the optics–microfluidics integration unit used in our iIACS machine is available in Supplementary Data 2 and presented in Fig. 13.

  2. 109

    Submit the design to a machine shop to fabricate components for the optics–microfluidics integration unit.

  3. 110

    Assemble the components, commercially available mechanical stages, and objective lenses into the optics–microfluidics integration unit.

    Critical step

    Make sure that the objective lenses and a side of the base plate are placed in parallel. Similarly, make sure that the slot and another side of the base plate are placed in parallel.

  4. 111

    Install the optics–microfluidics integration unit on the optics table.

Integration of the liquid pump, microfluidic chip, and optics–microfluidics integration unit

Timing 3–6 h

  1. 112

    Connect the tubing of the liquid pump to the chip holder of the optics–microfluidics integration unit.

  2. 113

    Set the microfluidic chip onto the chip holder.

    Critical step

    Make sure that the inlet and outlet holes of the microfluidic chip are aligned with the corresponding holes of the chip holder.

    Critical step

    Be sure to place O-rings at the inlets and outlets of the microfluidic chip.

  3. 114

    Insert the chip holder into the slot of the optics–microfluidics integration unit.

    troubleshooting

Integration of the IC, IA, and TM nodes and the sort driver

Timing 3–6 h

  1. 115

    Prepare a 10-Gbps Ethernet switch, 10-Gbps Ethernet cables, and SMA cables.

  2. 116

    Connect the TM node, sort driver, and IC node via SMA cables.

  3. 117

    Connect the IC node, IA node, TM node, and sort driver via the 10-Gbps Ethernet switch.

    Critical step

    Be sure to use category 6A cables for 10-Gbps ports.

Characterization of the liquid pump

Timing 3–6 h

  1. 118

    Fill the microchannels, including the sheath channels, sample channel, and bubble-releasing channel, with liquid (e.g., DI water, commercially available sheath fluid for conventional FACS).

  2. 119

    Put the sample liquid (DI water) into the sample tube and flow the liquids (two sheath solutions and the sample liquid) in the automatic control mode.

    troubleshooting

  3. 120

    Check the stability of the flow. If needed, tune the PID parameters to achieve a stable flow.

    Critical step

    Make sure that the coefficients of variation (CVs) of the flow speed and the sample flow rate are <1% and <10%, respectively.

Alignment and characterization of the monitoring optics

Timing 1–3 h

  1. 121

    Turn on the LED light sources and the high-speed CMOS camera.

  2. 122

    Adjust the mirrors and lenses to monitor flows in the microchannels, using the high-speed CMOS camera.

Characterization of the hydrodynamic and acoustic cell focusers

Timing 1–3 h

  1. 123

    Prepare a sample tube that contains 1-µm particles (typically with a high concentration (~109 particles/mL) in DI water).

  2. 124

    Flow the particles at a flow rate ratio of sample/sheath1/sheath2 = 1:16:40 (a fine-focusing mode). Check the performance of the hydrodynamic cell focuser (Fig. 7).

    troubleshooting

  3. 125

    Flow the particles at a flow rate ratio of sample/sheath1/sheath2 = 3:16:40 to visualize the effect of acoustic cell focusing.

  4. 126

    Activate the acoustic cell focuser and tune the amplitude and frequency of the sinusoidal driving signal for the focuser. In our iIACS machine1, the amplitude and frequency are 3.66 MHz and 142 Vpp, respectively.

  5. 127

    Check the performance of the acoustic cell focuser (Fig. 7).

    troubleshooting

  6. 128

    Flush the microchannels with sheath solutions and stop the liquid flows.

Characterization of the cell sorter

Timing 0.5–1 h

  1. 129

    (Optional) Activate the acoustic cell focuser.

    Critical step

    The acoustic cell focuser is necessary if the position of the flowing cells cannot be maintained at the center of the microchannel until they arrive at the sort point. Without the acoustic cell focuser, large cells (>15 µm) or non-spherically shaped cells (e.g., elliptically shaped cells) are generally difficult to maintain at the center of the microchannel because of gravity, inertial forces, and other factors.

  2. 130

    Prepare a sample tube that contains 1-µm particles (typically with a high concentration (~109 particles/mL) in DI water).

  3. 131

    Flow the particles in the fine-focusing mode (Step 124).

  4. 132

    Set the input signals for actuating the sorter and activate the sorting actuators.

    Critical step

    Use artificial internal trigger signals instead of event signals of the flowing particles because the event rate of highly concentrated particles can be too high for operating the cell sorter. In our iIACS machine1, the frequency of the artificial internal trigger signals is 100 Hz.

  5. 133

    Record videos of particle sorting with the high-speed CMOS camera and check the performance of the sorting actuators (Fig. 12).

    troubleshooting

  6. 134

    Flush the microchannels with sheath solutions and stop the liquid flows.

Alignment and characterization of the speed meter

Timing 1–3 h

  1. 135

    Turn on the laser, APDs, and oscilloscope.

    Caution

    The laser output is strong enough to damage eyes. Do not look into the laser. Wear laser safety glasses to protect your eyes whenever the laser is on.

  2. 136

    Adjust the mirrors to direct the beams for OI1 and OI3 such that the beam spots at OI1 and OI3 are centered in the microchannel in the vertical direction and are equidistant from the excitation beam spots.

    Critical step

    Monitor the elliptical spots OI1 and OI3 with the high-speed CMOS camera. If needed, use a color filter in front of the high-speed CMOS camera to prevent the camera from saturating.

  3. 137

    Adjust the positions of the lens pairs for controlling the beam divergence along the beam path such that the beam waist in the flow direction is minimized.

  4. 138

    Place mirrors and a lens to image the sorting point in the microchannel with the high-speed CMOS camera. Monitor the elliptical laser spot at OI4.

  5. 139

    Adjust the mirrors in the path of the OI4 beam such that the beam spot at OI4 is centered in the microchannel in the vertical direction.

    Critical step

    The angle between the microfluidic chip and the optical axis can be adjusted if the long axis of the beam spot is not perpendicular to the flow direction.

  6. 140

    Prepare a sample tube that contains standard calibration particles.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  7. 141

    Flow the particles in the microchannel.

  8. 142

    Observe the peaks of forward-scattered light from the flowing particles at OI1, OI3, and OI4 with the oscilloscope (Fig. 10). Verify that the waveform obtained at OI1 is roughly identical to those at OI3 and OI4, except for the time shift due to the latency between the particle arrival times at OI1 and OI3.

  9. 143

    Readjust the positions of the lens pairs for controlling the beam divergence along the beam path such that the peak widths in the waveforms measured at OI1 and OI3 are minimized.

  10. 144

    Adjust the position of the objective lens that focuses the OI4 beam along the beam path in order to minimize the peak width in the waveform measured at OI4.

  11. 145

    Adjust the positions of the spatial masks in front of the APDs such that the ratios of the peak heights to the background are maximized.

  12. 146

    Adjust the gain of the APDs such that the heights of peaks measured at OI1, OI3, and OI4 are identical.

  13. 147

    Flush the microchannel with sheath solutions and stop the liquid flows.

Alignment and characterization of the FDM microscope

Timing 1–3 h

  1. 148

    Turn on the laser, the amplifiers for the driving signals to the AODs, and the APDs.

    Caution

    The laser output is strong enough to damage eyes. Do not look into the laser. Wear laser safety glasses to protect your eyes whenever the laser is on.

  2. 149

    Generate multitoned driving signals using the waveform-generation program and apply them to the AODs.

  3. 150

    Adjust the position of the microfluidic chip in the focal plane of the objective lens A2 such that the laser beam for the FDM microscope illuminates the center of the microchannel.

    Critical step

    Check the laser position with the high-speed CMOS camera.

  4. 151

    Monitor the power spectrum of the excitation beam, using the image acquisition program. Maximize the powers of the desired frequency components and minimize unwanted cross-talk by adjusting the positions and angles of the AODs and mirrors between AOD1 and the polarizing beam splitter.

    Critical step

    Make sure that the microchannel is full of liquid; otherwise, the alignment will not be effective for flowing particles.

    troubleshooting

  5. 152

    Save an averaged power spectrum for calibrating images.

  6. 153

    Prepare a sample tube that contains fluorescent particles of an appropriate emission spectrum that can be detected by the FDM microscope.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  7. 154

    Flow the particles in the microchannel.

  8. 155

    Monitor bright-field and fluorescence images of the flowing particles, using the image acquisition program.

  9. 156

    Adjust the positions of the microfluidic chip and the APDs such that in-focus images of the particles appear at the center of the field of view.

    Critical step

    The gains of the APDs for the fluorescence image acquisition should be adjusted so that the waveforms fit within the dynamic range of the digitizer.

  10. 157

    Prepare a sample tube that contains standard 6-peak fluorescent particles.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  11. 158

    Flow the particles in the fine-focusing mode and measure the detection sensitivity, dynamic range, and linearity of the FDM microscope (Fig. 9).

  12. 159

    Prepare a sample tube that contains fluorescent calibration particles of various emission spectra.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  13. 160

    Flow the particles in the fine-focusing mode and evaluate the fluorochrome separation of the FDM microscope (Fig. 9).

  14. 161

    Prepare a sample tube that contains standard calibration particles of various sizes.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  15. 162

    Flow the particles in the fine-focusing mode and evaluate the spatial resolution of the FDM microscope (Fig. 9).

  16. 163

    Prepare a sample tube that contains cells of various types and sizes.

    Critical step

    The typical concentration of cells is ~106cells/mL (in culture medium). To check the quality of the images of the FDM microscope, it is recommended to use not only the cells that will be used for applications, but also cells of various types and sizes. Microalgal cells are useful because a staining process is unnecessary because of their autofluorescence. Moreover, microalgal cells are diverse in size. Separately preparing several kinds of microalgal cells makes it easy to check the spatial resolution of the images.

  17. 164

    Flow the cells in the fine-focusing mode and evaluate the image quality of the FDM microscope (Fig. 16)

  18. 165

    Flush the microchannel with sheath solutions and stop the liquid flows.

Development of classic image analysis algorithms

Timing 1–7 d

Critical

As long as the total signal-processing time fits within the sort latency, it is possible to choose a classifier based on any image analysis algorithm. In this protocol, the development of classic image analysis algorithms and deep CNN classifiers is described. If a deep CNN classifier is chosen, skip the section below (Steps 166–179) and go to Step 180.

  1. 166

    Prepare a sample tube that contains fluorescent particles.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  2. 167

    Flow the particles in the fine-focusing mode (Step 124).

  3. 168

    Check the image quality of the FDM microscope. Realign the FDM microscope if needed.

  4. 169

    Acquire and save several thousands of FDM-microscope images of events.

  5. 170

    Flush the microchannel with sheath solutions and stop the liquid flows.

  6. 171

    Use an image-processing software program such as OpenCV, ImageJ, MATLAB, or CellProfiler101.

  7. 172

    Adjust the aspect ratio if the shape of the pixel is not a square.

  8. 173

    Apply a filter function such as a Gaussian filter and median filter to remove noise.

    Critical step

    Choose a linear filter to reduce the execution time. Nonlinear filters such as bilateral filters increase the processing time.

  9. 174

    Develop a binary image mask from bright-field images of the particles.

  10. 175

    Extract morphological features of the particles, such as area, perimeter, shape, and intensity information, including the averages and standard deviations of the bright-field and fluorescence signals.

  11. 176

    Perform data processing to draw scatter plots or histograms, using a statistical analysis software program such as R or MATLAB to classify the particles.

    Critical step

    Remove anomalous images (e.g., cluttered images) by placing a gate on the scatter plots with appropriate features such as area and shape.

  12. 177

    Choose suitable features to classify target particles. Use machine learning to classify target particles if multi-dimensional nonlinear classification is required.

    Critical step

    If it is difficult to construct a classification algorithm from the combination of the features based on the classic image analysis, try to construct a classification algorithm based on a deep CNN.

  13. 178

    Define a sort-decision function to maximize sort accuracy.

  14. 179

    Implement the developed algorithm with the sort-decision function as follows: code it with an appropriate image-processing library, using an appropriate language. In our iIACS machine1, OpenCV and C++ were used as a library and language, respectively.

(Optional) Development of deep CNN classifiers

Timing 1–7 d

Critical

Images can also be processed using classic image analysis algorithms (Steps 166179), in which case the section below (Steps 180189) can be skipped.

  1. 180

    Prepare several thousands of FDM-microscope images by performing Steps 166–170 to train and validate deep CNN classifiers.

    Critical step

    Be sure to collect images from multiple experiments to cover a sufficiently large degree of variation. The background and artifacts of the acquired images may vary day by day or experiment by experiment because of, e.g., cell samples and optical settings. It is important to ensure a variety of images to appropriately train the classifiers without overfitting for a certain condition on the basis of an individual experiment.

  2. 181

    Perform Steps 171–173 to perform image preprocessing.

  3. 182

    Construct class labels as the ground truths by manually classifying the FDM-microscope images.

  4. 183

    Separate the images into two datasets: a training dataset and a validation dataset.

    Critical step

    Remove anomalous images (e.g., cluttered images).

  5. 184

    Install a deep-learning software program(s) into a computer.

    Critical step

    Use the same software program(s) and the same version of the deep-learning software program(s) as were installed in the IA node for easy implementation of the deep CNN classifiers into the IA node. Install and use an additional software program(s) for building model architectures of the CNN classifiers if needed. In our iIACS machine1, model architectures were built using a combination of Python and Keras, whereas the CNN classifiers were trained, validated, and tested using TensorFlow.

  6. 185

    Build an appropriate model architecture of a CNN classifier (see our CNN-classifier codes and raw data, available at http://www.goda.chem.s.u-tokyo.ac.jp/intelligentIACS/software.zip).

    Critical step

    Use the He normal distribution as an initializer if TensorFlow was used in Step 185.

  7. 186

    Train the CNN classifier using the training dataset.

    Critical step

    Be sure to flip and/or rotate images during the training to increase the variation of the training dataset. Balance the number of images across each class.

  8. 187

    Tune the model parameters of the CNN classifier to maximize the classification accuracy computed using the validation dataset.

    Critical step

    Do not use the training dataset for tuning the model parameters.

  9. 188

    Add a sort-decision function to the CNN classifier.

  10. 189

    Implement the CNN classifier with the sort-decision function on the IA node.

Characterization of the image processor

Timing 1–7 d

  1. 190

    Prepare a sample tube that contains fluorescent particles.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  2. 191

    Flow the particles in the fine-focusing mode (Step 124).

  3. 192

    Activate all sort operations on the intelligent real-time image processor as follows: activate the IA node with the implemented sort decision algorithm, turn on the image data transfer of the IC node, and activate the sort signal output.

  4. 193

    Save all data recorded by the IA node.

  5. 194

    Flush the microchannel with sheath solutions and stop the liquid flows.

  6. 195

    Check the total image-processing time. Debug if needed.

    troubleshooting

  7. 196

    Check the sequence of the recorded cell IDs. Debug if needed.

    troubleshooting

  8. 197

    Check the sort decision accuracy. Debug if needed.

    Critical step

    Apply the image-processing algorithm/classifier with the sort-decision function in an offline manner and compare the sort decision accuracy with that in the characterization experiment.

    troubleshooting

(Optional) Construction of the glass-bottom chamber

Timing 3–4 d

Critical

This procedure (Steps 198–203) can be used for a large-volume (typically >300 µL) sample, which is too much to load into a commercial glass-bottom chamber.

  1. 198

    Pour PDMS solution into a hollow-cylinder-shaped mold (o.d., 12 mm; i.d., 8 mm; length, 40 mm; made with SUS304 (or SAE 304) stainless steel), followed by curing for 1 h at 80 °C.

  2. 199

    Deburr the cured PDMS.

  3. 200

    Extract un-cross-linked oligomers from the cured PDMS in n-hexane for 24 h.

  4. 201

    Rinse the PDMS with acetone and methanol for 24 h each, and dry the PDMS.

  5. 202

    Wash a microscopy-grade coverslip with a sonicator in a mixture of organic solvents (acetone/isopropanol = 9:1 (vol/vol)).

  6. 203

    Bond the cylindrical PDMS with the coverslip by air plasma (5 mA, 90 s), followed by incubation at 80 °C for 30 min.

    Pause point

    The glass-bottom chamber can be stored in a desiccator at room temperature until use.

Characterization of the iIACS machine

Timing 1–7 d

  1. 204

    Prepare a sample tube that contains fluorescent particles.

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  2. 205

    Flow the particles in the fine-focusing mode (Step 124).

  3. 206

    Activate all sort operations on the image processor as follows: activate the IA node with the implemented sort decision algorithm, turn on the image data transfer on the IC node, and activate the sort signal output.

  4. 207

    Check the event rate (Fig. 14), FDM microscope image quality, flow stability, and total image-processing time (Fig. 15).

  5. 208

    Optimize the settings for the sort latency prediction by projecting the relation between the travel times of each particle from OI1 to OI3 and from OI1 to OI4 (Fig. 10). The slope of the linear regression of the projected relation is the parameter for the prediction.

    troubleshooting

  6. 209

    Record videos of the sorting process with the high-speed CMOS camera to check the actuation performance of the cell sorter and the accuracy of the sort timing (Fig. 15). Trigger the record function of the high-speed CMOS camera, using the sort signals to synchronize the record timing with the sort timing.

    troubleshooting

  7. 210

    Abandon the initial 1.5-min sample to avoid unwanted carryovers.

  8. 211

    Set collection and waste tubes and wait for ~1 min.

  9. 212

    (Optional) Record videos of the sorting process with the high-speed CMOS camera. Watch the videos to check the efficiency of sorting (Fig. 15).

  10. 213

    Remove the tubes.

  11. 214

    Flush the microchannel with sheath solutions and stop the liquid flows.

  12. 215

    Load the collection tube into the glass-bottom chamber and centrifuge it at 300g for 10 min at room temperature to collect the particles at the bottom of the chamber with swing-bucket rotors.

  13. 216

    Scan the whole viewing area under a commercially available fluorescence microscope equipped with a digital CMOS camera and a 20× objective lens.

  14. 217

    Perform image analysis to count particles with a commercially available software program (e.g., NIS-Elements AR) (Fig. 15) to evaluate the sort yield and purity.

  15. 218

    (Optional) When the liquid in the collection tube is too much to load into the commercial glass-bottom chamber, use a custom glass-bottom chamber (Steps 198–203). The chamber should have a volume of 1 mL and a viewing area of 8 mm in diameter, and is fabricated by bonding a glass substrate and a cylindrical structure made of PDMS.

    Critical step

    Incubate the chamber with blocking solution (1% (wt/vol) BSA in PBS solution, sterilized with a 0.22-µm filter) before loading the collection tube in order to avoid non-specific binding of the particles to PDMS.

Cell sample preparation

Timing Variable

  1. 219

    Follow standard sample preparation procedures used in conventional FACS14,17,19,102.

Cell sorting experiment

Timing 1–6 h

Critical

See the flowchart shown in Fig. 17 for a schematic overview of a typical cell sorting experiment.

  1. 220

    Repeat Steps 166–170, but with cells, to acquire images of events (e.g., single cells, cell clusters, cell debris, dust particles).

    Critical step

    Flow the sample in the boost mode (sample/sheath1/sheath2 = 3:16:40) for 2 min if the event rate is too low in the fine-focusing mode to acquire a sufficient number of images to develop an image analysis algorithm.

  2. 221

    Process the images using an appropriate method. Typically, image-processing methods can be classified into two categories: classic image analysis (Steps 166–179) and CNN-based analysis (Steps 180–189), but it is also possible to use other image analysis algorithms, as long as the total image-processing time fits within the sort latency.

  3. 222

    Repeat Steps 171–179 (with particles replaced by cells) or Steps 181–189 (with particles replaced by cells) to develop a classic image analysis algorithm or a deep CNN classifier, respectively.

    Critical step

    Image-processing algorithms vary, depending on the cellular species. Also, image-processing algorithms for cells should be very different from the algorithms for particles. Develop an algorithm for the cells under test carefully.

  4. 223

    Flow fluorescent particles in the fine-focusing mode (Step 124).

    Critical step

    The concentration of the particles should be ~106 particles/mL (in DI water) or higher.

  5. 224

    Check the FDM microscope image quality and stability of the flow rates.

  6. 225

    Repeat Steps 204–214 (with particles replaced by cells) to sort events.

    Critical step

    Flow the sample in the boost mode for 2 min if the event rate is too low in the fine-focusing mode (Step 124). The duration of the cell sorting run depends on the experiment.

Analysis of sorted and/or unsorted events

Timing 2–4 h

  1. 226

    Repeat Steps 215–218 (with particles replaced by cells).

    Critical step

    If the custom glass-bottom chamber is used, incubate the chamber with blocking solution (1% (wt/vol) BSA in PBS solution, sterilized with a 0.22-µm filter) before loading the sample in order to avoid non-specific binding of cells to the PDMS.

Troubleshooting

Troubleshooting advice can be found in Table 2.

Table 2 Troubleshooting table

Timing

Assuming that all components are readily available, a research team of six to ten members with experience in optics, electronics, digital signal processing, microfluidics, mechatronics, and flow cytometry can complete this protocol within 3 months. The throughput of the iIACS setup described here is determined by the processing time on the LabVIEW program right after the image acquisition. This limitation can be overcome by using a hardware-based (such as FPGAs) image construction scheme, whereas a software-based (such as LabVIEW) image reconstruction scheme is easier to adjust by modifying its algorithm. All construction steps (Steps 15–20, Steps 21–39, Steps 40–46, Steps 47–71, Steps 72–82, Steps 83–88, Steps 89–91, Steps 92–95, Steps 96-107, and Steps 108-111) can be performed concurrently.

  • Steps 1–12, system design: 30–60 d

  • Steps 13 and 14, preparation of components: 30–60 d

  • Steps 15–20, construction of the liquid pump: 7–14 d

  • Steps 21–39, construction of the microfluidic chip: 7–14 d

  • Steps 40–46, construction of the cell sorter and hydrodynamic and acoustic cell focusers: 1–3 d

  • Steps 47–71, construction of the FDM microscope: 1–3 d

  • Steps 72–82, construction of the speed meter: 1–3 d

  • Steps 83–88, construction of the monitoring optics: 1–3 h

  • Steps 89–91, construction of the IC node in the image processor: 5–7 d

  • Steps 92–95, construction of the IA node in the image processor: 30–60 d

  • Steps 96–107, construction of the TM node and sort driver in the image processor: 30–60 d

  • Steps 108–111, construction of the optics–microfluidics integration unit: 7–14 d

  • Steps 112–114, integration of the liquid pump, microfluidic chip, and optics–microfluidics integration unit: 3–6 h

  • Steps 115–117, integration of the IC, IA, and TM nodes and the sort driver: 3–6 h

  • Steps 118–120, characterization of the liquid pump: 3–6 h

  • Steps 121 and 122, alignment and characterization of the monitoring optics: 1–3 h

  • Steps 123–128, characterization of the hydrodynamic and acoustic cell focusers: 1–3 h

  • Steps 129–134, characterization of the cell sorter: 0.5–1 h

  • Steps 135–147, alignment and characterization of the speed meter: 1–3 h

  • Steps 148–165, alignment and characterization of the FDM microscope: 1–3 h

  • Steps 166–179, development of classic image analysis algorithms: 1–7 d

  • Steps 180–189, (optional) development of deep CNN classifiers: 1–7 d

  • Steps 190–197, characterization of the image processor: 1–7 d

  • Steps 198–203, (optional) construction of the glass-bottom chamber: 3–4 d

  • Steps 204–218, characterization of the iIACS machine: 1–7 d

  • Step 219, cell sample preparation: variable

  • Steps 220–225, cell sorting experiment: 1–6 h

  • Step 226, analysis of sorted and/or sorted events: 2–4 h

Anticipated results

Following the above procedure, the iIACS machine should be ready for practical sorting experiments in various settings. As mentioned above, a number of sorting experiments can be conducted with the iIACS machine, such as sorting morphologically abnormal budding yeast cells for functional genomic applications73,74,75, sorting cells with abnormal nuclear shapes, nucleus-to-cytoplasm ratios or chromosome numbers, compaction, and location for cancer detection22,23,64,65,66,67,68,69,70, sorting adherent cells and cells with localized organelles to identify their physiology76,77, and sorting microalgal cells with abnormally distributed proteins for carbon fixation71,72. Once sorting is done, the cells can be used for downstream detailed single-cell analysis. For example, single-cell RNA-seq (scRNA-seq), one of the omics analysis technologies, integrated with microfluidics has recently emerged as a powerful tool for biomedical research103,104,105, offering a more accurate interpretation of gene expression in individual cells to identify new cell types or subtypes. Shalek et al. performed scRNA-seq on mouse bone-marrow-derived dendritic cells and found heterogeneous gene expression, suggesting the presence of distinct states or a subpopulation of the bone-marrow-derived dendritic cells103. Although potential applications of iIACS are numerous, here we describe two concrete examples of the usage of the iIACS machine demonstrated in Nitta et al.1 in detail.

The first example is high-content sorting of rare C. reinhardtii mutants for studying microalgal photosynthesis (Fig. 18). C. reinhardtii is a microalgal model organism often used to study a low-CO2 inducible protein B (LCIB), which is highly conserved in algae with a biophysical carbon-concentrating mechanism (CCM). Under low CO2 conditions, LCIB normally changes its localization from dispersion in the chloroplast to a ring-like structure in the vicinity of the pyrenoid when the CCM is fully induced106. Unfortunately, a full understanding of the CCM is challenging, although the algal CCM is the primary target of engineering photosynthesis in synthetic microalgae for increasing their photosynthetic efficiency and yield. We used our iIACS machine to isolate C. reinhardtii mutants with aberrant LCIB localization under low-CO2 conditions. Specifically, strain BC-9 was transformed by random insertional mutagenesis, and rare mutants (~1% of the total population) with an aberrant CCM pattern were sorted by the iIACS machine and evaluated under a conventional fluorescence microscope. This sorting process would normally take ~6 months if it were manually performed by microscopy and pipetting, but the iIACS machine conducted it within 40 min.

Fig. 18: High-content sorting of rare Chlamydomonas reinhardtii mutants with the iIACS machine.
figure18

a, Experimental procedure. b, Scatter plot of the cells under high CO2 (blue) and low CO2 (green) conditions in the LCIB-Clover intensity and CV (left), along with images of the cells (right): representative cells extracted by gating in the yellow area (top); representative cells extracted by gating in the red area (bottom). Scale bars, 5 µm. c, Scatter plot of the total population (yellow) and the sorted population (blue). d, Colonies of the sorted mutants. e, Differential image contrast (DIC) and confocal fluorescence images of BC-9 and three representative mutants. Scale bars, 5 µm. Chl, chlorophyll. Image adapted from Nitta et al.1.

The second example is high-content sorting of platelet aggregates in human blood for studying atherothrombosis, including acute coronary syndromes and acute ischemic stroke (Fig. 19). Atherothrombosis is a disorder initiated by disrupting atherosclerotic plaques, followed by further platelet activation, resulting in local occlusion or distal embolism by resultant platelet aggregates. Although evaluation of activated platelets in peripheral blood is effective for predicting and assessing disease state, previous technologies fail to detect and isolate activated platelets accurately because platelets are sensitive to mechanical and chemical stimulation, including staining. To perform high-throughput sorting of activated platelets in fixed blood without staining, we used our iIACS machine to sort platelet aggregates on the basis of bright-field imaging and an eight-layer CNN. We successfully isolated platelet aggregates with a high specificity and sensitivity of 99.0% and 82%, respectively. This sorting process would normally take ~1 d if it were manually conducted by microscopy and pipetting, but the iIACS machine achieved it within 1 min.

Fig. 19: High-content sorting of platelet aggregates in human blood with the iIACS machine.
figure19

a, CNN model used for identifying platelet aggregates. b, Probability distributions of platelet aggregates, single platelets, and leukocytes. Scale bar, 10 µm. c, Progressive gating strategy on the CNN to sort platelet aggregates. d, Correlation matrix between the ground truth and sort decision. e,g, Processing time of the image construction, data transfer, and image analysis with and without stimulation with a thrombin receptor activator peptide (TRAP). f,h, Histogram of gated events for TRAP-stimulated and unstimulated blood samples with the sorting statistics. The blood samples for training and validating the CNN classifier were prepared from four healthy volunteers (23-year-old male, 38-year-old male, 39-year-old male, and 50-year-old male), and the blood sample for the sorting experiment was prepared from a 69-year-old healthy female volunteer. This study was approved by the Institutional Ethics Committee of Faculty of Medicine, the University of Tokyo (no. 11049-5) and was conducted according to the Declaration of Helsinki. All volunteers provided written informed consent. Image adapted from Nitta et al.1.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Data and code availability

The data and code are available as Supplementary Data and upon reasonable request.

References

  1. 1.

    Nitta, N. et al. Intelligent image-activated cell sorting. Cell 175, 266–276 (2018).

  2. 2.

    Mikami, H. et al. Ultrafast confocal fluorescence microscopy beyond the fluorescence lifetime limit. Optica 5, 117–126 (2018).

  3. 3.

    Kanno, H., Mikami, H., Kaya, Y., Ozeki, Y. & Goda, K. Simple, stable, compact implementation of frequency-division-multiplexed microscopy by inline interferometry. Opt. Lett. 44, 467–470 (2019).

  4. 4.

    Shivhare, P. K., Bhadra, A., Sajeesh, P., Prabhakar, A. & Sen, A. K. Hydrodynamic focusing and interdistance control of particle-laden flow for microflow cytometry. Microfluid. Nanofluidics 20, 86 (2016).

  5. 5.

    Park, J. W. et al. Acoustofluidic harvesting of microalgae on a single chip. Biomicrofluidics 10, 034119 (2016).

  6. 6.

    Grenvall, C., Antfolk, C., Bisgaard, C. Z. & Laurell, T. Two-dimensional acoustic particle focusing enables sheathless chip Coulter counter with planar electrode configuration. Lab Chip 14, 4629–4637 (2014).

  7. 7.

    Sakuma, S., Kasai, Y., Hayakawa, T. & Arai, F. On-chip cell sorting by high-speed local-flow control using dual membrane pumps. Lab Chip 17, 2760–2767 (2017).

  8. 8.

    Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  9. 9.

    Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. Deep Learning (MIT Press, Cambridge, 2016).

  10. 10.

    LeCun, Y. et al. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1, 541–551 (1989).

  11. 11.

    Krizhevesky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Proc. 25th International Conference on Neural Information Processing Systems (NIPS 2012) (eds Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q.) 1097–1105 (Curran Associates, 2012).

  12. 12.

    Herzenberg, L. A. et al. The history and future of the fluorescence activated cell sorter and flow cytometry: a view from Stanford. Clin. Chem. 48, 1819–1827 (2002).

  13. 13.

    Tung, J. W. et al. Modern flow cytometry: a practical approach. Clin. Lab. Med. 27, 453–468 (2007).

  14. 14.

    Liu, L., Cheung, T. H., Charville, G. W. & Rando, T. A. Isolation of skeletal muscle stem cells by fluorescence-activated cell sorting. Nat. Protoc. 10, 1612–1624 (2015).

  15. 15.

    Hayatsu, N. et al. Analyses of a mutant Foxp3 allele reveal BATF as a critical transcription factor in the differentiation and accumulation of tissue regulatory T cells. Immunity 47, 268–283 (2017).

  16. 16.

    de St Groth, B. F., Zhu, E. rhu., Asad, S. & Lee, L. Flow cytometric detection of human regulatory T cells. Methods Mol. Biol. 707, 263–279 (2011)

  17. 17.

    Shapiro, H. M. Practical Flow Cytometry (John Wiley & Sons, 2005).

  18. 18.

    Herzenberg, L. A., Gottlinger, C., Muller, W., Radbruch, A. & Recktenwald, D. Flow Cytometry and Cell Sorting (Springer, 1992).

  19. 19.

    Lindmo, T., Peters, D. C. & Sweet, R. G. Flow Cytometry and Sorting (Wiley-Liss, 1990).

  20. 20.

    Kawata, S., Hori, M., Kado, H. & Tamiya, E. Biological Imaging and Sensing (Springer, 2004).

  21. 21.

    Wang, P. & Wu, C. Micro/Nano Cell and Molecular Sensors (Springer, 2016).

  22. 22.

    Boutros, M., Heigwer, F. & Laufer, C. Microscopy-based high-content screening. Cell 163, 1314–1325 (2015).

  23. 23.

    Caicedo, J. C. et al. Data-analysis strategies for image-based cell profiling. Nat. Methods 14, 849–863 (2017).

  24. 24.

    Boutros, M. & Ahringer, J. The art and design of genetic screens: RNA interference. Nat. Rev. Genet. 9, 554–566 (2008).

  25. 25.

    Carpenter, A. E. Image-based chemical screening. Nat. Chem. Biol. 3, 461–465 (2007).

  26. 26.

    Boutros, M. et al. Genome-wide RNAi analysis of growth and viability in Drosophila cells. Science 303, 832–835 (2004).

  27. 27.

    Lum, L. et al. Identification of Hedgehog pathway components by RNAi in Drosophila cultured cells. Science 299, 2039–2045 (2003).

  28. 28.

    Kiger, A. et al. A functional genomic analysis of cell morphology using RNA interference. J. Biol. 2, 27 (2003).

  29. 29.

    Liu, T., Sims, D. & Baum, B. Parallel RNAi screens across different cell lines identify generic and cell type-specific regulators of actin organization and cell morphology. Genome Biol. 10, R26 (2009).

  30. 30.

    Arpali, S. A., Arpali, C., Coskun, A. F., Chiang, H. H. & Ozcan, A. High-throughput screening of large volumes of whole blood using structured illumination and fluorescent on-chip imaging. Lab Chip 12, 4968–4971 (2012).

  31. 31.

    Zhang, Y. et al. High-throughput screening of encapsulated islets using wide-field lens-free on-chip imaging. ACS Photonics 5, 2081–2086 (2018).

  32. 32.

    Lei, C., Guo, B., Cheng, Z. & Goda, K. Optical time-stretch imaging: principles and applications. Appl. Phys. Rev. 3, 011102 (2016).

  33. 33.

    Mikami, H. et al. High-speed imaging meets single-cell analysis. Chem 4, 2278–2300 (2018).

  34. 34.

    Mikami, H., Gao, L. & Goda, K. Ultrafast optical imaging technology: principles and applications of emerging methods. Nanophotonics 5, 497–509 (2016).

  35. 35.

    Porichis, F. et al. High-throughput detection of miRNAs and gene-specific mRNA at the single-cell level by flow cytometry. Nat. Commun. 5, 5641 (2014).

  36. 36.

    Wu, J. L. et al. Ultrafast laser-scanning time-stretch imaging at visible wavelengths. Light Sci. Appl. 6, e16196 (2017).

  37. 37.

    Mahjoubfar, A. et al. Time stretch and its applications. Nat. Photonics 11, 341–351 (2017).

  38. 38.

    Lai, Q. T. K. et al. High-throughput time-stretch imaging flow cytometry for multi-class classification of phytoplankton. Opt. Express 24, 28170–28184 (2016).

  39. 39.

    Han, Y. & Lo, Y. Imaging cells in flow cytometer using spatial-temporal transformation. Sci. Rep. 5, 13267 (2015).

  40. 40.

    Han, Y., Gu, Y., Zhang, A. C. & Lo, Y. H. Review: imaging technologies for flow cytometry. Lab Chip 16, 4639–4647 (2016).

  41. 41.

    Rane, A. S., Rutkauskaite, J., DeMello, A. & Stavrakis, S. High-throughput multi-parametric imaging flow cytometry. Chem 3, 588–602 (2017).

  42. 42.

    Miura, T. et al. On-chip light-sheet fluorescence imaging flow cytometry at a high flow speed of 1 m/s. Biomed. Opt. Express 9, 3424–3433 (2018).

  43. 43.

    Jiang, Y. et al. Label-free detection of aggregated platelets in blood by machine-learning-aided optofluidic time-stretch microscopy. Lab Chip 17, 2426–2434 (2017).

  44. 44.

    George, T. C. et al. Distinguishing modes of cell death using the ImageStream® multispectcal imaging flow cytometer. Cytometry A 59A, 237–245 (2004).

  45. 45.

    Kobayashi, H. et al. Label-free detection of cellular drug responses by high-throughput bright-field imaging and machine learning. Sci. Rep. 7, 12454 (2017).

  46. 46.

    Muñoz, H. E. et al. Single-cell analysis of morphological and metabolic heterogeneity in Euglena gracilis by fluorescence-imaging flow cytometry. Anal. Chem. 90, 11280–11289 (2018).

  47. 47.

    Guo, B. et al. High-throughput, label-free, single-cell, microalgal lipid screening by machine-learning-equipped optofluidic time-stretch quantitative phase microscopy. Cytometry A 91A, 494–502 (2017).

  48. 48.

    George, T. C. et al. Quantitative measurement of nuclear translocation events using similarity analysis of multispectral cellular images obtained in flow. J. Immunol. Methods 311, 117–129 (2006).

  49. 49.

    Basiji, D. A., Ortyn, W. E., Liang, L., Venkatachalam, V. & Morrissey, P. Cellular image analysis and imaging by flow cytometry. Clin. Lab. Med. 27, 653–670 (2007).

  50. 50.

    Lee, D., Mehta, N., Shearer, A. & Kastner, R. A hardware accelerated system for high throughput cellular image analysis. J. Parallel Distrib. Comput. 113, 167–178 (2018).

  51. 51.

    Goda, K. & Jalali, B. Dispersive Fourier transformation for fast continuous single-shot measurements. Nat. Photonics 7, 102–112 (2013).

  52. 52.

    Wong, T. T. W. et al. Asymmetric-detection time-stretch optical microscopy (ATOM) for ultrafast high-contrast cellular imaging in flow. Sci. Rep. 4, 3656 (2014).

  53. 53.

    Lau, A. K. S., Shum, H. C., Wong, K. K. Y. & Tsia, K. K. Optofluidic time-stretch imaging-an emerging tool for high-throughput imaging flow cytometry. Lab Chip 16, 1743–1756 (2016).

  54. 54.

    Lei, C. et al. High-throughput imaging flow cytometry by optofluidic time-stretch microscopy. Nat. Protoc. 13, 1603–1631 (2018).

  55. 55.

    Guo, B. et al. Optofluidic time-stretch quantitative phase microscopy. Methods 136, 116–125 (2018).

  56. 56.

    Goda, K. et al. High-throughput single-microparticle imaging flow analyzer. Proc. Natl. Acad. Sci. USA 109, 11630–11635 (2012).

  57. 57.

    Lei, C., Nitta, N., Ozeki, Y. & Goda, K. Optofluidic time-stretch microscopy: recent advances. Opt. Rev. 25, 464–472 (2018).

  58. 58.

    Lei, C. et al. GHz optical time-stretch microscopy by compressive sensing. IEEE Photonics J. 9, 1–8 (2017).

  59. 59.

    Hiraki, K. et al. All-IP-Ethernet architecture for real-time sensor-fusion processing. In Proc. SPIE BiOS 9720 97200D (2016). https://doi.org/10.1117/12.2212016

  60. 60.

    Inaba, M. & Hiraki, K. Network processing hardware. In Proc. Second Asian International Conference on Technologies for Advanced Heterogeneous Network (eds Cho, K. & Jacquet, P.) 103–112 (Springer, 2006).

  61. 61.

    Okada, K. et al. Protocol design for all-IP computer architecture. In Proc. International Conference on Information Networking 2008 (ICOIN2008) (eds Kaiser, B., Madden, S. & Suri, S.) 1–5 (IEEE, 2008).

  62. 62.

    Hao, N., Budnik, Ba & Gunawardena, J. Tunable signal processing through modular control of transcription factor translocation. Science 339, 460–464 (2013).

  63. 63.

    Von Erlach, T. C. et al. Cell-geometry-dependent changes in plasma membrane order direct stem cell signalling and fate. Nat. Mater. 17, 237–242 (2018).

  64. 64.

    Sarioglu, A. F. et al. A microfluidic device for label-free, physical capture of circulating tumor cell clusters. Nat. Methods 12, 685–691 (2015).

  65. 65.

    Moor, A. E. et al. Global mRNA polarization regulates translation efficiency in the intestinal epithelium. Science 357, 1299–1303 (2017).

  66. 66.

    Zenker, J. et al. A microtubule-organizing center directing intracellular transport in the early mouse embryo. Science 357, 925–928 (2017).

  67. 67.

    Pernas, L., Bean, C., Boothroyd, J. C. & Scorrano, L. Mitochondria restrict growth of the intracellular parasite Toxoplasma gondii by limiting its uptake of fatty acids. Cell Metab. 27, 886–897 (2018).

  68. 68.

    Cho, E. H. et al. Characterization of circulating tumor cell aggregates identified in patients with epithelial tumors. Phys. Biol. 9, 016001 (2012).

  69. 69.

    Aceto, N. et al. Circulating tumor cell clusters are oligoclonal precursors of breast cancer metastasis. Cell 158, 1110–1122 (2014).

  70. 70.

    Molnar, B., Ladanyi, A., Tanko, L., Sréter, L. & Tulassay, Z. Circulating tumor cell clusters in the peripheral blood of colorectal cancer patients. Clin. Cancer Res. 7, 4080–4085 (2001).

  71. 71.

    Wang, L. et al. Chloroplast-mediated regulation of CO2-concentrating mechanism by Ca2+-binding protein CAS in the green alga Chlamydomonas reinhardtii. Proc. Natl. Acad. Sci. USA 113, 12586–12591 (2016).

  72. 72.

    Mackinder, L. C. M. et al. A spatial interactome reveals the protein organization of the algal CO2-concentrating mechanism. Cell 171, 133–147 (2017).

  73. 73.

    Ohnuki, S. & Ohya, Y. High-dimensional single-cell phenotyping reveals extensive haploinsufficiency. PLoS Biol. 16, 1–23 (2018).

  74. 74.

    Suzuki, G. et al. Global study of holistic morphological effectors in the budding yeast Saccharomyces cerevisiae. BMC Genomics 19, 149 (2018).

  75. 75.

    Iwaki, A., Ohnuki, S., Suga, Y., Izawa, S. & Ohya, Y. Vanillin inhibits translation and induces messenger ribonucleoprotein (mRNP) granule formation in Saccharomyces cerevisiae: application and validation of high-content, image-based profiling. PLoS ONE 8, e61748 (2013).

  76. 76.

    Treiser, M. D. et al. Cytoskeleton-based forecasting of stem cell lineage fates. Proc. Natl. Acad. Sci. USA 107, 610–615 (2010).

  77. 77.

    Thery, M. et al. Anisotropy of cell adhesive microenvironment governs cell internal organization and orientation of polarity. Proc. Natl. Acad. Sci. USA 103, 19771–19776 (2006).

  78. 78.

    Wu, C. Y. et al. Shaped 3D microcarriers for adherent cell culture and analysis. Microsyst. Nanoeng. 4, 21 (2018).

  79. 79.

    Lancaster, M. A. & Knoblich, J. A. Organogenesis in a dish: modeling development and disease using organoid technologies. Science 345, 1247125 (2014).

  80. 80.

    Orange, J. S. Formation and function of the lytic NK-cell immunological synapse. Nat. Rev. Immunol. 8, 713–725 (2008).

  81. 81.

    Dustin, M. L., Chakraborty, A. K. & Shaw, A. S. Understanding the structure and function of the immunological synapse. Cold Spring Harb. Perspect. Biol. 2, a002311 (2010).

  82. 82.

    Ingham, P. W. The molecular genetics of embryonic pattern formation in Drosophila. Nature 335, 25–34 (1988).

  83. 83.

    Mullins, M. C., Hammerschmidt, M., Haffter, P. & Nüsslein-Volhard, C. Large-scale mutagenesis in the zebrafish: in search of genes controlling development in a vertebrate. Curr. Biol. 4, 189–202 (1994).

  84. 84.

    Fabritius, A. et al. Imaging-based screening platform assists protein engineering. Cell Chem. Biol. 25, 1554–1561 (2018).

  85. 85.

    Környei, Z. et al. Cell sorting in a Petri dish controlled by computer vision. Sci. Rep. 3, 1–10 (2013).

  86. 86.

    Das, A. et al. Adaptive from innate: human IFN-γ+ CD4+ T cells can arise directly from CXCL8-producing recent thymic emigrants in babies and adults. J. Immunol. 199, 1696–1705 (2017).

  87. 87.

    Jin, A. et al. A rapid and efficient single-cell manipulation method for screening antigen-specific antibody-secreting cells from human peripheral blood. Nat. Med. 15, 1088–1092 (2009).

  88. 88.

    Yoshimoto, N. et al. An automated system for high-throughput single cell-based breeding. Sci. Rep. 3, 1191 (2013).

  89. 89.

    Dura, B. et al. Longitudinal multiparameter assay of lymphocyte interactions from onset by microfluidic cell pairing and culture. Proc. Natl. Acad. Sci. USA 113, E3599–E3608 (2016).

  90. 90.

    Ogunniyi, A. O., Story, C. M., Papa, E., Guillen, E. & Love, J. C. Screening individual hybridomas by microengraving to discover monoclonal antibodies. Nat. Protoc. 4, 767–782 (2009).

  91. 91.

    Yao, X. et al. Tumor cells are dislodged into the pulmonary vein during lobectomy. J. Thorac. Cardiovasc. Surg. 148, 3224–3231 (2014).

  92. 92.

    Piatkevich, K. D. et al. A robotic multidimensional directed evolution approach applied to fluorescent voltage reporters. Nat. Chem. Biol. 14, 352–360 (2018).

  93. 93.

    Brasko, C. et al. Intelligent image-based in situ single-cell isolation. Nat. Commun. 9, 1–7 (2018).

  94. 94.

    Grys, B. T. et al. Machine learning and computer vision approaches for phenotypic profiling. J. Cell Biol. 216, 65–71 (2017).

  95. 95.

    Hennig, H. et al. An open-source solution for advanced imaging flow cytometry data analysis using machine learning. Methods 112, 201–210 (2017).

  96. 96.

    Abadi, M. et al. Tensorflow: a system for large-scale machine learning. In Proc. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16) 265–283 (USENIX, 2016).

  97. 97.

    Chollet, F. Keras: the Python deep learning library. https://keras.io (2015).

  98. 98.

    Kasai, Y., Sakuma, S. & Arai, F. On-chip multi-sorting using high-speed and high-accuracy flow control. In Proc. 22nd International Conference on Miniaturized Systems for Chemistry and Life Sciences (MicroTAS2018) (eds Tseng, F.-G. & Lee, G.-B.) 1237–1238 (Chemical and Biological Microsystems Society, 2018).

  99. 99.

    Paszke, A. et al. Automatic differentiation in PyTorch. In Proc. 31st Conference on Neural Information Processing Systems (NIPS 2017) (eds Guyon, I. et al.)1–4 (Curran Associates, 2017).

  100. 100.

    Tokui, S., Oono, K., Hido, S. & Clayton, J. Chainer: a next-generation open source framework for deep learning. In Proc. Conference on Neural Information Processing Systems (NIPS 2015) (eds Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M. & Garnett, R.) 1–4 (Curran Associates, 2015).

  101. 101.

    Carpenter, A. E. et al. CellProfiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7, R100.1–R100.11 (2006).

  102. 102.

    Abrams, C. S. et al. Direct detection of activated platelets and platelet-derived microparticles in humans. Blood 75, 128–138 (1990).

  103. 103.

    Shalek, A. K. et al. Single-cell RNA-seq reveals dynamic paracrine control of cellular variation. Nature 510, 363–369 (2014).

  104. 104.

    Kalisky, T. & Quake, S. R. Single-cell genomics. Nat. Methods 8, 311–314 (2011).

  105. 105.

    Rotem, A. et al. Single-cell ChIP-seq reveals cell subpopulations defined by chromatin state. Nat. Biotechnol. 33, 1165–1172 (2015).

  106. 106.

    Yamano, T. et al. Light and low-CO2-dependent LCIB-LCIC complex localization in the chloroplast supports the carbon-concentrating mechanism in Chlamydomonas reinhardtii. Plant Cell Physiol. 51, 1453–1468 (2010).

Download references

Acknowledgements

This work was supported primarily by the ImPACT program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and partly by the JSPS Core-to-Core Program and White Rock Foundation. We thank M. Kanematsu, M. Urakawa, A. Komiya, and S. Aihara for assistance. N.N. is an ISAC Marylou Ingram Scholar.

Author information

K.G. conceived iIACS. A.I., N.N., T. Iino, and K.G. designed the protocol. A.I., H.M., K. Hiramatsu, S.S., Y.K., T. Iino, T.Y., A.Y., Y. Oguchi, N.S., Y.S., T. Ito, K. Hiraki, S.M., T.H., F.A., T.S., Y. Ozeki, and N.N. performed the experiments. H.F. and Y.Y. helped prepare the blood and microalgal samples. T.E., M.Y., and T.S. developed the digital image-processing algorithms. K. Hiraki developed the all-IP network. A.I., H.M., K. Hiramatsu, S.S., Y.K., T. Iino, T.Y., A.Y., Y. Ozeki, F.A., T.S., Y. Oguchi, N.N., and K.G. prepared the figures and tables. K.G. supervised the work with the help of T. Ito, Y. Hoshino, Y. Hosokawa, A.N., S.U., T.S., Y. Ozeki, and N.N. A.I., H.M., K. Hiramatsu, S.S., Y.S., M.Y., D.D., T.S., N.N., and K.G. mainly wrote the manuscript. All authors contributed to the writing of the manuscript.

Correspondence to Keisuke Goda.

Ethics declarations

Competing interests

H.M. and K.G. are inventors on a patent covering the FDM microscope. S.S., F.A., and T.H. are inventors on a patent application covering the dual-membrane push–pull cell sorter. N.N., T.S., and K.G. are inventors on a patent covering the data analysis and display method. N.N. is the president of CYBO, Inc. N.N., T.S., and K.G. are shareholders of CYBO, Inc.

Additional information

Peer review information: Nature Protocols thanks Kenneth K. Y. Wong and other anonymous reviewer(s) for their contribution to the peer review of this work.

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

Key references using this protocol

Nitta, N. et al. Cell 175, 266–276.e13 (2018): https://www.cell.com/cell/fulltext/S0092-8674(18)31044-4

Mikami, H. et al. Optica 5, 117–126 (2018): https://doi.org/10.1364/OPTICA.5.000117

Sakuma, S., Kasai, Y., Hayakawa, T. & Arai, F. Lab Chip 17, 2760–2767 (2017): https://pubs.rsc.org/en/content/articlelanding/2017/lc/c7lc00536a

Supplementary information

Supplementary Video 1

Operation of the iIACS machine. The iIACS machine is composed of optical, microfluidic, electrical, computational, and mechanical parts. An interdisciplinary team of trained operators is needed to run the iIACS machine. First, a sample of suspended cells is prepared before a sorting run. Second, a tube containing the sample is placed at the injection port for the sorting run. Third, the process of each subsystem is monitored on multiple computer panels during the sorting run. Fourth, when the sorting run is finished, collection and waste tubes containing sorted and unsorted cells, respectively, are removed from the iIACS machine. Fifth, cells in the tubes are inspected under an optical microscope to evaluate the results of the sorting run. Sixth, the microscope images are automatically analyzed and then manually verified. Finally, the operators discuss the outcomes and reach a conclusion.

Reporting Summary

Supplementary Data 1

AutoCAD design file for the microfluidic chip.

Supplementary Data 2

SolidWorks design file for the optics–microfluidic integration unit.

Supplementary Data 3

Source codes for the IA node.

Supplementary Data 4

Source codes for the TM node.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.