Learning from droplet flows in microfluidic channels using deep neural networks

A non-intrusive method is presented for measuring different fluidic properties in a microfluidic chip by optically monitoring the flow of droplets. A neural network is used to extract the desired information from the images of the droplets. We demonstrate the method in two applications: measurement of the concentration of each component of a water/alcohol mixture, and measurement of the flow rate of the same mixture. A large number of droplet images are recorded and used to train deep neural networks (DNN) to predict the flow rate or the concentration. It is shown that this method can be used to quantify the concentrations of each component with a 0.5% accuracy and the flow rate with a resolution of 0.05 ml/h. The proposed method can in principle be used to measure other properties of the fluid such as surface tension and viscosity.

www.nature.com/scientificreports www.nature.com/scientificreports/ have excellent accuracy (as low as 0.01% concentration of IPA versus 0.5% for our demonstration); however, fine resolution optical refractometers generally rely on resonant mechanisms which are difficult to integrate into microfluidic chips. Optical probes and DNNs can be implemented to measure compounds concentrations for applications such as ammonium concentration measurement in oceanography 25 , hydrocarbon concentration measurement in reacting flows 26 , concentration measurement in microfluidic mixers 27,28 , and measurement of contaminant concentrations in water 29 . From a practical point of view, an interesting feature of the method presented in this paper is that a single mechanism can in principle be used for the measurement of a variety of different quantities, such as surface tension or viscosity, in addition to the ones demonstrated here. The challenge is to design the system and the training process such that the effects due to an individual parameter of the system dominate the pattern of the droplets. Figure 1 depicts the microfluidic device with input flows of silicone oil and water-IPA solutions. Droplets of the water-IPA solution are generated at the cross junction. Images of the droplets are captured using a wide-field microscope. Figure 2 shows examples of such images for an IPA in water concentration of 5.5% for different flow rates. A network with the structure shown in Fig. S2b is trained with 6000 images of the droplets (400 images for each flow rate) at flow rates ranging from 0.1 ml/h to 1.5 ml/h with 0.1 ml/h steps. The silicone oil flow rate is kept constant at 2 ml/h. The output of the network is a single unit whose analog value is trained to be the flow rate of the corresponding image. The trained network is tested with 2900 new images (100 images for each flow rate) recorded at flow rates ranging from 0.1 ml/h to 1.5 ml/h with 0.05 ml/h increments (Fig. 2). Therefore, the ability of the network in determining new flow rates is evaluated in addition to its ability to recognize flow rates from  www.nature.com/scientificreports www.nature.com/scientificreports/ the training set. Figure 3a shows the results of the DNN for all concentrations. The mean error of the prediction is calculated by averaging the absolute relative difference between the predicted value and the true value. The mean errors for the trained and new flow rates are 2.9% and 5.7%, respectively. For non-trained flow rates, there are some cases in which the predicted flow rate is significantly different from the true value. This happens because a flow regime transition takes place as the flow rate is changed. Two examples of such transitions are shown in Fig. 3b. For example, the flow rate at 1.45 ml/h is not predicted correctly by the neural network because the flow pattern switches from individual droplets at a flow rate below 1.45 ml/h into a jet regime above this flow rate. The results of droplet flow rate measurement at a different silicone oil flow rate (1.5 ml/h) are shown in Fig. S4 which confirms the ability of the DNN in the flow rate measurement. We can improve the performance of the neural network by acquiring multiple recordings under different conditions that undergo the flow regime transitions at different flow rates. For example, we could include multiple channels on a single chip with different dimensions. We discuss this idea further in the following discussion on concentration measurement. Figure 4 shows droplet flows at different concentrations of the water-IPA solution where the water-IPA flow rate and the oil flow rate are fixed at 0.2 ml/h and 1.0 ml/h, respectively. The DNN shown in Fig. S2b is trained with 3600 images of each of the four concentrations ranging from 4% to 7% with 1% increments. The trained network is tested with 400 images of each of the seven concentrations ranging from 4% to 7% with 0.5% resolution to evaluate the ability of the network in predicting new concentrations (4.5%, 5.5%, and 6.5%). The diagram of Fig. 5a compares the predicted values with the ground truth. The mean errors of the network for the trained and non-trained concentrations are 1.5% and 9.3%, respectively.

Results
As we can see in Fig. 5a, the network correctly estimates the 4.5% and 6.5% concentrations but it cannot correctly identify the 5.5% concentration. This shows that the change in the pattern from 5.0% to 6.0% is not monotonic. This non-monotonic behavior is the result of the transition between different regimes of the droplet flow pattern due to the change in the concentration of IPA in the solution. This flow regime transition is described in more detail in the Supplementary Material Section (Fig. S3).  Since the flow regime transition is due to a small change in concentration, the DNN cannot predict accurate values close to this point. However, the concentration at which the transition occurs can be relocated by changing the flow rate of the fluid. In this way the outlier points originating from these transitions can be corrected. We tested this idea by recording images of the flow patterns for six different flow rates and using them to train six DNNs independently. We tested the system by feeding six images of the droplet flows from the six flow rate sets to the corresponding DNNs to generate six concentration estimates. Maximum three outliers are removed to reduce the standard deviation of the predicted values. The closest value to the average of the remaining values is taken as the predicted value for the concentration. Figure S5 shows the result of this procedure. The performance of the system can be enhanced further by recording multiple instances of the droplet patterns for each flow rate. We estimate the concentration by averaging the estimates obtained from the individual samples. This procedure reduces the variance of the estimates but it does not change the mean value. Using ten samples as an input, the mean error for concentrations that were included in the training set is reduced to 0.259% and for the concentrations outside the training set to 2.583%. Figure 5b compares the predicted values with true values.
As a final evaluation, we tested the sensitivity of the DNN results with respect to the microfluidic chip. The experiment for the concentration measurement was repeated for a new microfluidic chip but with the same design. Unfortunately, the DNNs that were trained to estimate the IPA concentration using data from the first chip did not give accurate readings for the new chip. When new DNNs were trained with data from the new chip only, the performance obtained was practically the same as for the first chip (Fig. S6 in the Supplementary Materials Section) even though the flow regime transitions occurred at different concentration levels. This indicates that the polydimethylsiloxane (PDMS) fabrication process of the microfluidic channels does not replicate the devices with sufficient precision for the DNNs to be used across different chips. We believe this is mainly due to variations in the PDMS wall properties after O 2 plasma treatment and possibly small dimensional variations. A possible solution is to train the DNNs using data from multiple PDMS devices and make the measurements insensitive to the change of chips.

Discussion
In this paper, we discuss how to measure fluidic properties using microfluidics and DNNs. We show that the classification of fluid properties using traditional image processing methods of the fluid flow patterns does not lead to satisfactory results (section S7 in Supplementary Materials Section). Using DNNs, the accuracy of the classification is drastically increased. Although image processing can be used to measure the droplets size, eccentricity, and velocity [30][31][32] , it is challenging to measure fluid properties such as concentration where the function that relates the effective parameters to the fluid properties is not known. DNNs are shown to be capable of learning the effective factors in two-phase droplet flow for the classification, thus highlighting the potential of deep learning in the microfluidics domain. We use a network with a single analog value output which corresponds to the value of a fluid property. We show that many fluidic properties and parameters can be inferred "on the fly" from this type of DNN, which receives its inputs from a simple imaging system. High accuracy prediction of the composition of the dispersed phase and flow rates are presented as two examples in our work.
The method described in this paper can be used to measure the flow rate even when the flow rate is changing within the device. This change in flow rate happens for instance in micro-reactors where gaseous products are generated or consumed in fluidic channels [33][34][35][36] . In these devices, it is important to measure the flow rate of the gas phase in order to determine their throughput and efficiency. Using DNNs is a suitable approach to measure the gas flow rate online. Moreover, DNNs measure the flow rate in the cases where the exact flow rate inside the microfluidic chip is unknown 37,38 . Furthermore, this method can be used to measure other fluid parameters. For instance, droplets in microfluidic channels are being used as the reaction media 39,40 . This application of droplet microfluidics requires the assessment of the droplet contents [41][42][43] . DNNs can be utilized for this purpose as chemical reactions change the properties of the droplets. Surface tension and viscosity measurements are other www.nature.com/scientificreports www.nature.com/scientificreports/ parameters that can be measured using DNNs. Many studies have investigated the effect of surface tension and viscosity on the droplet flow pattern [44][45][46] . In another approach, the changes in the patterns can be used in neural networks to obtain these influential properties.
One of the limitations of our method is the sensitivity of the result to channel dimensions, pressure, and temperature fluctuations. These parameters should be the same during the measurement and during training to achieve highly accurate results. Intelligent control of fluid parameters using machine learning 47 is a recently developed approach that improves the repeatability of an experiment in a microfluidic device. By combining our approach with the control method of 47 , we can potentially maintain the same conditions of the training experiment and perform the measurement. In addition to this approach, we can train a DNN with inputs from a range of channel dimensions, temperatures, and pressures. A network trained with these inputs will predict more accurately because it has learned a range of parameter values. However, if the range of fluctuations increases, more complex design of the microfluidic device is needed to provide more information from the fluid to the network during the training.
This network, in conjunction with a microfluidic chip, can be potentially used for the measurement of multiple fluid/flow properties at the same time. However, it should be taken into consideration that it is possible to reach the same result by changing two parameters separately. In this case, the information in the pictures is not enough to distinguish these parameters. The voting system used for the concentration measurement is an approach to prevent incorrect measurement under these circumstances. This system considers outputs from multiple networks and determines the correct concentration of the liquid by neglecting outlier output(s). In another approach, the addition of obstacles in the microfluidic channel and changing the geometries of the channels can force droplets to undergo different deformations, which would increase the information present in the images and especially be useful for surface tension and viscosity measurement. Fast and reliable readout of such parameters is important in many microfluidic applications such as chemical synthesis in multiphase flow systems 48,49 , cell culture 50 , and electrochemical microreactors 33,35 .
We remark that although many fluidic properties can be measured using traditional sensors physically connected to the chip, their integration adds to the cost, size, and complexity of a microfluidic-based system. The presence of such cumbersome peripheral hardware is in fact what makes most of today's microfluidic devices best described as "Chip-in-a-lab" instead of a "Lab-on-a-chip" 51,52 . We believe that many functionalities of these physical systems can be integrated into a compact on-chip imaging system 53,54 that benefits from a pretrained learning algorithm to decode the rich information encoded in the images.

Methods
A flow-focusing geometry is used to create droplets of water-IPA (Fisher Scientific UK) solution in silicone oil (viscosity 10 cSt, SIGMA-ALDRICH). The water-IPA mixture enters the main channel from the middle and the oil flows from the sides, as illustrated in Fig. 1. The microfluidic chip containing this geometry is fabricated from PDMS using standard photolithography. The PDMS is plasma bonded to a glass slide to seal the microfluidic channels and subsequently heated at 80 °C for 48 hours to retrieve its hydrophobic properties.
For the concentration measurements, the IPA volume concentration in water is varied in 0.5% steps from 4% to 7%. For the flow rate measurements, the dispersed phase is a solution of 5.5% IPA in water. The flow rate of the water-IPA solutions is varied in 0.05 ml/h steps from 0.1 ml/h to 1.5 ml/h. Considering the precision of the pipettes and the syringes used, the accuracy of the IPA volume concentration in water is 0.08%. During the experiments, the inlet and the outlet temperatures were monitored and the mean temperature was 22.5 °C. The temperature variations were less than 1 °C during data acquisition, a negligible fluctuation which did not have a noticeable effect on the performance of DNNs.
There is one inlet for each liquid and one output for the two phases. The flow rates of the liquids are regulated using syringe pumps (Cronus Sigma 1000 Series). The images are captured by a high-speed camera (Photron FASTCAM Mini UX100) and the image resolution is 1.8 μm/pixel. Images are recorded from the channel in a region downstream of the flow focusing geometry in which the channel dimensions are 100 μm by 1440 μm and 30 μm depth. The smallest droplet in experiments is 12 μm in diameter and is therefore fully resolved in the images. For each experiment, 4000 images are recorded at 50 frames per second. An image denoising algorithm from the OpenCV library 55 and an edge detection algorithm from Python's scikit-image library (the standard deviation of the Gaussian filter is set to 1.2) are applied to the images to detect the droplet boundaries and remove the effect of the background and the light profile in the images. Image processing used in this paper is explained in more detail in the Supplementary Materials Section. This procedure is important to make sure that the DNN does not learn something related to the geometry of the channel or the light intensity. The processed images are used to train the DNNs. The structure of the DNNs is shown in the Supplementary Materials Section (Fig. S2).

Data Availability
The data and the code used for this study are available from the corresponding author on reasonable request.