Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Nonmechanical parfocal and autofocus features based on wave propagation distribution in lensfree holographic microscopy

Abstract

Performing long-term cell observations is a non-trivial task for conventional optical microscopy, since it is usually not compatible with environments of an incubator and its temperature and humidity requirements. Lensless holographic microscopy, being entirely based on semiconductor chips without lenses and without any moving parts, has proven to be a very interesting alternative to conventional microscopy. Here, we report on the integration of a computational parfocal feature, which operates based on wave propagation distribution analysis, to perform a fast autofocusing process. This unique non-mechanical focusing approach was implemented to keep the imaged object staying in-focus during continuous long-term and real-time recordings. A light-emitting diode (LED) combined with pinhole setup was used to realize a point light source, leading to a resolution down to 2.76 μm. Our approach delivers not only in-focus sharp images of dynamic cells, but also three-dimensional (3D) information on their (x, y, z)-positions. System reliability tests were conducted inside a sealed incubator to monitor cultures of three different biological living cells (i.e., MIN6, neuroblastoma (SH-SY5Y), and Prorocentrum minimum). Altogether, this autofocusing framework enables new opportunities for highly integrated microscopic imaging and dynamic tracking of moving objects in harsh environments with large sample areas.

Introduction

Nowadays, continuous monitoring of morphological change and motion in living cell culture is typically performed by direct imaging using a conventional optical microscope installed into a sealed incubator1,2,3,4,5,6. However, the available setups are limited by the requirements of lenses with a complex configuration. This leads to a fixed focal length, a small field-of-view, a short working distance, bulky, expensive instruments, and difficult integration into incubators. Compact conventional microscopes often do not reach the optical quality of large optical systems in terms of focal lengths, magnifications, and optical aberrations, limiting their functionality7,8. In addition, complex optical lens systems often lead to inhomogeneous resolution across the overall image, which is very sharp at the image center, but decreasing towards the edges. This may cause a problem to determine the shape and position of a moving object of interest. Recently, several microscope manufacturers (e.g., Tokai Hit Co., Ltd.) and research groups have attempted to build optical microscopes with an integrated mini incubator that is specifically designed to enable live-cell monitoring and analysis9,10. Despite their good functionality, those incubator-combined microscopes still face several limitations (e.g., less configuration flexibility, small incubator size, and high cost).

Lensless holographic microscopy emerges as a promising alternative to its conventional lens-based optical counterparts in life sciences, reaching a similar resolution while still being mobile, cost-effective, robust, and reliable for biomedical applications. It can be easily customized to be used with standard sample slides, petri dishes or specific microfluidics, making this lensfree imaging system advantageous over conventional table-top optical microscopes11,12, especially in terms of simultaneous parallel investigation and usage in harsh environment settings (i.e., incubators with high temperature and humidity). It creates a compact setup with a short sample-to-sensor distance and an equally sharp reconstructed image at all positions across a large field-of-view13. In contrast to lens-based systems, a holographic interference pattern is captured in the lensfree setup instead of an already focused image of the object. To solve the image reconstruction problem, a digital holographic image reconstruction based on numerical approaches (e.g., angular spectrum14 and Fresnel diffraction methods15,16) can be implemented to back-propagate the recorded interference pattern into the correct z-position. The raw diffraction image of the illuminated sample is transformed by computational reconstruction into a wave representation of the sample.

Although different designs and approaches of lensless digital holographic microscopes have been reported to be used in a wide range of biomedical and environmental monitoring applications14,17, they have not employed statistical approaches (e.g., skewness, chi-square, power-divergence, Jarque–Bera, kurtosis, normality test, and Gini index) for analyzing the wave propagation distributions. Moreover, the benefits of the parfocal function in lensless microscopy were also not described. From the other report, a lensfree imaging system was integrated with microfluidics for point-of-care testing12. The captured images were however not reconstructed with either manual or automatic process. Therefore, besides the unsatisfactory interference pattern, the resulting image has poor quality. In that setup, a charge-coupled device (CCD) sensor was utilized instead of a complementary metal oxide semiconductor (CMOS) image sensor. It is known that even though CCD technology has been dominant in the market for visible photon detection and image acquisition over the last two decades, it has several drawbacks (e.g., slow serial access to image for large-size arrays, high power dissipation because of intrinsic capacitive nature of gates, and high power consumption)18. Thus, CMOS detectors are preferable to be used in microscopy application due to their intrinsic advantages (i.e., low power consumption, high readout rate, low noise, and high integration capability down to the pixel level)18,19.

Previous studies mostly demonstrated that cell detection in holographic microscopy could only be performed on clear shadow images, where the small size and high coherence degree of the light source possessed a key role to reduce the geometric unsharpness20,21,22,23. Here, lasers have been commonly used in holographic system because of their high temporal and spatial coherence24,25. However, a too high coherence can also result in enormous speckle noise lowering the produced image quality26. Thus, a light-emitting diode (LED), has been selected to tackle the speckle problem and similar interference issues26,27. The emission of an LED is incoherent in phase, but coherent in space, depending on its geometrical dimension. Therefore, attaching a pinhole structure on the LED is suitable method to yield an ideal light source for lensless holographic imaging.

In respect to the base material, gallium nitride (GaN)-based LED has been widely accepted and employed as a promising solid-state light source because of its high efficiency, robust structure, high brightness, fast modulation, and long lifetime. These are achieved not only for large-area high power LEDs used in solid state lighting, but also for LEDs with dimensions of below 10 µm (i.e., microLEDs) as structured micro-illumination sources28,29,30,31,32. To support the development of the compact microscope, micro- and nanoscale GaN-based LEDs can be fabricated using either bottom-up or top-down approach, as well as processed to be a highly integrated pinhole microLED array, which is beneficial for increasing spatial coherence and hence the imaging capability22,33,34,35,36.

Despite its simple mechanical configuration, the extraction of a real image from the diffraction pattern faces several challenges. This contains the shadowing of structures, the irregular background intensity, and the appearance of out-of-focus interference patterns that hinder the distinguishability between sample structures and background noise37. Moreover, the image reconstruction process requires the z-distance between the sensor and the sample as an input parameter, which is still estimated either manually or by complex and time-consuming algorithms38,39,40,41. To address these issues, the autofocus feature42,43,44,45 and parfocal technique described in this work can be employed simultaneously. The former is dedicated to automatically and instantly find the sharpest image possible at all times, while the latter is assigned to keep the object of interest staying in-focus when the focal length changes due to object movement in z-direction or alteration in magnification level46,47,48. The parfocal system is an important building block in many imaging instruments and can be commonly found in bright-field microscopes. Nonetheless, for the conventional parfocal technique, a complicated setup combining special objectives and several magnifying lenses are required to obtain the in-focus effect. This leads to several drawbacks, i.e., higher cost for setup realization, difficult incorporation into specific applications, and time-consuming process for long-term and continuous experiments49. To perform an automatic focus tracking, Ferraro et al. have developed a digital holographic microscope consisting of a 532 nm laser light source and an objective with specific focal length and numerical aperture23. Despite its success in the phase-shifting measurement of holographic interference fringes, the microscope has not been tested for tracking biological cells. Moreover, because of its complex and bulky setup, the system was not suitable for being used inside a conventional incubator to perform a real-time cell monitoring.

Lately, several statistical and computational methods have been proposed to evaluate the focus quality in lensless holographic microscopy reconstructions. Among other techniques (e.g., non-linear correlation, power spectrum, variance, summed gradient, Laplacian, and high-frequency discrete cosine transform50,51,52,53), a method based on the Gini index has become one of the most popular methods to deliver the best in-focus position54,55,56. Nevertheless, this algorithm requires pre-processing steps in form of background subtraction and edge detection, increasing the required processing time41. Besides, with the rise of interest to create smarter imaging systems, self-learning algorithms like deep learning and machine learning were also investigated lately regarding their autofocus usage41,57. However, again, a complex computational architecture, which requires a lengthy and arduous training process on image recognition, should firstly be performed before the autofocus can be used on the object of interest, resulting in a longer processing duration.

Hann-filter algorithm was also reported to be used to find the best focus values of the images through an interval search approach25. It could perform three-dimensional (3D) tracking of moving cells. However, again, the presented holographic imaging system still employed lens and laser light source. In another lens-based microscope, a single-shot mechanical autofocusing method was demonstrated for fluorescence imaging, in which a motorized stage was employed to move the sample position and keep the image always in focus45. The critical sampling of the complex field was used to improve the processing speed by adjusting the holographic image sampling size. The main drawbacks of this approach are its slow response and low accuracy during the autofocusing operation, which originated from the limited speed and high slip in the used motor.

In this work, we developed a novel parfocal autofocusing method for real-time imaging in a pinhole LED-based lensless holographic microscope. Instead of using complex and expensive laser light source that has several issues (e.g., high speckle noise and adverse effect on both biological cells and CMOS sensor), low-power LED was utilized as a light source to illuminate our sample, in which a pinhole plate was added to improve its spatial coherence and to produce optimum interference pattern of the holographic image. The function of a parfocal lens was substituted with a computational analysis of the wave propagation distribution from the sampled dataset, which was performed by calculating the similarity of the intensity distribution to the Gaussian profile and analyzing the asymmetrical statistical distribution, resulting in a continuous autofocus condition. Due to the computational holographic reconstruction, any mechanical adjustment of microscope parts could be completely omitted, i.e., to adjust the lens or sample-to-objective-lens distance. We explored the usage of various light sources and statistical approaches (i.e., skewness, chi-square, power-divergence, Jarque–Bera, kurtosis, normality test, and Gini index) to find the optimum image quality and obtain a faster processing speed of the information from the raw hologram images. Besides, to investigate its reliability and performance, the system was directly tested inside a sealed incubator for a live monitoring of different cell cultures. Our parforcal autofocusing holographic microscope was not only capable of performing long-term observation (for more than seven days), but also providing a 3D position information of the moving cells (in x-, y-, and z-directions) while maintaining in-focus images automatically.

Results and discussion

Light source optimization based on spatial coherence

The main goals of this study are to obtain the sharpest holographic image and to continuously keep it in-focus while the object position changes, especially in z-direction. Instead of employing a laser as the light source, an LED was integrated into the lensfree microscope (Fig. 1a,b) mainly because of its partial coherence characteristic, stable intensity, and harmlessness to living samples or CMOS image sensors28,58. However, the spatial coherence of the LED-illumination has to be adjusted to reduce the geometric unsharpness. The coherence of the light source can generally be divided into temporal and spatial coherences. The former is related to phase coherent emission in combination with the spectral bandwidth of the light source, while the latter describes the phase uniformity of a wavefront in space by analyzing its space correlation and ability to produce stable interference patterns59. As the spatial coherence can be also influenced by LED size and propagation distance, integrating a microscale pinhole (e.g., with a hole diameter of < 300 µm) onto an LED as a spatial filtering component can increase the spatial coherence and consequently the imaging capability26.

Figure 1
figure 1

Portable LED-based lensless microscope and its limit of resolution in pinhole and non-pinhole configurations. (a) Schematic of digital in-line holographic microscope consisting of a light-emitting diode (LED), a pinhole, a specimen, and a complementary metal oxide semiconductor (CMOS) image sensor. 3D visualization was obtained using Paint 3D (Microsoft Corporation). (b) Point light source produced by a blue LED emitting at a wavelength of 467 nm and a covering pinhole of 100 μm used in a lensless microscope coupled with an x–y–z micrometer stage. (c) Raw and reconstructed intensity images using white and blue LEDs with and without pinhole setup. The resolved resolution limit is measured by the smallest recognizable distance in the USAF test pattern. Python 2.7.14 (https://www.python.org) and Opencv 3.4.1 (https://opencv.org) were used to generate the images. (d) Line scan over the first element of group 6 in the USAF chart produced by different LEDs with and without pinhole setup. The spatial and temporal coherences affected the achieved system resolution, where the resolved resolution limit is shown from the element 4 of group 7 after image reconstruction. OriginPro 2019b was employed for plotting the figures (https://www.originlab.com).

We tested both polychromatic and nearly monochromatic LEDs (i.e., white and blue LEDs) having conditions of with and without pinhole to find the optimum light source for the later experimental setup by comparing the clarity and sharpness of the reconstructed images obtained from the United States Air Force (USAF) 1951 resolution test target (Fig. 1c,d). On this test chart, nine patterned groups with different sizes are available, in which each group comprises six elements of vertical and horizontal black bars with precisely defined width and spacing. This target was used to determine the best in-focus position and the resolution of the imaging system, which was indicated by the clearly distinguished space between the two smallest possible bars. The raw images captured by the CMOS sensor were reconstructed for various z-positions using an angular spectrum algorithm (see “Methods” section), while the wave propagation distribution analysis was used to find the best focal position and the sharpest image within several z-planes.

A high spatial coherence of the object illumination is required to produce a distinct diffraction pattern with a high resolution. Otherwise, only a blurry random pattern will be formed. Using a white LED, the interference pattern formed on the CMOS active area is produced by a range of wavelengths59. Therefore, the illumination led to a mixture of several fringes from different wavelengths, which can increase the pattern unsharpness (Fig. 1c,d). In lensless imaging, samples with a high object density and narrow object spacing can result in a low quality of the reconstructed image due to the phenomenon of diffraction limit60. As the light wave enters the aperture between the edges of objects with very small distance, the diffracted waves will interfere. This interference like the one appearing from the first element of group 6 in the USAF chart can be either constructive or destructive (Fig. 1c,d).

The reduced spatial coherence length of the light sources without pinhole leads to unclear interference patterns (Fig. 1c columns 1 and 2), indicating that the center of the Gaussian spot has been increased leading to a lower system resolution and a reduced noise pattern. The experimental results demonstrated that pinhole-covered LEDs possessed a higher spatial coherence compared to the bare LEDs. The pinhole could significantly decrease the aberration of the diffraction pattern from the USAF target (Fig. 1d). However, it also simultaneously reduced the light intensity that was used to expose the object. To mitigate this limitation, a shorter light-source-to-sample distance and a post-image processing could be opted and performed, respectively, resulting in better image intensity. Again, it is clear from the comparison of the interference patterns produced by non-coherent (without pinhole) and partially coherent (with pinhole) light sources that the clearest images with the highest resolution were obtained when the pinhole-covered blue LED was used in the setup. This could be proven by comparing the horizontal and vertical profiles in the fourth element of group 7 in the USAF (indicated in blue and red lines, respectively, in Fig. 1c,d). With this light source configuration, the lensless microscope was able to observe the sample with a size of down to 2.76 μm, which is considered as our microscope resolution (Fig. S1 in Supplementary Information). Moreover, the fifth element of group 7 in the USAF with a size of 2.46 μm was still recognizable, but not very clear (Figs. 1c and S1). These results indicate that using a blue LED with a pinhole can reach the image resolution that is very close to the limit of the CMOS pixel size (2.2 μm), which however, is only of relevance when the object is positioned directly on top of the CMOS sensor. Thus, from this point onward, we used a blue LED with pinhole setup as the light source for further measurements with the lensfree microscope.

A 100 µm pinhole was used to limit the light emission from the 150 µm × 200 µm LED source. The optical power densities of the blue LED before and after being integrated with the pinhole are 298 and 0.257 µW/cm2, respectively. To analyse the temporal coherence of our setup, we compared the measured spectra of the blue LED with and without pinhole (see Fig. S6a in Supplementary Information). It was found that both spectra possess similar shape with a peak wavelength of 467 nm, which indicates the unchanged temporal coherence. However, in case of the pinhole LED, a lower illumination intensity was yielded.

Autofocus and parfocal features

After optimizing the light source used in the setup, we then placed a USAF test target directly above the sensor in the microscope to start developing the autofocus and parfocal features. As a result, the interference patterns could be seen clearly, making the USAF pattern unrecognizable in the interference pattern (Fig. 2a; z = 0 μm). This phenomenon is due to the fact that the interference pattern was recorded at z = 0, which is the plane of the CMOS sensor device. Nonetheless, the actual object position was at an unknown z-position, given by the gap between the active layer of the photodiode inside the CMOS sensor and the outer edge of the sample surface, as well as the glass layer thickness of the USAF sample. Thus, the wave propagation distribution analysis was employed to find the right z-plane for the reconstruction of the images. The highest resolution was perfectly solved by combining the angular spectrum method and wave propagation distribution analysis to back-propagate and find the correct z-position of the sample at the position, where the “clearest” reconstructed image occurred (Fig. 2a; z = 590 μm). When the z-plane of the reconstruction has been further shifted up to z = 2000 μm, the yielded USAF pattern image blurriness increased again, due to the large discrepancy between the supposed and actual positions. Identifying the sharpest image does not only represent the highest quality of the image, but also provides an estimate of the distance between sensor and object. The z-value is a parameter in the angular spectrum algorithm for calculating the back propagation of the image to the object plane. While moving mathematically in the z-direction, the wave propagation distribution analysis calculates the distribution of intensity to find the highest non-normal distribution (as compared to the intensity distribution of input images) automatically without requiring any pre-processing step. The first-order histogram parameter assessment was conducted in the distribution analysis. Our automated algorithm demonstrated a significant increase in sensitivity, specificity, and accuracy in perceiving the best in-focus image.

Figure 2
figure 2

Wave propagation distribution analysis. (a) Enlarged view of groups 6 and 7 from the calculated distances of z = 0 μm (input image), 590 μm, and 2000 μm, respectively. The first-order histograms (blue) including their parameters (mean, median, and mode) compared to normal distribution curves (green) show that the best in-focus position of the USAF target sample is found at z = 590 μm, where the largest difference between the histogram and normal distribution occurs. (b) Different distribution analysis criteria of USAF sample along various z-planes. The yellowish dashed circle exhibits the best in-focus position at z = 1.63 mm above the sensor. (c) Micro dot image at z = 0 and reconstructed dot image at z = 462 µm. The point spread function (PSF) shows the scanning over z-position and the limited resolution in the axial direction. USAF and PSF figures were created using Python 2.7.14 (https://www.python.org) and Opencv 3.4.1 (https://opencv.org), while intensity and normalized distribution analysis figures were plotted using OriginPro 2019b (https://www.originlab.com).

The first-order histogram parameters exhibit various intensity distributions from the raw (z = 0) and reconstructed images (Fig. 2a). The blue bars and green lines indicate the intensity histogram and the normal distribution model obtained from its intensity, respectively. An obvious dissimilarity between the normal distribution and histogram can be seen in Fig. 2a, while the skewed histogram difference also appears in different z-planes. To prove this, a single 15 μm dot image was reconstructed by conducting a rough z-scan ranging from 0 to 2 mm with a step size of 10 μm (Fig. 2c). As a result, the best focal position of z = 462 µm was found from a narrow area in the Gaussian curve, obtaining the highest resolution in the z-propagation.

The basic concept for the autofocus feature is to analyze the density and distribution of pixel intensity from the histograms, where the sharpness and blurriness in the images can be differentiated (Fig. S2 in Supplementary Information). In general, the wave propagation distribution was measured using the intensity distribution as input data. Here, the result was zero if all the data were identical with the Gaussian curve (Fig. 2a). Meanwhile, it would increase in number as the data became more diverse.

A key challenge is the identification of the “best” image, giving at the same time the “correct” z-position of the object plane. Here, we extracted quality parameters from the back-propagated image for a given (but changing) z-position based on various statistical approaches for analyzing the wave propagation distributions (i.e., skewness, chi-square, power-divergence, Jarque–Bera, kurtosis, normality test, and Gini index). All these statistical approaches were performed on the USAF pattern images to identify the best and most stable procedure.

Skewness is an asymmetrical statistical distribution, where the resulting curve is distorted and tends to be skewed to the left or right. The skewness value (Sk) can be quantified by representing the difference between a given distribution and a normal distribution, as follows50,61,62:

$${S}_{k}=\frac{\frac{1}{n}\sum {({x}_{i}- \bar{x} )}^{3}}{{\sqrt{\frac{1}{n-1}\sum {({x}_{i}- \bar{x} )}^{2}}}^{3}}$$
(1)

where n, \({x}_{i}\), and \(\bar{x}\) are the number of data points, independent data variables, and mean, respectively.

Kurtosis is a statistical measure that is usually used to assess the peak levels or flatness in a distribution. A high kurtosis value describes a distribution with special peaks and large tails, while a low kurtosis value represents a flatter distribution. The kurtosis value (Ku) can be calculated by50,63:

$${K}_{u}=\frac{\frac{1}{n}\sum {({x}_{i}- \bar{x} )}^{4}}{{\left(\frac{1}{n}\sum {({x}_{i}- \bar{x} )}^{2}\right)}^{2}}-3$$
(2)

where 3 is the value for standard normal distribution.

In some cases, chi-square (χ2) can be used as a test for normality. To implement it, a normal distribution of the observed data becomes the null hypothesis for the expected value. χ2 can be described as64:

$${\chi }^{2}=\sum_{i=1}^{k}\frac{{\left({O}_{i}-{E}_{i}\right)}^{2}}{{E}_{i}}$$
(3)

where \({O}_{i}\) and \({E}_{i}\) are the observed and expected values, respectively. In the power-divergence (PD), the fit of the model can be assessed by comparing the expected frequencies for each outcome with the observed frequencies from our sample using65:

$${P}_{D}=\frac{2}{\lambda (\lambda +1)}\sum_{i=1}^{k}{O}_{i}\left[{\left(\frac{{O}_{i}}{{E}_{i}}\right)}^{\lambda }-1\right]$$
(4)

where \(\lambda\) is a real-valued parameter that is chosen by the user. In this experiment, the employed power-divergence statistic is Pearson \({\chi }^{2}\) statistic with \(\lambda \hspace{0.17em}=\hspace{0.17em}\)1.

In statistics, the Jarque–Bera (JB) test is a normal distribution test of sampled data. This method investigates whether the sample data have any skewness and kurtosis compared to a normal distribution. Here, the statistical test of JB is defined as66,67,68:

$${J}_{B}=\frac{n}{6}\left({S}_{k}^{2}+\frac{1}{4}{\left({K}_{u}-3\right)}^{2}\right).$$
(5)

For the normality test (K2), the quality of the fit is measured by a Gaussian function in a normal distribution based on D'Agostino-Pearson approach using69:

$${K}^{2}={Z}^{2}\left(\sqrt{{b}_{1}}\right)+{Z}^{2}\left({b}_{2}\right)$$
(6)

where \(Z\left(\sqrt{{b}_{1}}\right)\) and \(Z\left({b}_{2}\right)\) are the normal approximations to the sample estimates of skewness and kurtosis, respectively.

For the Gini index (GI), a statistical dispersion is measured in most cases. This method was also reported being able to perform an autofocus image scanning in holographic imaging52,56,70,71. The GI can be calculated as54,55,56:

$${G}_{I}=1-2\sum_{k=1}^{N}\frac{{a}_{\left[k\right]}}{Sum\left(C\right)}\left(\frac{N-k+0.5}{N}\right)$$
(7)

where \({a}_{\left[k\right]}\) is the sorted data observation and c the data vector. In case GI = 0, it indicates that the data have an equal amount of energy among them. Whereas, for GI = 1, the input data have the highest inequality. In other words, the sparsest data are concentrated only in one element of energy.

All the statistical metrics that are briefly described above can be used to identify the “sharpest” image. The calculated results in Fig. 2b show that the wave propagation distribution analysis could produce the sharpest image in the z-propagated estimation from the angular spectrum algorithm by finding either global maxima (i.e., in the chi-square, power divergence, Jarque–Bera, kurtosis, normality test and Gini index) or global minima (i.e., in the skewness) of the metric values. Despite the gap between the USAF sample and the CMOS sensor, both global maxima and minima can be found as indicated by the yellowish dashed circle in Fig. 2b, in which the best in-focus position of the sample was found at z = 1.63 mm above the sensor. This outcome has therefore demonstrated that the optimum in-focus position of an object in the z-plane could be determined without any additional processing step prior to the angular spectrum or distribution analysis process.

To test the reliability of our autofocus approach as well as to manifest the computational parfocal feature in our lensfree microscope, the inspected USAF target was then moved continuously in z-direction using an x–y–z micrometer stage (Fig. 1b), in which several frames of the raw data and their reconstructed results from the recorded videos are displayed in Fig. 3a, respectively. For the raw images, the diffraction and interference patterns grew with increasing z-position as a result of the incrementally increasing distance between USAF sample and CMOS sensor. However, once the angular spectrum method and wave propagation distribution analysis have been applied to each frame from those sequentially captured images, the reconstructed USAF images were able to be kept in-focus, even when the sample was moving vertically in z-direction. This effect is similar to the feature of a parfocal lens in conventional microscopes. A time-lapse video demonstrating our nonmechanical parfocal function in the USAF target experiment is presented in Supplementary Materials (Video S1).

Figure 3
figure 3

Demonstration of parfocal feature in a portable holographic microscope using a USAF target, which was moved along the z-axis. (a) Selected frames from a real-time video of continuous USAF sample movement in z-direction, showing a growth in diffraction and interference patterns along with the increasing sample-to-sensor distance. The parfocal feature for lensless microscopes was revealed from the reconstructed images, which have been processed using angular spectrum method and wave propagation distribution analysis. The computationally reconstructed USAF images could stay in-focus during the change of sample position in z-direction. The figure was generated using Python 2.7.14 (https://www.python.org) and Opencv 3.4.1 (https://opencv.org). (b) Comparison result between our statistical computation based on different distribution analysis methods and the manual step of the micrometer stage (green line). (c) The processing time needed to complete the computation for image reconstruction using various statistical distribution methods in respect to the incremental size of the region of interest on the observed image. (b) and (c) were created employing OriginPro 2019b (https://www.originlab.com).

Our study has revealed that the histogram parameters were capable of distinguishing the difference between the clear and distorted images. By analyzing the density and distribution of pixel intensities, it was found that the crisp and sharp images mostly possessed non-symmetric distributions or were in “skewed” conditions. Since the basic principle of lensless in-line holographic microscopy is to capture the shadow image from the sample, the sharpness and crispness of the object can be analyzed from the shadow edge. In other words, how the intensity is distributed at the shadow edge may discriminate the object of interest from its background. From the z-micrometer stage movement experiment, skewness, Jarque–Bera, kurtosis, and normality test appeared to be promising to differentiate the sharpness and blurriness of the image in the holographic microscopy (Fig. 3b). Like the feature in a parfocal lens, these methods enabled the finding and maintaining of the sharpest image when the object changed its position without losing the focal point, resulting in the same gradient of z-position changes as the mechanical movement of the z-stage. Skewness was able to find the sharpest image independent on its origin (i.e., either from bright-field or dark-field microscopy). In that approach, the crispest image was found by searching for the skewed values that tended to be negative for bright field microscopy or positive for dark field microscopy. Meanwhile, the other methods (i.e., Jarque–Bera, kurtosis, and normality test) tended to find the largest distance from the normal distribution. On the other hand, weaknesses were found in the chi-square, power-divergence and Gini methods because they produced more than one local maximum at different z-positions, which resulted in ambiguities in determining the focus position (i.e., several peaks were produced along the focus position).

In general, a simple but powerful algorithm offers an enormous advantage in statistical computing. The use of distribution analysis in this system does not burden the central processing unit (CPU) usage. For all the conducted experiments in this report, a standard laptop with i5 processor of 2.2 GHz and random-access memory (RAM) of 8 GB was employed without graphics processing unit (GPU) acceleration, where the processing time required to calculate and compare the best in-focus value was moderately short (e.g., < 0.2 s for an image with size of 1 MP as indicated in Fig. 3c). Here, regardless of their accuracies, the chi-square and Jarque–Bera statistics have been observed as the quickest and lengthiest processing methods, respectively. In the real-time image processing, the frame rate depends on the size of the region of interest. The smaller area of interest, the shorter the computation time. Thus, the process requirements can be flexibly adjusted. For instance, in case of tracking a specific moving cell, the region of interest can be concentrated around that cell only, resulting in an even faster image processing speed. Meanwhile, if the monitoring should cover a wider field-of-view, the region of interest can be enlarged, hence the condition of that cell dynamic can be compared with those of other surrounding cells. During their cultures, the cells often grow as agglomerates at several different spots. Nonetheless, this rapid processing is achieved due to the fact that the image is directly calculated without requiring any pre-treatment or training process like in machine or deep learning. The image reconstruction can be performed in real-time, allowing the sample in the video to move along the z-axis, while simultaneously maintaining the sharpest image through a combined calculation of angular spectrum algorithm and wave propagation distribution analysis. However, the spatial resolution as well as the geometric magnification might change for different z-positions (see Fig. S5 in Supplementary Information). It should also be mentioned again that this analysis returns back the z-position of the tracked object.

Although sharp images could be obtained using the proposed system, few artefacts (i.e., twin images) are still apparent on the output images. These artefacts found in the images reconstructed with the angular spectrum method are attributed to spatial frequency aliasing, which is well known in the fast Fourier transform (FFT) algorithm72,73. Various methods (e.g., twin image reduction38 or deep neural network to perform cross-modality image transformation74) can be applied to holographic images for reducing speckle noises and image artefacts. Hence, images with a better signal-to-noise ratio can be produced. However, applying this technique will increase the required processing time since an additional iteration of post-processing step is needed causing a significant time-lag in the real-time monitoring process.

3D motion tracking of living cells

Having a reliable automatic program based on the angular spectrum method and wave propagation distribution analysis for parfocal features, we then applied our lensless microscope to real-time and long-term observations of various cell cultures (i.e., MIN6, neuroblastoma (SH-SY5Y), and microalgae Prorocentrum minimum (P. minimum) cells) inside a sealed incubator chamber (Fig. S3 in Supplementary Information), in which the cell motility became the main interest of study (Fig. 4a–c). The experiments were performed from several days up to more than a week at a stable temperature resulting in high-quality images of the cells with the automated best in-focus position, even when they were actively moving. All the recorded videos containing the raw data and the reconstructed results of SH-SY5Y and P. minimum cells are presented in the Supplementary Materials (Videos S2 and S3, respectively).

Figure 4
figure 4

Real-time, label-free monitoring of biological cells using a parfocal autofocus-featured lensless microscope inside a sealed incubator. The cultured cells were placed in petri dishes. (a) Raw and reconstructed images of three different living cells (i.e., MIN6 cell line, P. minimum, and neuroblastoma (SH-SY5Y)). Motion tracking of (b) neuroblastoma and (c) P. minimum cells in x- and y-directions on a stable in-focus reconstruction mode. The green arrow indicates the moving direction of the inspected cells. The images were generated using Python 2.7.14 (https://www.python.org) and Opencv 3.4.1 (https://opencv.org).

From the obtained results, it was demonstrated that our lensless holographic microscope was able to observe and clearly differentiate the cultured cells of various shapes in a petri dish (Fig. 4a, first row). Initially, all cells appeared to have a similar circular or roundish form in the raw images, where the roundish occurrence was caused by diffraction. However, after applying the parfocal autofocus features to the reconstructed images, different irregular shapes of the MIN6 and neuroblastoma cells as well as the triangular oval-round form of the P. minimum cells could be clearly distinguished (Fig. 4a, second row). Fig. S4 in Supplementary Information depicts proliferation of MIN6 cells and formation of cell clusters during the long-term culture inside the incubator. These preformed clusters are the first step in the formation of spherically shaped pseudo-islets. Being able to monitor those cells in real time, we could then plan other experiments to gain more insights on cell growth mechanism. However, a deep bio-analysis was not considered as a focus in this study. This paper has been concentrated on the technical proof-of-concept for the development and integration of parfocal autofocus feature into portable LED-based lensless microscope equipped with CMOS image sensor.

It should be noted that these computational approaches can be activated directly from the beginning of the experiments to automate and provide a more accurate image reconstruction process in real-time, especially when dealing with a large amount of images or a long video duration. Furthermore, in comparison to the conventional optical microscopes that have a small field-of-view and depend highly on the field number and total magnification, our developed method allows the lensfree imaging system to produce very sharp images with a larger field-of-view (i.e., up to 5.7 mm × 4.3 mm depending on the employed image sensor). From the cell motion tracking experiments, this system was capable of delivering not only two-dimensional (2D) information of cell movement in x- and y-axes, but also the cell position in z-direction yielding a complete three-dimensional (3D) information dataset (Figs. 4b,c and 5a–d).

Figure 5
figure 5

3D motion tracking of living neublastoma and P. minimum cells using a parforcal autofocus-featured lensless microscope. (a) In-focus z-position and (b) 3D trajectory of an actively moving neuroblastoma cell performing a diving motion. (c) In-focus z-position and (d) 3D trajectory of a tracked P. minimum cell having a drifting motion. Different wave propagation distribution analysis methods (i.e., skewness, Jarque–Bera, kurtosis, and normality test) were evaluated to find the best in-focus positions for both cells. All the figures were plotted using OriginPro 2019b (https://www.originlab.com).

In Fig. 4b, the tracked neuroblastoma cell seemed to only swim forward along the x-axis, without actively changing the movement direction. To obtain 3D information of the cell trajectory, we implemented our method of cell tracking in the z-direction. According to the results from z-plane movement during USAF target experiments with a moving micrometer stage (Fig. 3b), we decided to implement only four different statistical methods (i.e., the skewness, Jarque–Bera, kurtosis, and normality test) for enabling the acquisition of the z-position of the cell. By combining the 2D movement tracking in x–y-direction and the z-value obtained from those calculation methods, our system could gain 3D trajectories of the neuroblastoma cell movements while keeping up with the ever-changing focus position (Fig. 5a). It was finally discovered that the cell was actually moving not only in x-axis but also in the z-direction within a range of up to 500 µm (Fig. 5b), resulting in observation of a diving motion. Nonetheless, this value is not the limit for our system. According to our previous USAF target experiments with a moving stage, an even larger z-trajectory range of up to 5 mm can be easily resolved.

In case of P. minimum cell tracking, although the cell was only moving within a range of 100 µm in z-direction (Fig. 5c), it was actively making abrupt changes in its motion direction along the x–y-axes, demonstrating a drifting motion at the beginning of the tracking (Figs. 4c and 5d). The cell activity was able to be further analyzed by monitoring its reconstructed 3D trajectory. Such a complete 3D position information with a sharp image is important for biologists and pharmacologists to investigate the reactions of the active target cells towards the introduced drugs during the development of novel pharmaceutical therapies for various specific diseases75,76,77. It was also demonstrated that the light emitted from a pinhole LED was harmless to the cells within a long-period observation78. The developed parfocal autofocus algorithm has rendered the lensless microscope very suitable as a cell culture monitoring device, especially in a harsh environment (e.g., inside the incubator with a specific temperature and humidity). A computational parfocal method combining the angular spectrum algorithm with the wave propagation distribution analysis has been proven to improve the qualities of the captured images from the USAF resolution test target and biological cells by comparing the image sharpness values of the objects of interest in different z-positions. Here, skewness and kurtosis methods appeared to be the most promising measures to perform discrimination between sharp and blurred images for finding the best in-focus plane in real-time holographic microscopy without any image pre-processing, which could then shorten the processing time significantly (Fig. 5a,c). Meanwhile, Jarque–Bera and normality test approaches provided good reconstruction results, since both are based on skewness and kurtosis methods.

In conclusion, the wave propagation distribution has been measured to analyze the intensity at all spatial frequencies from the distances of different objects to the CMOS sensor in the parfocal lensless holographic microscope having a pinhole LED-based light source. Here, a USAF resolution target and different biological cells have been investigated as the objects. The distribution data obtained by various statistical measures (i.e., skewness, Jarque–Bera, kurtosis, and normality) were then combined with the angular spectrum method to enable an automated finding of the focal point. Best in-focus positions of objects can be analyzed even during their movement in all directions (x-, y-, and z-axes). This method enables the compact microscope to perform a parfocal autofocus feature by real-time processing with neither using any lenses nor involving optical adjustment and a complex data training and pre-processing step. The in-focus position also provides the z-distance between the sensor and the object, thus also obtaining the 3D position and movement of the biological cells (i.e., MIN6, neuroblastoma, and P. minimum cells) inside the incubator, allowing long-term observations (more than a week). In the future, a deeper analysis of the influence of refractive properties of different media in the wave front propagation will have to be carried out to acquire a more accurate object-to-sensor distance, resulting in 3D live-cell motion tracking with even higher accuracy and error-reduced information.

Methods

Portable lensless microscope

The portable lensless holographic microscope consists mainly of an LED, a pinhole, a CMOS sensor, a 3D-printed housing, and electronics (Fig. 1a). Light produced from a GaN-based LED passes through a pinhole providing a partially coherent light source. It then impinges on the sample or object of interest, producing an interference pattern of the sample. The pattern is captured as a holographic image by a CMOS image sensor consisting of an arranged array of light-sensitive photodiodes (photocells). This component can detect brightness values and generate their corresponding bright or dark pixels. The denser is the photocells in an image sensor, the higher the spatial resolution of the produced image will be. This microscope is equipped with a monochrome CMOS sensor to detect light intensity without performing a color separation. The sample needs to be placed very close to the sensor and illuminated by light with an optimum coherence to obtain a highest possible spatial pixel resolution (Fig. 1b). Several extra features are also added in our microscope to improve the sample observation accessibility (i.e., an embedded computer, a Python-programmed data socket communication interface, and a cloud storage library). The embedded machine operates the light source, controls the camera of the CMOS sensor, and transmits the images using a wireless local area network connection to another computer or a smartphone for viewing and processing purposes. The Python data socket communication interface serves as a communication platform between the back-end system and the computer or smartphone. To perform a long-term monitoring, a cloud storage library is also inserted in our Python code. This library will send the images to the cloud data storage automatically, enabling observation, detection, and quantification of cells at anytime and anywhere.

Apart from the coherence optimization measurements that used different setups of LED and pinhole, all other experiments conducted in this work (for USAF and cell assessments) employed a blue LED with a wavelength of 467 nm as a light source. A 100 μm pinhole made of black oxide-finished stainless steel was adopted and placed over the image sensor to increase the spatial coherence of the LED (Fig. 1a,b). A CMOS camera with a field-of-view of 5.7 mm × 4.3 mm, a pixel array of 2592 × 1944, and a pixel size of 2.2 μm × 2.2 μm was integrated into the lensless microscope to capture the holographic images. A USAF 1951 resolution test target and various cells were used as samples to be investigated.

To find a compromise between spatial coherence and light intensity, the LED light source has been set to have a distance of 60 mm to the CMOS image sensor. This distance is estimated by calculating the first minimum from the complex degree of coherence produced by the pinhole LED, which corresponds to the resolution limitation through imperfect spatial coherence and inverse square law of the light intensity reduction by distance79. Placing the light source near to the sensor will reduce spatial coherence and therefore the resolution. Meanwhile, increasing the LED-to-detector distance will reduce the light intensity resulting in lower signal-to-noise ratio.

The employed blue LED with pinhole has an optical power density of 0.257 µW/cm2, which is much smaller than that without pinhole (i.e., 298 µW/cm2). Besides optical power, the exposure time was also investigated. For a real-time observation, a very short exposure time of the individual image frame has to be used within the setup. Therefore, it was set to be 0.1 s for each frame during continuous image acquisition. The gain value of the CMOS sensor adjusts the analog signal amplification. To obtain the optimum values of brightness and image capturing speed, the combination of exposure time and gain has been varied (see Figs. S6b and c in Supplementary Information). In our experiment, while we set the gain level at a value of 24, our lensless microscope was able to capture multiple frames per second with optimum brightness.

Parfocal autofocus method

A fast, passive autofocus method is used to obtain a parfocal effect in the lensless microscopy. In contrast to active autofocus, which depends on the physical characteristics of the system (i.e., lens properties and distance among the optical parts), passive autofocus employs an image processing and a computational analysis based on the information of the captured sample image49. Here, a series of images captured at multiple z-positions is processed by calculating the feature value of the sharpness to establish the sharpest image and a correct focal position. The passive autofocus is more reliable for our lensless microscope, since it does not rely on the lens usage and any mechanical moving parts. Nevertheless, it depends highly on the optimum setup arrangement of the light source, image sensor, algorithm complexity, and computational performance.

Automatic image reconstruction

The image captured by the CMOS sensor is defined by the area, occupied by the sample laid directly over it. A shadow image of the object is produced by the active pixel sensor or photodiode inside the camera chip. When a lens is attached onto the CMOS camera, the lens will gather and focus the light reflected from the object and subsequently direct it to the photodiode, enabling a sharp and crisp image creation. However, the focal length from the lens to the sample causes a sufficiently large gap between the CMOS, lens, and object, which decreases the field-of-view of the object. Thus, a lensless microscope that does not use any lens can minimize the gap between the CMOS and the object, providing a larger field-of-view. However, in the CMOS sensor, some transparent protection layers still exist above the active pixel sensor to filter, e.g., UV and infrared lights, to protect the photodiode, and to encapsulate the wire bonding. As consequence, the CMOS sensor cannot capture sharp images of the sample, especially for small scale objects, but rather produces a holographic image because the field changes due to the wave propagation. This phenomenon is related to the Rayleigh criterion of resolution and Abbe’s diffraction limit, where the created in-between distance affects the smallest individual object dimension that can be resolved. The image resolution depends on the wavelength of the light source, sample-to-sensor distance, and the diameter of the aperture.

The angular spectrum method is used to address this problem by reconstructing the wavefront to the desired z-position. Three key steps are needed to implement this algorithm. First, the field \({U}_{(x,y,{z}_{0})}\) is decomposed by transforming the image (x, y) in the plane \({z}_{0}\) into 2D plane waves via a spatial Fourier transform. Second, a complex notation of each plane wave is propagated into a 3D monochromatic plane wave plane in \({z}_{n}\) position. The wave vector of z depends on the sign, which is determined by whether the field propagates in positive z (above the sensor) or negative z direction (below the sensor). Third, the Fourier transform of the plane wave is inversed to the original domain to find the image reconstruction \({U}_{(x,y,{z}_{n})}\), as follows:

$${U}_{(x,y,{z}_{n})}={FT}^{-1}\left\{FT\left\{{U}_{(x,y,{z}_{0})}\right\}\left({f}_{x},{f}_{y}\right){e}^{2\pi i{z}_{n}\sqrt{{\frac{1}{\lambda }}^{2}-{f}_{x}^{2}-{f}_{y}^{2}}}\right\}(x,y)$$
(8)

where fx and fy are the spatial frequencies of 2D plane waves (x, y), respectively. Finally, the angular spectrum algorithm is completed by conducting all those three steps and adding all the propagated plane waves, allowing the calculation of the total propagated field using Eq. (8).

First-order histogram analysis

The histogram of an image represents the distribution of the intensity, which is described by a statistical function related to the number of pixels in the whole image62. A histogram constructed from a grayscale image holds intensity values I(u, v) in a range of:

$$I\left(u,v\right)\in [0, k-1]$$
(9)

with intensity I for each pixel having a value of intensity range k. In the first-order histogram, histogram \(H\left(i\right)\) represents the intensity levels of the image region and can be defined as:

$$H\left(i\right)=C\left\{\left(u,v\right)|I\left(u,v\right)=i\right\}$$
(10)

where C is the cardinality of a certain number set of pixels.

In holographic image processing, the sharpness of an image can be calculated by analyzing the data distribution. In our experiments, the intensity distribution is used to express the image sharpness level (i.e., level of blurriness or sharpness from the image), which can be found by analyzing its normality or non-normality level. Thus, the sharpest image along the wave propagation can be predicted. The closer the shape of image intensity distribution to the Gaussian curve is, the blurrier the image will be. On the other hand, the highest difference between the distribution shape and the Gaussian curve will yield the sharpest reconstructed image. Several types of data distribution analysis (i.e., skewness, chi-square, power divergence, Jarque–Bera, kurtosis, normality test, and Gini index) can be expressed as the distributional parameters to estimate and measure the inequality in the frequency distributions.

In general, a normal distribution will produce a classic and symmetrical bell-shaped curve. The frequency data distribution (\({f}_{(x)}\)) can be calculated by

$${f}_{(x)}= \frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{1}{2}{(\frac{i- \bar{x} }{\sigma })}^{2}}$$
(11)

with

$$\sigma =\sqrt{\frac{\sum {({x}_{i}- \bar{x} )}^{2}}{n}}$$
(12)

where \(\bar{x}\) and \(\sigma\) are mean and standard deviation, respectively.

Cell culture

Three different types of biological cells (i.e., MIN6, neuroblastoma (SH-SY5Y), and microalgae Prorocentrum minimum (P. minimum) cells) were cultured as objects of interest for the developed parfocal lensless holographic microscope. First, the insulin-secreting MIN6 cells (kindly provided by Jun-Ichi Miyazaki) were cultured in Dulbecco's modified Eagle medium (DMEM) with a high glucose concentration (4.5 g/l) of 25 mM, a final glutamine concentration of 6 mM, 10% fetal bovine serum (FBS), and penicillin/streptomycin in a humidified atmosphere of 95% air and 5% CO2 at 37 °C80. MIN6 pseudo-islets were generated by culturing MIN6 cells under the same conditions as described. To avoid cell attachment on the surface of the plastic material and to promote the formation of cell aggregates, dishes for suspension cell culture were used.

Second, the neuroblastoma cells were taken from the commercially available cell line SH-SY5Y (ATCC CRL-2266). This subclone of the original clone SK-N-SH cell line originates from a human neuroblastoma bone marrow biopsy and cultured in DMEM high glucose with 10% FBS81. It is easier and less expensive alternative to primary neuronal cultures since it is able to be differentiated into neuronal cell. Incubation was set in a humidified atmosphere at 37 °C and 5% CO2 to seed the undifferentiated cell inside 12-well cell culture plates. The viable cells, showing an irregular shape, are adherent to the surface of the plates, while non-viable cells are more roundish in shape and cannot adhere well to the surface. Third, the P. minimum, which is a eukaryotic single-cell algae belonging to the superclass of Dinoflagellate, was used in this study. P. minimum cells typically possess a triangular oval-round shape with length and width ranging from 17 to 25 µm and 14–23 µm, respectively82. Moreover, they are equipped with two flagella structures leading to their capability of actively moving through a medium. P. minimum culture was conducted at 20 °C in a petri dish containing artificial seawater medium under a day/night cycle of 12/12 h.

Additional information

Information regarding the limit of resolution of the lensless microscope, the intensity simulation, the imaging setup for long-term cell culture observation, the imaged object size and distance relationship, and the light source properties can be found in the Supplementary Information. The recorded videos of moving USAF 1951 target, neuroblastoma, and P. minimum cells can be seen in Supplementary Materials.

Data availability

The data that support the figures within this paper and other findings of this study are available from the corresponding authors upon reasonable request.

Code availability

The code that supports the lensless microscopy monitoring and image analysis of this study is available from the corresponding authors upon reasonable request.

References

  1. Grist, S. M. et al. Long-term monitoring in a microfluidic system to study tumour spheroid response to chronic and cycling hypoxia. Sci. Rep. 9, 1–13 (2019).

    CAS  Article  Google Scholar 

  2. Rappaz, B., Breton, B., Shaffer, E. & Turcatti, G. Digital holographic microscopy: a quantitative label-free microscopy technique for phenotypic screening. Comb. Chem. High Throughput Screen. 17, 80–88 (2014).

    CAS  PubMed  Article  Google Scholar 

  3. Tristan-Landin, S. B., Gonzalez-Suarez, A. M., Jimenez-Valdes, R. J. & Garcia-Cordero, J. L. Facile assembly of an affordable miniature multicolor fluorescence microscope made of 3D-printed parts enables detection of single cells. PLoS ONE 14, 1–17 (2019).

    Article  CAS  Google Scholar 

  4. Flight, R. et al. Automated noninvasive epithelial cell counting in phase contrast microscopy images with automated parameter selection. J. Microsc. 271, 345–354 (2018).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  5. Niioka, H. et al. Classification of C2C12 cells at differentiation by convolutional neural network of deep learning using phase contrast images. Hum. Cell 31, 87–93 (2018).

    CAS  PubMed  Article  Google Scholar 

  6. Ghosh, S. et al. In-vivo & in-vitro toxicity test of molecularly engineered PCMS: a potential drug for wireless remote controlled treatment. Toxicol. Rep. 5, 1044–1052 (2018).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  7. Satat, G., Tancik, M. & Raskar, R. Lensless imaging with compressive ultrafast sensing. IEEE Trans. Comput. Imaging 3, 398–407 (2017).

    MathSciNet  Article  Google Scholar 

  8. Singh, A. K., Pedrini, G., Takeda, M. & Osten, W. Scatter-plate microscope for lensless microscopy with diffraction limited resolution. Sci. Rep. 7, 1–8 (2017).

    Article  CAS  Google Scholar 

  9. Arce, S. H., Wu, P. H. & Tseng, Y. Fast and accurate automated cell boundary determination for fluorescence microscopy. Sci. Rep. 3, (2013).

  10. Allen, J. Robotic microscopy with the Nikon Ti2 for high-content analysis applications. Nat. Methods 13, i–ii (2016).

    Article  CAS  Google Scholar 

  11. Sobieranski, A. C. et al. Portable lensless wide-field microscopy imaging platform based on digital inline holography and multi-frame pixel super-resolution. Light Sci. Appl. 4, e346 (2015).

    CAS  PubMed  Article  Google Scholar 

  12. Moon, S. et al. Integrating microfluidics and lensless imaging for point-of-care testing. Biosens. Bioelectron. 24, 3208–3214 (2009).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  13. Wang, M., Feng, S. & Wu, J. Multilayer pixel super-resolution lensless in-line holographic microscope with random sample movement. Sci. Rep. 7, 1–8 (2017).

    ADS  PubMed  PubMed Central  Article  CAS  Google Scholar 

  14. Wu, Y. & Ozcan, A. Lensless digital holographic microscopy and its applications in biomedicine and environmental monitoring. Methods 136, 4–16 (2018).

    CAS  PubMed  Article  Google Scholar 

  15. Pégard, N. C. et al. Three-dimensional scanless holographic optogenetics with temporal focusing (3D-SHOT). Nat. Commun. 8, 1–14 (2017).

    ADS  Article  CAS  Google Scholar 

  16. Cetin, A. E. et al. Handheld high-throughput plasmonic biosensor using computational on-chip imaging. Light Sci. Appl. 3, e122–e122 (2014).

    CAS  Article  Google Scholar 

  17. Scholz, G. et al. Continuous live-cell culture imaging and single-cell tracking by computational lensfree LED microscopy. Sensors (Switzerland) 19, 1–13 (2019).

    CAS  Article  Google Scholar 

  18. Magnan, P. Detection of visible photons in CCD and CMOS: a comparative view. Nucl. Instrum. Methods Phys. Res. Sect. A Accel. Spectrom. Detect. Assoc. Equip. 504, 199–212 (2003).

    ADS  CAS  Article  Google Scholar 

  19. El Gamal, A. & Eltoukhy, H. CMOS image sensors. IEEE Circuits Devices Mag. 21, 6–20 (2005).

    Article  Google Scholar 

  20. Göröcs, Z. & Ozcan, A. Biomedical imaging and sensing using flatbed scanners. Lab Chip 14, 3248–3257 (2014).

    PubMed  PubMed Central  Article  Google Scholar 

  21. Göröcs, Z. et al. Giga-pixel fluorescent imaging over an ultra-large field-of-view using a flatbed scanner. Lab Chip 13, 4460–4466 (2013).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  22. Mariana, S. et al. Pinhole microLED array as point source illumination for miniaturized lensless cell monitoring systems. Proceedings 2, 866 (2018).

    Article  Google Scholar 

  23. Ferraro, P., Coppola, G., De Nicola, S., Finizio, A. & Pierattini, G. Digital holographic microscope with automatic focus tracking by detecting sample displacement in real time. Opt. Lett. 28, 1257 (2003).

    ADS  PubMed  Article  Google Scholar 

  24. Jericho, M. H. & Kreuzer, H. J. Coherent Light Microscopy 46 (Springer, Berlin, 2011).

    Google Scholar 

  25. Langehanenberg, P., von Bally, G. & Kemper, B. Autofocusing in digital holographic microscopy. 3D Res. 2, 1–11 (2011).

    Article  Google Scholar 

  26. Deng, Y. & Chu, D. Coherence properties of different light sources and their effect on the image sharpness and speckle of holographic displays. Sci. Rep. 7, 1–12 (2017).

    ADS  Article  CAS  Google Scholar 

  27. Repetto, L., Piano, E. & Pontiggia, C. Lensless digital holographic microscope with light-emitting diode illumination. Opt. Lett. 29, 1132 (2004).

    ADS  CAS  PubMed  Article  Google Scholar 

  28. Wasisto, H. S., Prades, J. D., Gülink, J. & Waag, A. Beyond solid-state lighting: miniaturization, hybrid integration, and applications of GaN nano-and micro-LEDs. Appl. Phys. Rev. 6, 041315 (2019).

    Article  CAS  Google Scholar 

  29. Bornemann, S. et al. Femtosecond laser lift-off with sub-bandgap excitation for production of free-standing GaN light-emitting diode chips. Adv. Eng. Mater. 22, 1901192 (2020).

    CAS  Article  Google Scholar 

  30. Markiewicz, N. et al. Micro light plates for low-power photoactivated (gas) sensors. Appl. Phys. Lett. 114, 053508 (2019).

    ADS  Article  CAS  Google Scholar 

  31. Schmidt, I. et al. Gold-modified indium tin oxide as a transparent window in optoelectronic diagnostics of electrochemically active biofilms. Biosens. Bioelectron. 94, 74–80 (2017).

    CAS  PubMed  Article  Google Scholar 

  32. Qomaruddin, et al. Visible light-driven p-type semiconductor gas sensors based on CaFe2O4 nanoparticles. Sensors (Switzerland) 20, 1–12 (2020).

    Article  CAS  Google Scholar 

  33. Mariana, S. et al. Vertical GaN nanowires and nanoscale light-emitting-diode arrays for lighting and sensing applications. ACS Appl. Nano Mater. 2, 4133–4142 (2019).

    CAS  Article  Google Scholar 

  34. Casals, O. et al. A parts per Billion (ppb) sensor for NO2 with microwatt (μW) power requirements based on micro light plates. ACS Sens. 4, 822–826 (2019).

    CAS  PubMed  Article  Google Scholar 

  35. Fatahilah, M. F. et al. Top-down GaN nanowire transistors with nearly zero gate hysteresis for parallel vertical electronics. Sci. Rep. 9, 1–11 (2019).

    CAS  Article  Google Scholar 

  36. Fatahilah, M. F. et al. 3D GaN nanoarchitecture for field-effect transistors. Micro Nano Eng. 3, 59–81 (2019).

    Article  Google Scholar 

  37. Ahn, D., Lee, J., Moon, S. & Park, T. Human-level blood cell counting on lens-free shadow images exploiting deep neural networks. Analyst 143, 5380–5387 (2018).

    ADS  CAS  PubMed  Article  Google Scholar 

  38. Mudanyali, O. et al. Compact, light-weight and cost-effective microscope based on lensless incoherent holography for telemedicine applications. Lab Chip 10, 1417–1428 (2010).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  39. Su, T. W., Erlinger, A., Tseng, D. & Ozcan, A. Compact and light-weight automated semen analysis platform using lensfree on-chip microscopy. Anal. Chem. 82, 8307–8312 (2010).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  40. Su, T.-W., Xue, L. & Ozcan, A. High-throughput lensfree 3D tracking of human sperms reveals rare statistics of helical trajectories. Proc. Natl. Acad. Sci. 109, 16018–16022 (2012).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

  41. Liu, T. et al. Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 1–13 (2019).

    ADS  Article  CAS  Google Scholar 

  42. Dubois, F., Schockaert, C., Callens, N. & Yourassowsky, C. Focus plane detection criteria in digital holography microscopy by amplitude analysis. Opt. Exp. 14, 5895 (2006).

    ADS  Article  Google Scholar 

  43. Langehanenberg, P., Kemper, B., Dirksen, D. & von Bally, G. Autofocusing in digital holographic phase contrast microscopy on pure phase objects for live cell imaging. Appl. Opt. 47, D176 (2008).

    PubMed  Article  Google Scholar 

  44. Liebling, M. & Unser, M. Autofocus for digital Fresnel holograms by use of a Fresnelet-sparsity criterion. J. Opt. Soc. Am. A 21, 2424 (2004).

    ADS  MathSciNet  Article  Google Scholar 

  45. Fatih Toy, M. et al. Accelerated autofocusing of off-axis holograms using critical sampling. Opt. Lett. 37, 5094 (2012).

    ADS  CAS  PubMed  Article  Google Scholar 

  46. de Groot, P. J. & Biegen, J. F. Interference microscope objectives for wide-field areal surface topography measurements. Opt. Eng. 55, 074110 (2016).

    ADS  Article  Google Scholar 

  47. de Groot, P. J. & Biegen, J. F. A new class of wide-field objectives for 3D interference microscopy. Opt. Meas. Syst. Ind. Insp. IX 9525, 95250N (2015).

    Google Scholar 

  48. Simon, J. M. & Comastri, S. A. The compound microscope: Optical tube length or parfocalization?. Eur. J. Phys. 26, 1101–1105 (2005).

    Article  Google Scholar 

  49. Wei, L. & Roberts, E. Neural network control of focal position during time-lapse microscopy of cells. Sci. Rep. 8, 1–10 (2018).

    ADS  Article  CAS  Google Scholar 

  50. Koho, S., Fazeli, E., Eriksson, J. E. & Hänninen, P. E. Image quality ranking method for microscopy. Sci. Rep. 6, 1–15 (2016).

    Article  CAS  Google Scholar 

  51. Cabazos-Marín, A. R. & Álvarez-Borrego, J. Automatic focus and fusion image algorithm using nonlinear correlation: Image quality evaluation. Optik (Stuttg) 164, 224–242 (2018).

    ADS  Article  Google Scholar 

  52. Zhang, Y., Wang, H., Wu, Y., Tamamitsu, M. & Ozcan, A. Edge sparsity criterion for robust holographic autofocusing. Opt. Lett. 42, 3824 (2017).

    ADS  PubMed  Article  Google Scholar 

  53. Chiang, C. S. et al. The study of auto-focus system for biomedical digital microscope. In 3rd International Conference on Bioinformatics Biomedical Engineering. iCBBE 2009 1–4 (2009). https://doi.org/10.1109/ICBBE.2009.5163670.

  54. Memmolo, P., Paturzo, M., Javidi, B., Netti, P. A. & Ferraro, P. Refocusing criterion via sparsity measurements in digital holography. Opt. Lett. 39, 4719 (2014).

    ADS  PubMed  Article  Google Scholar 

  55. A-Iyeh, E. & Peters, J. F. Gini index-based digital image complementing in the study of medical images. Intell. Decis. Technol. 9, 209–218 (2015).

    Article  Google Scholar 

  56. Zhang, Y., Wang, H., Wu, Y., Ozcan, A. & Tamamitsu, M. A robust holographic autofocusing criterion based on edge sparsity: comparison of Gini index and Tamura coefficient for holographic autofocusing based on the edge sparsity of the complex optical wavefront. In Quantitative Phase Imaging IV 19 (eds Popescu, G. & Park, Y.) (SPIE, Bellingham, 2018). https://doi.org/10.1117/12.2291179.

    Chapter  Google Scholar 

  57. Senaras, C., Khalid Khan Niazi, M., Lozanski, G. & Gurcan, M. N. DeepFocus: detection of out-of-focus regions in whole slide digital images using deep learning. PLoS ONE 13, 1–13 (2018).

    Google Scholar 

  58. Dubois, F., Yourassowsky, C., Callens, N., Minetti, C. & Queeckers, P. Applications of digital holographic microscopes with partially spatial coherence sources. J. Phys. Conf. Ser. 139, 012027 (2008).

    Article  Google Scholar 

  59. Saastamoinen, K., Tervo, J., Turunen, J., Vahimaa, P. & Friberg, A. T. Spatial coherence measurement of polychromatic light with modified Young’s interferometer. Opt. Exp. 21, 4061 (2013).

    ADS  Article  Google Scholar 

  60. Wong, A. M. H. & Eleftheriades, G. V. An optical super-microscope for far-field, real-time imaging beyond the diffraction limit. Sci. Rep. 3, 1715 (2013).

    ADS  CAS  PubMed  PubMed Central  Article  Google Scholar 

  61. Chen, X., Tian, G., Ding, S. & Wu, J. Investigation of skewness feature for evaluation of defects using eddy current pulsed thermography. IEEE Sens. J. 19, 12118–12125 (2019).

    ADS  CAS  Article  Google Scholar 

  62. Just, N. Improving tumour heterogeneity MRI assessment with histograms. Br. J. Cancer 111, 2205–2213 (2014).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  63. Kesavan, S. V. et al. High-throughput monitoring of major cell functions by means of lensfree video microscopy. Sci. Rep. 4, 1–11 (2014).

    Google Scholar 

  64. Horgan, D. & Murphy, C. C. On the convergence of the Chi square and noncentral Chi square distributions to the normal distribution. IEEE Commun. Lett. 17, 2233–2236 (2013).

    Article  Google Scholar 

  65. Salem, O., Serhrouchni, A., Mehaoua, A. & Boutaba, R. Event detection in wireless body area networks using kalman filter and power divergence. IEEE Trans. Netw. Serv. Manag. 15, 1018–1034 (2018).

    Article  Google Scholar 

  66. Núñez, J. A., Contreras-Valdez, M. I. & Franco-Ruiz, C. A. Statistical analysis of bitcoin during explosive behavior periods. PLoS ONE 14, 1–22 (2019).

    Google Scholar 

  67. Zhang, H., Wu, H. C. & Lu, L. Analysis and algorithm for robust adaptive cooperative spectrum-sensing. IEEE Trans. Wirel. Commun. 13, 618–629 (2014).

    Article  Google Scholar 

  68. Bradley, D. & Morris, J. M. On the performance of negentropy approximations as test statistics for detecting sinusoidal RFI in microwave radiometers. IEEE Trans. Geosci. Remote Sens. 51, 4945–4951 (2013).

    ADS  Article  Google Scholar 

  69. Ruan, S., Jaggi, C., Xue, J., Fadili, J. & Bloyet, D. Brain tissue classification of magnetic resonance images using partial volume modeling. IEEE Trans. Med. Imaging 19, 1179–1187 (2000).

    CAS  PubMed  Article  Google Scholar 

  70. Bian, Y., Zhang, Y., Yin, P., Li, H. & Ozcan, A. Optical refractometry using lensless holography and autofocusing. Opt. Exp. 26, 29614 (2018).

    ADS  CAS  Article  Google Scholar 

  71. Bian, Y. et al. Portable multi-spectral lens-less microscope with wavelength-self-calibrating imaging sensor. Opt. Lasers Eng. 111, 25–33 (2018).

    Article  Google Scholar 

  72. de Guélis, T. V., Shcherbakov, V. & Schwarzenböck, A. Diffraction patterns from opaque planar objects simulated with Maggi-Rubinowicz method and angular spectrum theory. Opt. Exp. 27, 9372 (2019).

    ADS  Article  Google Scholar 

  73. Wu, P., Kazys, R. & Stepinski, T. Analysis of the numerically implemented angular spectrum approach based on the evaluation of two-dimensional acoustic fields. Part I. Errors due to the discrete Fourier transform and discretization. J. Acoust. Soc. Am. 99, 1339–1348 (1996).

    ADS  Article  Google Scholar 

  74. Wu, Y. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light Sci. Appl. 8, 1–7 (2019).

    Article  CAS  Google Scholar 

  75. Ramzy, L., Metwally, A. A., Nasr, M. & Awad, G. A. S. Novel thymoquinone lipidic core nanocapsules with anisamide—polymethacrylate shell for colon cancer cells overexpressing sigma receptors. Sci. Rep. https://doi.org/10.1038/s41598-020-67748-2 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  76. Obayemi, J. D. et al. LHRH-conjugated drugs as targeted therapeutic agents for the specific targeting and localized treatment of triple negative breast cancer. Sci. Rep. 10, 1–18 (2020).

    Article  CAS  Google Scholar 

  77. Su, C. et al. Mapping effector genes at lupus GWAS loci using promoter Capture-C in follicular helper T cells. Nat. Commun. 11, 3294 (2020).

    ADS  CAS  PubMed  PubMed Central  Article  Google Scholar 

  78. Albeanu, D. F., Soucy, E., Sato, T. F., Meister, M. & Murthy, V. N. LED arrays as cost effective and efficient light sources for widefield microscopy. PLoS ONE 3, e2146 (2008).

    ADS  PubMed  PubMed Central  Article  CAS  Google Scholar 

  79. Etcheverry, M. E., Pasquale, M. A. & Garavaglia, M. Photodynamic therapy of HeLa cell cultures by using LED or laser sources. J. Photochem. Photobiol. B Biol. 160, 271–277 (2016).

    CAS  Article  Google Scholar 

  80. Schulze, T. et al. Metabolic amplification of insulin secretion is differentially desensitized by depolarization in the absence of exogenous fuels. Metabolism 67, 1–13 (2017).

    CAS  PubMed  Article  Google Scholar 

  81. Meiser, J., Weindl, D. & Hiller, K. Complexity of dopamine metabolism. Cell Commun. Signal. 11, 34 (2013).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  82. Heil, C. A., Glibert, P. M. & Fan, C. Prorocentrum minimum (Pavillard) Schiller. Harmful Algae 4, 449–470 (2005).

    CAS  Article  Google Scholar 

Download references

Acknowledgements

This work was performed within LENA-OptoSense funded by the Lower Saxony Ministry for Science and Culture (N-MWK), the European project of ChipScope funded by the European Union’s Horizon 2020 research and innovation program under grant agreement no. 737089, the Quantum and Nano-Metrology program (Quanomet) funded by N-MWK, and Germany’s Excellence Strategy of EXC-2123 QuantumFrontiers – 390837967 funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation). Agus Budi Dharmawan thanks Indonesia Endowment Fund for Education (LPDP) and Directorate General of Higher Education at Ministry of Education and Culture of the Republic of Indonesia for the financial support in form of Ph.D. scholarship (BUDI-LN with contract number of 201710220111530) and Indonesian-German Center for Nano and Quantum Technologies (IG-Nano) for the research support. Shinta Mariana is grateful to Georg-Christoph-Lichtenberg for the Ph.D. scholarship (Tailored Light). Technical support from Vladislav Agluschewitsch and Karl-Heinz Lachmund is also acknowledged.

Funding

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and Affiliations

Authors

Contributions

A.B.D. and H.S.W. conceived the idea, concept, and experiments of parfocal autofocus features in lensless holographic microscopy. A.B.D. built the lensless microscope, programmed the software and codes for parfocal autofocus features, developed automatic image reconstruction method, and conducted the experiments. A.B.D., S.M., and G.S. designed and constructed LED-based pinhole setup. A.B.D., K.T., and H.S.W. proposed and discussed different statistical approaches for wave propagation distribution analysis. A.B.D., G.S., S.M., P.H., and T.S. performed the experiments using biological cells. P.H., K.H., T.S., and I.R. prepared for different cell cultures (MIN6, neuroblastoma, and P. minimum). A.B.D., S.M., G.S., and H.S.W. interpreted and discussed the microscope performance. A.B.D, S.M., and H.S.W. wrote the initial manuscript. A.B.D., S.M., G.S., M.G.-S., H.S.W., and A.W. revised the manuscript. H.S.W. and A.W. managed and led the projects. All authors approved the final manuscript.

Corresponding authors

Correspondence to Agus Budi Dharmawan, Hutomo Suryo Wasisto or Andreas Waag.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Video 1.

Supplementary Video 2.

Supplementary Video 3.

Supplementary Information.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dharmawan, A.B., Mariana, S., Scholz, G. et al. Nonmechanical parfocal and autofocus features based on wave propagation distribution in lensfree holographic microscopy. Sci Rep 11, 3213 (2021). https://doi.org/10.1038/s41598-021-81098-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41598-021-81098-7

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing