Introduction

The accurate determination of cell morphology is critical to many aspects of biomedicine1,2,3,4. For instance, histopathology can indicate the stage of cancer based on the organization and shape of cells present in a patient biopsy; yet, it still relies on experienced physicians to visually recognize the qualitative differences in cell phenotype4,5. If morphological analysis could be performed quantitatively, the greater potential to reveal subtle disparities in cell phenotype could radically improve the way we grade cancer3,4,5,6. Meanwhile, the drug screening industry has actively adopted computer-guided morphological assessment to uncover the potential of new drugs7,8,9. High content screening (HCS) platforms allow us to gain access to rich phenotypic information that can be quantitatively analyzed and statistically distinguished, but the priority of existing platforms is to foster the speed of image processing so some measurement resolution is often conceded7,8,10. Therefore, an advance in technology that improves our ability to rapidly and accurately quantify cell morphology can greatly impact the biomedical community.

However, the complexity of the fluorescent signal from a typical cell within a digital image presents a major barrier for the generation of accurate cell boundaries from segmentation12,13,14,15. The fastest way to generate a cell boundary is to compare the pixel values of an image to a single intensity threshold, which may be determined quickly using histogram-based approaches10,11,12,13. These kinds of segmentation strategies only roughly approximate cell boundaries and consequently produce a great amount of error in phenotypic parameters that would not otherwise be present during subsequent measurement and statistical analysis5,6,7,8. This diminishes the capacity to advance cellular biophysics using detailed morphological information to support the previously mentioned applications. In this regard, more sophisticated image processing methods were developed to offer better boundary resolution, but they often require prolonged computation time, user-interaction or specialize training for proper implementation and remain outside of the mainstream7,8,16,17,18,19,20,21. Hence, the development of a quick and accurate segmentation strategy to deliver rich cell phenotype information could dramatically advance patient diagnostics and drug discovery.

In a microscope system, the signal from a specimen first exists as emitted photons, which follow a light path to a photo-detector to be converted into a digital signal. During this process, several sources of error can obscure the true signal and cause a loss of spatial resolution in raw images. Light gathered from a fluorescent specimen is subject to the influence of the acquisition systems and its intensity is distributed spatially based on the system22. The interference from the path as the light passes through the microscope system from the fluorescent specimen to the detector makes the microscope essentially act as a physical spatial low pass (SLP) filter and blends fine image features, like thin lines or edges, to reduce the local contrast23.

This work introduces a segmentation scheme that takes advantage of the concept of spatial filtering to effectively address segmentation pitfalls that arise naturally during image acquisition. The method requires minimal user interaction, is computationally inexpensive; meanwhile, can produce accurate cell boundaries despite a range of image conditions and cell morphologies. Hence, this approach can be easily implemented as an add-on for existing open source software packages to improve their ability to reliably segment fluorescent cell images10,11 and can directly promote high content quantitative measurements of cell features, which are not accurately provided by current methods.

Results

Intrinsic features of fluorescent cell images impede segmentation

Cells expressing fluorescence proteins or dyed by a certain fluorescence reagent emit a light signal that varies in intensity spatially throughout the cell due to morphological features or spatial exclusion by cellular organelles and lipid membranes. Intensity profiles of raw images that capture the cell signal also contain a variable noise that intrinsically arises during image acquisition. In a histogram of the raw pixel intensities, the pixels corresponding to the background region of the image typically generate a large peak in occurrence at lower intensity values while pixels corresponding to the cell signal are distributed broadly among the higher intensity values, appearing as a ‘tail’ (Figure 1 a).

Figure 1
figure 1

The light signal from a fluorescent cell sample greatly affects digital image intensity profiles.

(a) A histogram of the intensity distribution in a typical fluorescent image of a NIH 3T3 fibroblast is shown. The raw image (both inset) was captured using τ = 100 ms and 1000 bins were used to construct the histogram. A segmentation threshold was calculated using the mixture of Gaussian (MoG) method and plotted as the vertical, red line, separating the histogram into a background class (gray) and the remaining pixel intensities. The boundary that the MoG threshold produces is drawn in red over the left inset image. A second segmentation threshold was calculated using Otsu's method and plotted as the vertical, blue line. The boundary that Otsu's threshold produces is drawn in blue over the right inset image. Arrows indicate important regions of the image that were segmented differently by each method. (b) Histograms of cropped images from two regions in the raw cell image are shown. The cropped image of one cell protrusion (left inset) produces a histogram (gray) with a background class of pixels and small ‘tail’ corresponding to the cell protrusion signal. The boundary of a MoG threshold is shown. The cropped image of the cell body (right inset) also produces a histogram (black) with a background class and a tail. The boundary of an Otsu threshold is shown. The peak intensity (arrow) for the cropped image of the protrusion falls into the background pixel class of the cell body histogram. (c) The intensity profile of a fluorescent image changes depending on the exposure time (τ) during image acquisition. Images taken at τ = 50, 100 and 300 ms (inset) generated histograms colored in red, black and blue, respectively. As τ increases, the average image intensity also increases and the background class appears to develop a wider and shorter intensity distribution. (d) The average background intensity of each different exposure was subtracted to adjust the intensity of each histogram for comparison. A threshold (vertical red line) was chosen by eye to segment each adjusted image (inset).

Full size image

Segmentation is the process of identifying the pixels that correspond to the cell signal in an image. For instance, segmentation by a single intensity threshold aims to distinguish the cell signal from the background by partitioning the image into two classes of pixels12,13. Setting the threshold near to the peak of the histogram may omit much of the background and grant good segmentation for low intensity cell features, such as cell protrusions; however, it also includes some of the background noise and any background pixels that have been influenced by the glare of nearby cellular fluorescence. Alternatively, setting a threshold further from the histogram peak will omit nearly all background pixels, but may also omit low intensity regions of the cell signal.

Closer examination of the intensity profile of a cell image over the cell body and one protrusion reveals that the peak intensity value of the signal from the protrusion falls below the threshold value that could properly segment the cell body region of the image (Figure 1 b). This is a natural consequence of the image acquisition process, which reveals that the pixels corresponding to the cell boundary in one region of an image do not necessarily possess similar intensity values to cell boundary pixels in another region of the same image, preventing accurate segmentation by a single threshold.

Furthermore, the intensity profile of a cell image can alter depending on the level of fluorescent protein expression, the current shape configuration of the cell and/or the parameters of image acquisition. For example, the exposure time of the camera can be manipulated to allow more or less photons to reach the camera, resulting in images with drastically different intensity profiles (Figure 1 c). As exposure time is increased, the average intensity value of the image will increase and a boundary determined solely by the statistics of the background noise will progressively broaden (Figure 1 d). This effect of average image intensity could cause some cells to seem larger after segmentation only due to the apparent signal rather than physical differences among cells for the aforementioned reasons. Segmentation strategies that do not account for this effect might generate inaccurate boundaries from cell images and further introduce error into later phenotypic measurements.

Automated spatial filtering generates rapid and accurate cell boundaries

Fluorescent cell images must be subjected to some level of processing to avoid the pitfalls inherent in the intensity profile of raw images and obtain a good segmentation result. Though some sophisticated methods exist to segment cell images16,17,18,19,20,21, we have developed a new procedure that generates accurate segmentation results without the need for high computational power or any user-interaction (see Supplementary Figure S1 online for comparison). Briefly, a spatial low pass filter is applied to raw images and the resulting low pass result is re-scaled and then subtracted from the raw image to produce the final processed image. This section describes the process in greater detail.

In a raw image, edges of objects are visually apparent because they possess high contrast from one pixel to the next; this contrast is considered to be high spatial frequency content in the image. On the other hand, the gradual changes in intensity that prevent good segmentation are considered to be low spatial frequency content. In plain terms, high spatial frequencies resemble tv static, edges or narrow stripes that are close together while low spatial frequencies appear as uneven illumination or blurriness. Thus, a potentially effective approach to aid segmentation is spatial filtering, which allows us to process images by enhancing or reducing only one type of spatial frequency content13,24.

For example, sharpening of raw images will enhance the high spatial frequencies and improve contrast by making the intensity differences between edge pixels more drastic. However, straightforward application of a high pass spatial filter also enhances the power of the pixel noise, which also varies from pixel-to-pixel. On the other hand, a low pass spatial filter can be applied first and then subtracted from the raw image values to achieve sharpening without aggravating the pixel noise12,25 (Figure 2 a). The applied low pass filter must match the scale of the object it is meant to reveal through processing and its dimensions can be calculated using an initial threshold to approximate the size of a cell for sharpening applications (see Supplementary Figure S2 online). However, without further processing, such sharpened images often identify contrast within the region of cell signal due to variability in cell morphology and could generate false edges during segmentation12,26 (see Supplementary Figure S1 online).

Figure 2
figure 2

Subtraction of a re-scaled spatial low pass result from a raw image drastically improves subsequent segmentation.

(a) Three images are shown: a raw fluorescence image (left), the result of applying a spatial low pass filter to the raw image (middle) and the result of subtracting the low pass image values from the raw image values (right), which produces a sharpened image, increasing contrast at the edges of the cell (arrows). (b) Selected intensity profiles for the raw image (black) and the low pass result (red) are plotted according to the index of the global minimum in the sharpened image. Part of the low pass intensity profile exceeds the raw image intensity (arrow). (c) Once the low pass result is subtracted from the raw image values (black), the sharpened image intensity profile (red) will contain a global minimum (red arrow). The vertical blue line demonstrates the index of the minimum point in the sharpened image. (d) The ratio of values between the raw (black) and low pass (red) image at the pixel index corresponding to the global minimum (vertical blue line) is set as a scale factor to reduce the intensity of the low pass result (blue arrows). The scaled low pass result (blue curve) now appears to ‘fit’ below the raw image intensity profile. (e) Subtraction of the scaled low pass result from the raw image values generates a new profile (black) drastically improving the ability to perform segmentation by a single MoG threshold (red dotted line). This process treats the index of the global minimum (vertical blue line) as part of the background since the value of the final image at that point equals the mean background intensity. (f) After processing, the histogram of the final image appears to feature two distinct distributions (arrow). The boundary that a MoG threshold (vertical red line) generates is shown (inset). (g) Histograms of two cropped regions in the final image are shown. Both the cropped image corresponding to one cell protrusion (left inset) and part of the cell body (right inset) can now be segmented by a single MoG threshold.

Full size image

This work introduces the key step of re-scaling the low pass result before subtraction from a raw image to avoid the issue of false edges and benefit segmentation greatly. In a cellular fluorescence image, the intensity profile of the low pass result always exceeds the value of the raw image intensity profile at high contrast regions in the image (Figure 2 b). After subtraction of the low pass result from the raw image, this disparity generates a global minimum pixel value in the high pass image (Figure 2 c). At that pixel position, the intensity value of the raw image and the low pass result can be extracted to create a re-scaling factor. Then, direct multiplication of the re-scaling factor with the entire low pass image (subtracting the average background intensity beforehand) reduces its intensity values so that its intensity profile can ‘fit’ underneath the original raw intensity profile (Figure 2 d). Notably, the raw image and re-scaled low pass result will have equivalent intensity at the point of the global minimum. Ultimately, subtraction of the re-scaled low pass result from the raw image intensity values yields a new image with boundary pixels that are nearly within a single plane of intensity (Figure 2 e).

Inspection of a histogram for the final processed image reveals a ‘dip’ between the background peak and cell signal tail and subsequent segmentation of the new image reveals excellent agreement with the apparent cell boundary (Figure 2 f). Further inspection of two cropped regions within the original image demonstrates how the respective histograms have automatically been ‘adjusted’ for differences in background level intensity. After processing, the cell signal is clearly distinguished by a single threshold in both cropped images (Figure 2 g). A closer look at each separate histogram suggests that processing has augmented the raw image so that the mean background intensity has been adjusted locally due to the subtraction of the re-scaled low pass result, greatly facilitating the ease of segmentation by a single threshold.

Automated spatial filtering is robust

Use of this automated approach to process images taken at various exposure times revealed segmentation results that were highly consistent (Figure 3 a). In addition, application of a de-noising algorithm, such as anisotropic diffusion27, during the process before segmentation can remove any remaining interference from the pixel noise at boundaries and generates a highly accurate segmentation result. In comparison to boundaries from raw images, boundaries determined by this segmentation strategy are highly accurate and consistent despite changes in image intensity profiles.

Figure 3
figure 3

Automated segmentation is robust and accurate.

(a) Fixed cells imaged using progressively longer exposure times generate progressively brighter images. A MoG threshold was used to segment each image and generate cell boundaries (top row) for images taken at τ = 50, 100 and 300 ms, colored red, white and blue, respectively. Each image was processed using the developed spatial filtering method and segmented by MoG thresholds to generate cell boundaries (middle row). Pixel noise remained an issue after processing, especially for dimmer images with less signal-to-noise ratio. Application of a de-noise algorithm (anisotropic diffusion) to the processed images before segmentation attenuated the pixel noise and greatly enhanced the final cell boundaries. (b) Variable pressure scanning electron microscopy (vpSEM) was performed to image cell samples at a higher resolution than fluorescence microscopy is capable without a coating such as gold. Segmentation was performed on raw images of the fluorescence signal (τ = 100 ms) based on the developed approach. The boundary of that segmentation was scaled and rotated to match the length scale of the vpSEM image and overlaid onto it (dotted black line). (c) Dynamic interference contrast (DIC) microscopy was performed to capture an image of the cell sample for further comparison. The same boundary generated from analysis of a fluorescence image was scaled to the length scale of the DIC image and overlaid onto it.

Full size image

The boundaries extracted by this method can be directly compared to other types of cell images taken at higher resolution to help support the segmentation accuracy. Variable pressure scanning electron microscopy (vpSEM) was performed on the same exact fixed cells that were imaged by fluorescence microscopy. This SEM technique does not require surface modification or coatings and was chosen for its ability to generate ultra-high resolved images of cells without damaging their integrity28. The boundary acquired from a fluorescent image was expanded and rotated (an affine transformation) to overlay onto the vpSEM image, showing excellent agreement with the apparent vpSEM image boundary (Figure 3 b). Similarly, comparison to a differential interference contrast (Nomarski) image of the same cell, taken without using the binning feature of the camera, showed very close agreement with the visually apparent boundary (Figure 3 c). From the cross-comparison, it suggests that the proposed method could detect subtle differences in cell morphology that may have been previously undetectable.

Discussion

Even ideally prepared samples, free of debris or out-of-focus objects, produce images that still contain segmentation pitfalls as a consequence of the image acquisition process. Various imaging process techniques have been developed to attack this issue from different perspectives depending on the training and tools available to individual researchers, each aiming to provide an acceptable measurement of relevant cellular information. However, image processing algorithms that are not capable of effective segmentation without prolonged computation or user-interaction only provide a roadblock to HCS applications; hence, less precise segmentation strategies are chosen to promote speed. Our work presents a straightforward segmentation strategy to automate the processing of raw images and tackle the existing issues that hinder the development of a more powerful class of assays based on morphology and localization.

The developed segmentation strategy possesses several advantages. Firstly, it utilizes parameters taken directly from images so that user involvement is minimal. Secondly, it is not iterative so that a stopping criterion is unnecessary and it can be performed rapidly. Thirdly, this method results in consistent boundaries even for heterogeneous intensity profiles and for a wide range of signal-to-noise ratios from various acquisition conditions. Hence, this method possesses the capacity to capture highly dynamic cell behavior, regardless of cell types that exhibit a wide range of morphologies and protrusions.

In conclusion, this interdisciplinary work will directly impact the transition from qualitative observation to quantitative measurement of cell phenotype by not only providing effective segmentation, but also fostering automated, high content cellular information. This will grant the capability to generate the massive amount of biophysical data necessary to identify important morphological subtleties and propel biomedicine research into a new era.

Methods

Fluorescence microscopy

A Nikon TE-2000 microscope (Nikon, Melville, NY), equipped X-Cite 120 PC fluorescent light source (EXFO, Ontario, Canada) and a Cascade:1K CCD camera (Roper Scientific, Tucson, AZ), was used to acquire microscopic images at 20× magnification. In addition, an on-stage incubator (In Vivo Scientific, St. Louis, MO) with temperature control and a supplementary CO2 system was operated along with the microscope to maintain the experimental environment at 10% CO2 and 37°C.

Cell culture, plasmids and transfection

NIH 3T3 fibroblasts (ATCC, Manassas, VA) were maintained in DMEM with 10% fetal bovine serum and 1% L-glutamine (all purchased from Mediatech Inc., Manassas, VA) in a humidified incubator at 37°C and 10% CO2. For transfection, cultured cells were prepared on fibronectin-coated glass bottom dishes (MatTek, Ashland, MA) under normal culturing conditions for 24 hours. Then the cell culture was incubated in 1 ml Opti-MEM I Reduced Serum Media (Invitrogen, Carlsbad, CA), which was supplemented with a premixed transfection solution, containing 4 μg pERFP-C-RS plasmid (Clontech, Mountain View, CA) and 5 μl Lipofectamine 2000 (Invitrogen), for 1 hour to introduce the plasmids into the cells. Afterward, the cells were then maintained in normal cell culture media for further experiments.

Variable pressure scanning electron microscopy

Highly resolved cell images were acquired by an S-3000N variable pressure scanning electron microscope (Hitachi, San Francisco, CA). An acceleration voltage of 5 kilovolts and 10.3-mm working distance were used. This type of scanning electron microscopy (SEM) uses a low vacuum (60 Pa) to enable the imaging of samples without a special coating such as gold. Cells expressing red fluorescence protein (RFP) were passed to glass slides and allowed to adhere before being fixed in glutaraldehyde for 10 minutes, washed in phosphate buffered saline and stored in de-ionized water. Cell samples were allowed to air dry before the vpSEM experiments and were imaged using fluorescence microscopy immediately before and after the vpSEM to allow for direct comparison of the same cells between the two techniques. In this case, sample shrinkage was not a concern since only the fluorescent signal was of interest; hence, special freeze-drying was not used to prepare samples.

Image analysis and threshold determination

Raw images were processed using a custom-designed routine in Matlab (The MathWorks, Natick, MA) as described in the results section.