Introduction

Multispectral imaging systems have been used to measure the absorption, reflection and transmission characteristics of numerous materials across a wide range of spectral bands. Because various materials have different spectral reflectance and transmission signatures, multispectral images provide unique information about a material's composition. This capability has given rise to a wide range of applications that range from remote sensing to biomedical imaging1. Multispectral imagers have been constructed using a variety of technologies, including mechanically switched bandpass filters, electronically tunable bandpass filters and fixed gratings2,3. In general, these technologies require scanning (spectral or spatial) to form a 3D data cube that consists of a series of 2D images of the same scene, where each 2D image represents a specific spectral band. Due to the requirement for scanning, the systems are generally incapable of capturing fast-moving objects in a scene. Non-scanning systems have been introduced to overcome this speed limitation. For example, the computed-tomography imaging spectrometer (CTIS) utilizes a fixed grating to capture the dispersion pattern of a scene on a monochromatic imaging sensor4. The CTIS employs a computationally expensive iterative reconstruction method to solve the 3D spatio-spectral data cube, which allows for real-time data acquisition but necessitates off-line spectral image reconstruction and display4. Researchers have recently demonstrated the real-time acquisition and display of multispectral images by combining a commercially available four-color camera and a four-band filter5. The device was capable of capturing four spectral images, each representative of a narrow spectral range within the visible part of the spectrum and was extensible to eight spectral images when a second camera was introduced into the setup5.

Recent work with nanohole arrays (NHAs) in an optically thick metal film has shown that NHAs behave as spectral filters due to the presence of extraordinary optical transmission (EOT) or resonances in the visible and near-infrared regimes6,7,8. The EOT is due to surface plasmons (SPs), where light interacts with the oscillation of electrons at the interface between a metal and a dielectric material6. The transmission resonances of metallic NHAs can be tuned in the visible and infrared regimes by simply changing the spacing between the holes. Furthermore, the bandwidth of the transmission resonance can be tuned by changing the hole size9. The design and fabrication of periodic metallic structures at sub-micron spatial scales are easily achieved with current fabrication technologies and can be applied to complementary metal-oxide semiconductor (CMOS) technology10,11,12. For example, structures have been fabricated using inexpensive one-time lithography on a single layer of metal film on a CMOS imaging sensor11,12. Also, a metallic C-shaped hole has been observed to increase the photocurrent density of a CMOS detector three-fold due to the presence of surface plasmons in the metallic structure13. Furthermore, periodic metallic nanostructures do not suffer significantly from performance degradation or crosstalk effects in spectral filtering applications when they are closely packed10,14.

However, transmission resonances of conventional NHAs are not efficient at separating colors due to low out-of-band blocking and broad resonance optical bandwidth. Therefore, we recently developed a NHA with cavities that benefited from SP-energy-matching properties (so called SP-matched NHA). The SP-matched NHA provided higher resonance transmission efficiency, narrower resonance bandwidth and higher out-of-band blocking compared to conventional NHAs15. Therefore, a SP-matched NHA is better suited to spectral band pass filtering compared to a conventional NHA.

Motivated by some of the advantages SP-matched NHAs possess over conventional NHAs and competing optical filter technologies and by the ultimate desire to perform 2D snapshot multispectral imaging in the visible and near-infrared regimes simultaneously, our goal was to design and develop an NHA-based 2D snapshot multispectral imaging system capable of broad spectral sensitivity suitable for the spectral unmixing of a scene into multiple bands. Our approach was to design and fabricate a grid of closely spaced NHAs in a gold film and to utilize this structure as a multi-element mosaic filter for multispectral transmission imaging. The transmission spectra of the fabricated NHAs were analyzed with respect to the relative location and transmission efficiency of the resonance peaks and minima. A correlation analysis among the NHAs within the mosaic filter provided insight into the number of spectral bands that could be unmixed. The device performance was tested with the spectral unmixing of illumination patterns presented to the device into defined spectral bands. Finally, the device was inserted into the optical path of an imaging system, which was used for snapshot 2D multispectral imaging in transmission and reflection modes.

Results

Figure 1 (a) displays a schematic of the mosaic filter, which was composed of 16 blocks in a 4-by-4 array; each block contained 9 NHAs in a 3-by-3 grid and each NHA within each block had a unique periodicity. The 9 NHAs in each block had holes distributed in a square lattice arrangement spaced at various periodicities from 360 nm to 440 nm in 10 nm increments. Each NHA was 15 μm × 15 μm in size and was spaced 15 μm from each neighbor. We employed electron-beam lithography (EBL) to fabricate the NHAs in a 100-nm gold film on a Pyrex substrate9. After fabrication, the substrate beneath each hole was partially wet-etched to produce a cavity beneath each hole15. The NHA structure with cavities enabled SP-energy matching between the top and bottom of the gold film and resulted in an enhanced resonance-transmission efficiency and greater blocking at the transmission minimum, as shown in our previous work15. Figure 1(a) displays an SEM image of a SP-matched NHA with a 100-nm deep cavity below each hole within the Pyrex substrate. Polymethyl methacrylate (PMMA 950 A2, MicroChem, Newton, MA) was spin-coated onto the top surface of the perforated gold film to a thickness of 200 nm. The PMMA was found to fill each cavity and to red-shift the resonance position of each NHA. The ratio between the total hole area to background area for each NHA was kept approximately constant (~0.09) to obtain similar (1,0)-resonance-transmission efficiencies and narrow-resonance bandwidths across all NHAs.

Figure 1
figure 1

(a) Schematic of the multispectral device, which consists of 4 × 4 block array, where each block contains a 3 × 3 grid of cells (each cell represents one NHA with a unique periodicity P (in nm)); a SEM image of a NHA with 100-nm deep cavities. (b) A color image of the device. (c) Optical transmission spectra for nine NHAs within a single block. d) Positions of the (1, 0) resonance, the (1, 1) resonance and the (1, 0) minimum for the spectra in (c) as a function of the periodicity.

Full size image

Figure 1 (b) shows a back-illuminated microscope image of the fabricated multispectral device taken with a color CCD camera. Each cell displayed a different visible color and intensity compared to the other cells in the block. The color and intensity variations were due to the unique transmission resonances and minima of each NHA. Shown in Figure 1 (c) are the optical transmission spectra (450 nm–900 nm) of each NHA within a single representative block. The spectra were captured with a transmission-based microscope attached to a photometer, a monochromator and a photo-multiplier tube. The microscope setup has been described elsewhere9. Two major optical resonance peaks related to the (1, 0) and (1, 1) SP excitations were observed in the transmission spectrum of each NHA. Also, a (1, 0) minimum related to Wood's anomaly was observed in each transmission spectrum. Transmission resonances were not observed below the bulk plasmon wavelength for gold (500 nm). Strong SP–light interaction was observed at wavelengths greater than 600 nm for all NHAs, which was evidenced by the higher resonance-transmission efficiency. The (1, 0) and (1, 1) transmission resonance and (1, 0) minimum positions versus hole periodicity are shown in Figure 1 (d). As predicted by the dispersion relation between light and SPs (see Ref. 6), the transmission resonance and minimum positions were red-shifted as the periodicity of the holes increased. Also, a linear relationship was observed between the positions of the resonances and minima and the periodicity of the holes. The (1, 0) resonance position was slightly more sensitive to hole periodicity compared to the position of the (1,1) resonance peak and the (1,0) Wood's anomaly.

For the multispectral device, we chose to use SP-matched NHAs (i.e., NHAs with the same refractive index on the top and bottom of the metal film) because they offer higher resonance-transmission efficiency and narrower resonance bandwidth compared to conventional NHAs. The measured (1, 0) resonance bandwidths (FWHM) of the fabricated NHAs were between 85 nm and 96 nm for hole sizes ranging from 120 nm to 148 nm in diameter. The blocking ratio between the (1, 0) transmission resonance and the (1, 0) transmission minimum was between 37:1 and 56:1 for the NHAs, which was greater than the blocking character of conventional NHAs15. The (1,0) resonance-transmission efficiency of each fabricated NHA was between 55% and 62%.

To evaluate the unmixing performance of the multispectral device, the illumination spectrum was divided into nine spectral bands between 644 nm and 804 nm. The spectral bands were selected to minimize the correlations among the transmission spectra of the nine NHAs within the block, as assessed by the characteristics of the (1, 0) and (1, 1) transmission resonances and the (1,0) transmission minima. Minimization of the correlations among measurements has been shown to enhance the accuracy of the color unmixing16. Figure 2 (a) displays the correlation map of the average transmission for the nine spectral bands between 644 nm and 804 nm. We computed the average transmission for each band by integrating the spectrometer measurements within the band. The correlations between pairs of NHAs decreased as the difference in hole periodicity increased. This result was expected because the transmission resonances and minima exhibited better separation when the difference in hole periodicity was large. Given the high correlation between adjacent NHAs as shown in Figure 2 (a), we chose four NHAs with well-separated correlations (i.e., periodicities of 360 nm, 390 nm, 410 nm and 440 nm). The correlation analysis on these four NHAs is shown in Figure 2 (b).

Figure 2
figure 2

The correlation-coefficient map of the transmission spectra representative of NHAs within a single block when their optical transmission was divided into (a) nine spectral bands between 640 and 806 nm for nine NHAs and (b) four spectral bands between 640 and 810 nm for four selected NHAs. Unmixed transmission signal (red bars) for illumination bands (blue bars) of 644–658 nm, 700–710 nm and 770–800 nm in nine unmixed spectral bands between 644 and 806 nm and in four unmixed spectral bands between 644 and 810 nm using (c) nine NHAs and (d) four selected NHAs, respectively.

Full size image

We used a least-squares estimation (LSE) technique to solve the linear mixture model (LMM) for unmixing each spectral pattern of illumination into nine bands between 644 nm and 804 nm. The LSE method is a well-known algorithm for solving general linear spectral-unmixing problems16. The LSE method assesses the abundance value of all fundamental endmembers. The LMM is described using the following matrix equation:

where , and represent a mixed spectrum (response variable), the estimated abundances of the endmembers (regression coefficients) and the random measurement error, respectively. The matrix A contains endmember spectra (regressor variable), where N is the number of measured values and M is the number of endmembers, with the condition that N ≥ M. The estimated abundance of the endmembers was calculated using LSE , where and where represents the norm operation. Prior to unmixing, A was estimated for each block in the device, where each column in A represented the measured transmission intensity for each NHA in the block (1 through N) when presented with a given spectral band (1 through M). In practice, each element of represents the light-transmission intensity measured at each NHA (1 through N) in a block, which represents the mixed spectrum for the block. To resolve the mixed spectrum into spectral bands, we employed the linear least-squares function (lsqnonneg; MATLAB R2012a, The MathWorks, Natick, MA, USA) for estimating the unmixed spectrum or the abundances of the endmembers represented by .

Figure 2 (c) displays unmixed transmission signals for four given illumination bands into nine spectral bands between 644 nm and 804 nm using 9 NHAs. The unmixed transmission signal in each spectral band was close to the intensity at each illumination band. However, due to the high correlation between NHAs, bleeding of signal was observed for the unmixed results (Fig. 2 (c)). Therefore, we then limited the unmixing analysis to 4 NHAs (Figure 2(b)) and observed that 4 NHAs with well-spaced periodicities were capable of accurately separating a test pattern that contained four illumination bands into four unmixed signals (Figure 2 (d)). To test the mosaic-NHA-filter concept for 2D multispectral imaging, we designed a new device that contained 400 blocks, as compared to the original 16-block device. To pack a greater number of blocks into the device and to take into account the findings presented in Figure 2, we chose to include only four NHAs within each block. The device was arranged into a grid of 20 × 20 blocks, where each block contained a 2 × 2 grid of NHAs and each NHA had a unique periodicity (360 nm, 390 nm, 410 nm, or 450 nm). Each NHA was 20 μm × 20 μm in size and was spaced 10 μm from each of its neighbors.

To test the multispectral imaging capabilities of the 400-block device, we designed and constructed an imaging system to capture both reflection and transmission multispectral images, as shown in Figure 3 (a). In transmission mode, white light from a 100 W halogen lamp was delivered by a 3-mm-diameter liquid light guide (Series 2000, Lumatec, Germany) to the image plane of a telecentric lens (0.125× magnification, NT55348, Edmund Optics, NJ, USA). The illumination setup resulted in a collimated beam of light that was approximately 24 mm in diameter. The target was placed in line with the collimated beam. A second telecentric lens (0.3× magnification, NT58428, Edmund Optics, NJ, USA) was aligned with the collimated illumination beam on the opposite side of the target and was used to focus an image of the target onto the front surface of the NHA-based device. An image of the back side of the device was transferred with a 20× objective (NA = 0.45; Nikon 93150) to a CMOS camera (MV1-D1312 (I/IE)-GB, Photon Focus, Lachen, Switzerland). In reflection mode, the liquid light guide from the illumination source was attached to a 5× beam expander placed at a 45-degree angle to the imaging axis on the same side of the target as the camera system. Similar to the transmission system, the reflected light from the target was collected by the second telecentric lens and was projected onto the NHA-based device. The image of the NHA device was focused onto the CMOS camera sensor using the 20× objective. An 800-nm short-pass and a 660-nm long-pass interference filter (Thorlabs, NJ, USA) were used to filter the light from the target prior to it reaching the multispectral device.

Figure 3
figure 3

(a) The setup for 2D snapshot multispectral imaging in either transmission or reflection mode. (b) Schematic diagram of an imaging target used in transmission mode. (c) Images of the target, including raw images of the NHAs (first column on left), unmixed transmission images of distilled water containing various concentrations of MB+ (10 μM, 30 μM, 50 μM) in the 662–714 nm, 705–752 nm, 745–794 nm and 788–832 nm spectral bands (second to fourth columns from left, respectively). For the leaf, unmixed reflection images are shown for the top and bottom surfaces in the 662–714 nm, 705–752 nm and 745–794 nm spectral bands (second to third columns from left, respectively). The right-most column shows the bright-field images of the top and bottom surfaces of the leaf.

Full size image

Each NHA in the multispectral device was optically characterized using the transmission setup to obtain the endmember spectra. Individual illumination bands (≈50 nm) were presented to all NHAs in parallel using a random-access illuminator with a halogen lamp (DeltaRAM V, PTI, Birmingham, NJ, USA). The band-limiting interference filters were removed during NHA characterization. A correlation-coefficient analysis on the end-member spectra indicated that the multispectral device was suitable for separating an unknown input spectrum into four bands between 662 nm and 832 nm.

We chose methylene blue (MB+) as the chromophore for the transmission-imaging tests. A series of MB+ dilutions (0 μM, 10 μM, 30 μM and 50 μM in distilled, deionized water) were contained in quartz cuvettes (QS 1.000, Hellma, Müllheim, Germany), schematically shown in Figure 3 (b). Each cuvette had an optical path length of 10 mm and an interior width of 2 mm and was placed centrally with respect to the illumination. The boundaries of the multispectral device were illuminated by light transmitted through the quartz regions of the cuvette, whereas the central region of the multispectral device was illuminated by light that traversed the MB+ sample. Figure 3 (c) displays images of MB+ samples of increasing concentration (and a water blank) collected with the multispectral device. Figure 3(c) also displays the unmixed transmission image of each sample in each spectral band (662–714 nm, 705–752 nm, 746–794 nm and 788–832 nm). For the image region representative of MB+, the unmixed signal intensity for the 662–714 nm spectral band decreased as the MB+ concentration increased. For the blank sample (distilled water), the transmitted-light intensity was uniform across the quartz cuvette. The light intensity measured in the 788–832 nm band was low because of the 800 nm short-pass filter, which allowed only a portion of the light to reach the multispectral device and the CMOS camera. For all cases in the quartz regions, the light intensity increased from the 662–714 nm band to the 746–794 nm band. This increase can be attributed to the spectral output of the halogen lamp, which increased in intensity in the wavelength range from 662 to 794 nm. Vertical line artifacts in the unmixed spectral transmission images were observed at the interface between the quartz and the lumen of the cuvette.

A sweet cherry leaf (Prunus avium) was selected as the target for the multispectral reflection-imaging experiments. The existence of chlorophyll in the top and bottom surfaces of a fresh leaf is known to result in light absorption at a central wavelength of 690 nm17. However, the chlorophyll content is known to be lower in the midrib, the vein and the bottom surface of the leaf than in the top surface17,18. Based on these differences in chlorophyll content, we imaged both the top and bottom surfaces of the leaf specimen. Figure 3 (c) displays reflection images of the top and bottom surfaces of the leaf taken with (NHA) and without (bright-field) the NHA device. Figure 3 (c) shows the unmixed reflection images for the 662–714 nm, 705–752 nm and 746–794 nm spectral bands. The unmixed reflection images revealed that both the top and bottom surfaces of the leaf had lower reflection (higher absorption) in the 662–714 nm spectral band. However, the top surface of the leaf had a lower reflection in the 662–714 nm band compared to the bottom surface. This result was consistent with the higher chlorophyll content within the palisade mesophyll of the leaf compared to the spongy mesophyll17,18. Consistent with the leaf morphology, the midrib and veins were observed as bright features in the image representing the 662–714 nm spectral band (Figure 3 (c)). The images representing the other spectral bands had uniform and similar brightness, indicative of the higher reflection resulting from the lower light absorption by the chlorophyll at longer wavelengths.

Discussion

The spectral resolution of the NHA-based multispectral device could be improved through the selection of NHAs with optical transmission spectra of lower correlation. A higher resolution would also result for NHAs of a smaller resonance bandwidth and higher blocking (i.e., a larger ratio between the transmission resonance and minimum). Recently, NHAs with a (1,0) resonance bandwidth as narrow as 4 nm (FWHM) have been produced using a lift-off-free fabrication process19. Therefore, the use of an NHA-based multispectral device to unmix spectra with a spectral resolution at or below 4 nm is feasible. Although a small number of blocks was incorporated into the fabricated device, the spatial resolution could be increased through the use of NHAs of a smaller size and/or a reduction in the number of NHAs per block, which would increase the block count for a given sensor area, perhaps at the expense of the spectral range.

The tunability of the EOT of NHAs by manipulation of the device geometry during fabrication is a feature that enables flexible design of NHA-based multispectral devices suitable for visible and NIR imaging. Furthermore, the EOT of NHAs can be tuned via selection of the refractive index of the dielectric material in contact with the metallic film, which provides an additional means to tune the spectral range of the NHA-based multispectral device and could potentially enable electronic tunability using liquid crystals20. Flexible tunability permits the designer to select the spectral sensitivity of the device across one or more application-specific regimes within the visible and near-infrared regions. Such device tunability, combined with the integration of NHAs into a CMOS image sensor, could provide the basis for a compact multispectral imager that utilizes a standard camera lens for 2D snapshot multispectral-imaging applications.

In summary, a 2D multispectral device based on the transmission resonances and minima of NHAs in the visible and near-infrared regimes was developed and tested. The transmission spectra of the NHAs were analyzed using a data-reduction technique that utilized spectral bands. Estimates of the correlations between spectral bands were obtained to assess the performance of a linear least-squares spectral-unmixing algorithm. Spectral unmixing of the device output for inputs of known spectral character successfully recovered the spectral bands in the mixed-input illumination. A 2D multispectral imager was tested in both reflection and transmission modes. The multispectral imager was capable of extracting multispectral transmission images of an MB+-filled cuvette and multispectral reflection images of a leaf. The NHA-based multispectral device could be employed in numerous multispectral imaging applications due to its 2D snapshot imaging capability, miniaturized structure and potential for integration into a variety of imaging sensors.

Methods

Fabrication of SP-matched nanohole arrays

Nanohole arrays in a 100-nm-thick gold film on a Pyrex substrate were fabricated using electron beam lithography (EBL) and a lift-off process. A 3-nm-thick Ti conductive layer was first deposited using electron beam evaporation (EBE) on to a bare Pyrex substrate. Then, a 500-nm-thick photo-resist (Negative Tone photo-resist ma-N 2403) was spin-coated and soft-baked at 90°C for 60 s on top of the 3-nm Ti conductive layer. An EBL machine (LEO, 1530 e-beam lithography) was then used to write the nanohole array pattern on to the photo-resist. The nanohole array patterns were designed using CAD software installed on the EBL machine. Pattern writing of small features was performed with an electron beam voltage of 30 kV and a beam aperture of 10 μm (diameter). The area dose varied from 80 μC/cm2 to 120 μC/cm2 depending on nanohole size. After pattern writing, the sample was developed, which left behind photo-resist pillars. A 3-nm Ti adhesion layer was deposited on top of the pillars (EBE). A 100-nm-thick gold layer was deposited on top of the adhesion layer (EBE). Finally, the photo-resist pillars were lifted off at 80°C with Remover-PG (Micro Chem Inc., Newton, MA, USA) and sonication, which resulted in a nanohole array in the gold film. In order to make a SP-matched nanohole array with cavities, we employed a wet-etching process with Ti TFT etchant (Transene Company Inc., Danvers, MA, USA). The sample was left for 30 s in Ti TFT solution at 25°C. Wet etching resulted in partial removal of the Ti conductive and adhesion layers as well removal of 100-nm of Pyrex beneath each nanohole. Finally, Polymethyl methacrylate (PMMA 950 A2, MicroChem, Newton, MA) was spin-coated at 1000 RPM for 45 s on to the top surface of the nanohole array and soft-baked at 180°C for 3 minutes. The PMMA 950 A2 thickness was approximately 200 nm.

Nanohole array optical characterization

The optical characterization setup consisted of an inverted microscope (Nikon, TE300, Tokyo, Japan) attached to a photometer (PTI, D104, New Jersey, USA), monochromator (PTI, 101, New Jersey, USA), photo-multiplier tube (PMT) housing (PTI, 710, New Jersey, USA) and PMT (Hamamatsu, R928, Shizuoka, Japan). The system was used to acquire optical transmission spectra of nanohole arrays within the range of 400 nm to 900 nm. The nanohole array sample was placed on the microscope stage and illuminated with light focused from the halogen lamp of the microscope using a bright field condenser (NA = 0.3). The diaphragm on top of the bright field condenser was set to 6 mm to restrict illumination to an angular deviation of less than 3° from the normal to the plane of the microscope stage. The light transmitted through the sample was collected by a 20× objective (NA = 0.45). The collected light was guided to the photometer for region of interest selection via adjustment of the photometer apertures and spectral measurement using the monochromator and PMT. Dark noise was subtracted from each transmission intensity measurement. Each dark subtracted measurement was normalized to the intensity of the halogen lamp transmitted through the Pyrex substrate.