Introduction

The interaction of light with surface plasmons (SPs) at an interface between a metal and a dielectric has enabled applications ranging from molecular sensing to cancer therapy1. A metallic structure with sub-wavelength features (e.g. a periodic array of nanoholes) allows light-SP coupling and produces unexpected characteristics such as transmission, reflection and absorption resonances2. Nanohole array (NHA) structures have been shown capable of extraordinary optical transmission (EOT), which has enabled new optical techniques such as sub-diffraction light focusing (super-lensing) and improved upon existing techniques such as particle trapping and Raman spectroscopy1,2,3,4,5,6. The dependence of the EOT on material composition and the geometrical parameters of NHAs has been studied by several groups7,8,9. For example, the transmission resonance position of a NHA can be tuned in the visible and near infrared regions by changing the spacing between the holes or selecting the refractive index of dielectric material on either side of the perforated metallic film7,8. Moreover, transmission efficiency and bandwidth of the EOT can be refined via nanohole size and arrangement8,10. The high degree of tunability coupled with device compactness have motivated several groups to test the usefulness of metallic NHA structures as optical elements for cameras, displays and photography11,12,13,14. For example, NHA structures have been used to create color photos with feature sizes below the optical diffraction limit, a feat not feasible with other optical techniques12. Furthermore, recent developments in the fabrication of NHAs on complementary metal-oxide semiconductor (CMOS) sensors have been tested with the goal of replacing dye-based RGB filters11,14,15,16. However, a demonstration of color imaging with NHA-CMOS devices is still lacking, likely due to wide transmission resonances and poor out-of-band blocking for NHAs fabricated on top of the CMOS sensor. Our group has recently reported on NHA structures with greatly improved EOT properties that enabled snapshot multispectral imaging (MSI) in both the visible and NIR spectral ranges. These enhanced NHA structures incorporated a cavity beneath each nanohole that resulted in SP-energy-matching of the EOTs at the top and bottom surfaces of the metal film. Compared to conventional NHA structures, the SP-energy-matched NHA structures demonstrated superior EOT characteristics such as deeper out-of-band blocking, narrower resonance bandwidth and higher transmission efficiency17,18.

Multispectral videography (MSV) has long been desirable for the capture of spectral information from moving objects and flow in various fields such as remote-sensing and biomedical imaging, but true MSV (i.e. simultaneous 2D image capture of many spectral bands) has been technically challenging due to the spatial or spectral scanning processes inherent to multispectral imaging systems19,20,21. Scanning can result in images with motion artefacts such as blur19. Recent developments in patterned filters have enabled MSI with camera sensors. For example, dichroic filters have been micro-patterned in a color filter array arrangement and shown capable of multispectral imaging in four spectral bands within the NIR region22. The fabrication difficulty of patterned dichroic filters has been a limitation of this technology23. With the goal of developing MSI devices with compactness, easy fabrication and higher image resolution, plasmonic nanostructures have been recently implemented for MSI imaging within visible and NIR spectral regions23,24. As an alternative to other optical filters, our group reported on a MSI system based on a tiled-mosaic arrangement of SP-energy-matched NHAs. This system was shown capable of snapshot multispectral imaging in both transmission and reflection modes for detection bands in the visible and NIR optical regions24.

In this paper, we describe nanofilter-based MSV (nMSV) using a tiled-mosaic arrangement of SP-energy-matched NHAs to capture videos representative of 4 optical bands in the NIR spectral region. The apparatus included a second image sensor that simultaneously captured color images of a scene that were co-registered to the multispectral images. The apparatus proved capable of MSV for analysis of fast processes such as the dispersion of droplets into water in an open cuvette, the flow of indocyanine green (ICG, a common optical contrast agent used in biomedical imaging) through a capillary tube on a microscope stage and the transit of ICG through the vasculature of the ear during a bolus injection in an anesthetized piglet. Furthermore, data-reduction techniques were applied to the time-dependent image data, which resulted in estimates of the heart rate, the distribution of transit times through the microvasculature and positive identification of near surface venous vessels and arterial components.

Results

To construct the nMSV prototype, we first designed and fabricated a NHA-based device that enabled simultaneous multispectral imaging in four spectral bands between 700 nm and 950 nm. Figure 1 (a) displays a schematic of the device consisting of 60 by 60 blocks in a tiled arrangement, where each block contained 4 SP-energy-matched NHAs with 380 nm, 425 nm, 460 nm and 510 nm periodicities in a mosaic arrangement. Each NHA was 20 µm by 20 µm in size and spacing between adjacent NHAs was 10 µm. Our previous work on cross-talk between closely packed NHAs suggested that a 10 µm spacing was adequate to reduce SP-based interaction between NHAs25. NHA structures with SP-energy-matched properties were used due to the higher spectral-band-pass filter performance at the resonance wavelengths and Wood's anomaly compared to non-SP-energy-matched NHA structures7,17. The SP-energy-matched NHA structures were fabricated with electron beam lithography (EBL) in a 70-nm optically thick gold film on a 1-inch BK7 substrate. A Ti wet-etching process was applied to remove the Ti adhesion layer within each nanohole as well as to create a cavity within the BK7 substrate beneath each nanohole18. The NHA structures were spin-coated with a 200-nm thick layer of polymethyl methacrylate (PMMA 950 A2, MicroChem, Newton, MA) that filled the cavities and covered the top surface of the perforated gold film. The PMMA provided a protection layer and a common index of refraction (n = 1.49 at 589 nm wavelength) above and below the gold film that resulted in a SP-energy-matching effect that was highly controlled. The ratio of the aggregate hole area to the film area for each NHA was 0.09 and provided similar (1, 0) resonance transmission efficiency between NHAs within each block. The NHA geometrical parameters were selected in such a way that the (1,0) SP-energy-matched resonance peak occurred at 731 nm, 793 nm, 855 nm and 917 nm for periodicities of 380 nm, 425 nm, 460 nm and 510 nm, respectively. Figure 1 (b) displays the optical transmission of NHAs for 4 spectral illumination bands within 700 nm and 948 nm and a pass band bandwidth (FWHM) of 48 nm. The (1, 0) resonance and (1, 0) minimum and the (1, 0) resonance SP lifetime of the NHA structure were responsible for the improved transmission and out-of-band blocking characteristics9. The average standard deviation of the transmission for 3600 NHAs with the same periodicity was 3% at each spectral band, which demonstrated good consistency among transmission spectra of all NHAs. The correlation coefficient map representative of signal overlap between pairs of NHAs is shown in Figure 1 (c). Most pairs of NHAs produced negative correlation coefficient values, which predicted that the design parameters of the NHAs would, in principle, result in the separation of optical signals into 4 spectral bands between 700 nm and 950 nm, with minimal cross-contamination. Spectral separation of the incoming light into the its component bands was performed with a linear unmixing algorithm based on the non-negative least-squares estimation (LSE) technique24,26. In the LSE technique, the unmixed signal at each spectral band (x (1 through 4)) was estimated using , where , A and represent a mixed spectrum (light transmission intensity of each NHA (1 through 4)), the estimated abundance of the endmembers (estimated light intensity at each spectral band (1 through 4)), the 2D calibration matrix of endmember spectra and the norm operation, respectively26. During the unmixing process, the abundance of endmembers was constrained by . The device was calibrated by imaging the optical transmission through the entire device for each illumination band independently. The 2D calibration matrix was constructed from 4 × 4 elements, where each element was the average transmission measured for NHAs of the same periodicity and calibration illumination band. Column elements of A were arranged by NHA periodicity and row elements were arranged by center wavelength of the illumination band used for calibration (please refer to Figure 1 (b)).

Figure 1
figure 1

(a) Schematic diagram of color filter arrays consisting of 60 by 60 blocks (left). Each block had 4 different NHAs with a periodicity of 380 nm, 425 nm, 460 nm, or 510 nm (middle). A SEM image showing a region of a NHA with 380 nm periodicity (right). (b) Average optical transmission of NHAs when independently illuminated with narrow band (FWFM = 62 nm) light at 731 nm, 793 nm, 855 nm and 917 nm. c) Map of correlation coefficients between optical transmission measurements for data in panel (b).

Full size image

Based on the NHA device, we constructed an apparatus for performing nMSV at 4 spectral bands between 700 nm and 950 nm. Figure 2 (a) illustrates the apparatus, which was comprised of the NHA device, 2 cameras, a dichroic mirror, 2 folding mirrors, blocking filters and a series of relay lenses. The first relay lens, L1 (f.l. = 100 mm; broadband) and L2 (f.l. = 50 mm; AC254-50-A; Thorlabs), transferred a visible color image from the input focal plane of the apparatus onto a color CMOS sensor (Logitech Webcam Pro 9000, Apples, Switzerland). The input focal plane of the apparatus was coincident with the image plane of the input objective (i.e. telecentric lens). The second relay lens, L1 and L3 (f.l. = 50 mm; AC254-50-B; Thorlabs), relayed a NIR image of the input image plane to the NHA device. A dichroic beam splitter DM (FF705-Di01-25x36, Semrock Inc., New York, USA) was placed between L1 and the other two lenses (L2 and L3) to reflect the visible components toward the CMOS color camera, as well as to transmit the NIR light toward the NHA device. An image of the NHA device was collected by the third pair of relay lenses, L4 (f.l. = 50 mm; AC254-50-B; Thorlabs) and L5 (f.l. = 100 mm; AC254-100-B; Thorlabs), which then was focused onto the monochromatic CMOS sensor (MSV1-D1312 (I/IE)-GB; 8 bit dynamic range; Photon Focus, Lachen, Switzerland). A 950-nm short-pass and a 700-nm long-pass interference filter (F1: FEL 700 and F2: FES 950, Thorlabs) were located between L4 and L5 to pass light within the 700 nm to 950 nm band to the monochromatic CMOS sensor. Two gold mirrors M1 and M2 (PF10-03-M01, Thorlabs) were used to fold the beam to reduce the length of the apparatus.

Figure 2
figure 2

a) Multispectral videography apparatus consisting of achromatic visible and NIR lenses (L1, L2, L3, L4 and L5), a dichroic beam splitter (DM: reflective < 700 nm), gold mirrors (M1 and M2), a 950 nm short-pass interference filter (F1), a 700 nm long-pass interference filter (F2), a color camera (C1), a monochrome camera (C2) and a tiled-mosaic arrangement of NHA arrays (NHA device). The illumination consisted of light transferred from a halogen lamp by a liquid light guide to a beam expander. The imaging target was a 10-mm optical path quartz cuvette containing 3 mL DI-H2O. Drops of dye (~15 µL per drop) were released into the DI-H2O and the dispersal process was imaged. b) Spectral absorption of 500 mM copper sulfite and 100 µM and 200 µM methylene blue (top panel); spectral absorption of ICG (20 µM, 50 µM and 100 µM; bottom panel). c) Select frames of a multispectral video taken from three experiments. Each column of frames represents a single exposure at a specific time point (lower scale) and includes the color camera image (Color), the image of the NHA device (NIR) and unmixed images for each detection band (700–762 nm, 762–824 nm, 824–886 nm and 886–948 nm). Experiments were performed with 1 M copper sulfite drops (left), 1 mM MB+ drops (middle) and 100 µM ICG drops (right) (See the supplementary videos for 1 M copper sulfite, 1 mM methylene blue and 100 μM ICG drops).

Full size image

In the first study, trans-illumination mode nMSV performance was tested by dye dispersal measurements. In the experimental setup, collimated white light from a 100 W halogen lamp was transferred by a liquid light guide to a beam expander to illuminate a 10 mm optical path quartz cuvette (3ES10, NSG Precision Cells Inc., NY, USA) filled with 3 mL of distilled-deionized water (DI-H2O). A C-mount telecentric lens (1× magnification; 55350, Edmond Optics, NJ, USA) was mounted on the apparatus to image the dispersion of drops of dye as they were released into the DI-H2O. Figure 2 (c) displays 3 sets of 5 frames extracted from multispectral videos of 9 drops of 1 M copper sulfite, 5 drops of 1 mM Methylene Blue (MB+) and 8 drops of 100 µM indocyanine green (ICG). To verify multispectral results, absorption spectra of 500 mM copper sulfite, 80 µM MB+ and 40 µM ICG were acquired with a spectrophotometer (DU 640, Beckman Coulter, Inc., Brea, CA, USA) and shown in Figure 2 (b). The absorption spectra for each dye was captured with a quartz cuvette with a 4-mm optical path and blanked to DI-H2O. Each drop had a volume of approximately 15 µL. Each frame consisted of a RGB color image, an image of the NHA device and 4 unmixed images of the NHA device representative of four spectral bands between 700 nm and 948 nm. The multispectral images were normalized to images of DI-H2O lacking dye to obtain estimates of maximal optical transmission. Multispectral videos of 1M copper sulfite drops revealed rapid dispersal into the DI-H2O and homogeneous absorption across all 4 bands within 67.5 s of the last drop (Figure 2 (c); left). The last multispectral frame contained transmission contrast due to copper sulfite even though there was no significant contrast observable in the corresponding color camera frame. However, due to the difference in refractive index between copper sulfite (n = 1.514) and DI-H2O (n = 1.33), dispersion of copper sulfite drops into the DI-H2O were apparent in the intermediate color frames. In the case of 1 mM MB+ (Figure 2 (c); middle), drops of MB+ tended to disperse slowly and were most evident in the frames representative of the 700–762 nm band. Contrast due to MB+ was also observed in the corresponding color camera frames. The results were expected since MB+ has a broad absorption peak at 660 nm (within range of the color sensor), which extends into the 700–762 nm and 762–824 nm detection bands (within range of the NHA device) as shown in Figure 2 (b). Small dye-dependent signals were found in the 886–948 nm bands, likely due to the high concentration of MB+ in the cuvette. At high concentrations, optical absorption was no longer linearly related to concentration and gave rise to an analysis artefact that appeared as signal contamination across unmixed detection bands27. In Figure 2 (c) (right), frames representative of drops of 100 µM ICG are shown as they disperse into the DI-H2O. Transmission contrast indicative of the dispersal process was observed in the images representative of the 700–762 nm and 762–824 nm unmixed transmission bands. Optical transmission appeared to be highest in the frames representative of the 762–824 nm band and lower in the 700–762 nm band; this was consistent with the absorption properties of ICG at concentrations less than 50 µM, which has a primary absorption peak at 780 nm and a smaller secondary absorption peak at 705 nm. Similar to the MB+ multispectral videos, a small degree of optical transmission contrast was observed for bands outside of the primary absorption bands of ICG at high concentrations. It is anticipated that the leakage artefact common to regions of high concentration can be reduced with a camera of higher dynamic range capable of measuring the small changes in optical transmission for incremental changes in dye concentration when the concentration is high.

The nMSV apparatus was capable of resolving optical absorption of various dyes into four spectral bands in the NIR simultaneously. The nMSV frames were of sufficiently high resolution to track the absorption features related to the convective movement of dye as it was dispersed into the DI-H2O. The nMSV frames were qualitatively similar to the corresponding higher resolution frames acquired with the color camera. The nMSV apparatus was capable of both tracking dye as it diffused into the DI-H2O and clearly separating absorption due to dye into the expected spectral bands. Unlike other MSI systems based on spatial and spectral scanning processes, nMSV with patterned 3D nanostructures was insensitive to smearing and motion artefacts due to the simultaneous capture of the 2D video frames at each spectral band.

In a second study, the nMSV apparatus was attached to an inverted microscope (TE 300, Nikon, Tokyo, Japan) through a C-mount to F-mount adapter, which enabled MSV of objects located within the focal plane of the microscope objective (Figure 3 (a)). The setup was used to quantify the dynamic range of the NHA device with respect to dye concentration in a capillary tube (10 µL; VWR Scientific, PA, USA). A solution of DI-H2O and ICG was pumped through the capillary tube to dynamically change the ICG concentration in the range of 0 to 1000 µM. Multispectral videos of the flow were captured at 8 frames per second. To facilitate dynamic changes in ICG concentration, the capillary tube was attached to two independent syringe pumps (22, Harvard Apparatus, Inc., MA, USA) with one pump holding a syringe loaded with DI-H2O and another pump holding a syringe loaded with 1000 µM ICG (in DI-H2O). The total flow rate for both pumps was maintained at 50 µL/min throughout each experiment. Initially, the flow of DI-H20 was set to 50 µL/min with zero flow rate on the ICG pump. Every 100 s, the flow rate on the ICG pump was increased by 10 µL/min and the flow rate on the DI-H2O pump was reduced by 10 µL/min simultaneously. The stepping of the flow continued until the complete multispectral video of the capillary tube for the full range of ICG concentrations (i.e. 0–1000 µM) was obtained. Figure 3 (b) displays multispectral video frames taken at select time points representative of each ICG concentration. Consistent with the ICG absorption spectrum, unmixed images revealed optical absorption within the 700–762 nm and 765–824 nm bands, but not in the longer wavelength bands. Furthermore, as expected, transmission in each of the first two bands declined as the concentration of ICG increased. Optical transmission differed across the diameter of the capillary tube, which was consistent with the range of optical path lengths related to the circular cross-section of the capillary tube and refraction effects due to the curved liquid-glass-air interfaces. For example, optical transmission measured near the wall of the capillary tube (short optical path length) was higher than the optical transmission measured near the center of the capillary tube (longer optical path length) for a given ICG concentration. A region of interest analysis of the multispectral video is shown in Figure 3 (c) where the absorbance for each of the 4 bands is plotted against time. The absorbance for a region of interest was calculated based on a logarithmic ratio between the average pixel intensity within each region of each video frame over the similarly acquired average pixel intensity capture with water alone. The plot clearly shows the increase in absorbance for the first 2 bands (700–762 nm and 765–824 nm) and almost no change for the last 2 bands. Based on the signal changes, nMSV was able to capture the mixing kinetics as a change in ICG concentration that approached a new asymptotic value after the flow rate on each pump was adjusted. The lack of optical transmission contrast within the last two detection bands was expected due to the lack of ICG absorption in these bands. Based on the time series data in Figure 3 (c), the dependence of absorbance on ICG concentration was analyzed (Figure 3 (d)). Absorbance in each of the first two bands increased linearly as ICG concentration increased. The standard deviation of the absorbance measured for the first two bands ranged from ± 0.046 to ± 1.148 for the lowest to highest ICG concentrations, respectively. The range in measured error was related to the range of absorbance measurements across the diameter of the capillary tube due to optical path length variations and refraction effects (see above). However, the standard deviation was lower for the last two bands (±0.012 and ±0.022, respectively) due to a lack of ICG absorption.

Figure 3
figure 3

(a) Multispectral videography apparatus attached to an inverted microscope using C-mount to F-mount adapter. Multispectral videos were collected of a capillary tube on the microscope stage, which was filled with a mixture of DI-H2O and ICG. The concentration of ICG within the capillary tube was controlled by the ratio of the flow rate of 1000 µM ICG to the flow rate of DI-H20 through two independent syringe pumps (not shown) connected to the capillary tube through inlets. Mixing occurred at the inlet of the capillary tube. The illumination source was the halogen lamp onboard the microscope. White light was focused on to the capillary tube using the condenser of the microscope. (b) Select frames of a multispectral video showing images representative of the color camera, the NHA device and 4 unmixed detection bands (700–762 nm, 762–824 nm, 824–886 nm and 886–948 nm) for specific ICG concentrations (0 µM, 200 µM, 400 µM, 600 µM, 800 µM and 1000 µM) (See the supplementary video for various ICG concentrations flowing through the capillary tube). (c) Absorbance for a region of interest (represented for one band by red dashed box in panel (b)) for each detection band as a function of time. d) Absorbance (±1 SD) for a region of interest (red dashed box in panel (b)) as a function of ICG concentration (0 µM, 200 µM, 400 µM, 600 µM, 800 µM and 1000 µM) measured at time points indicated by the vertical green dashed lines in panel (c).

Full size image

In a third study, nMSV was used during an animal imaging experiment to capture the transit of a bolus injection of ICG through the microvasculature of a new born Duroc piglet (< 48 h old). The piglet was anesthesized (induction with 3–4% isoflurane), tracheotomized and mechanically ventilated during the experiment. A catheter was inserted into the left ear vein to facilitate ICG injection (1 mL, 2 mM). A cannula was inserted into a femoral artery for continuous monitoring of blood pressure. The arterial ICG concentration was measured with a pulse dye densitometer (PDD) (DDG 2001 A/K, Nihon Kohden, Tokyo, Japan) attached to a hind foot. During the imaging experiment, the average blood pressure, heart rate, respiratory rate and body temperature of the piglet were 55 mmHg, 165 bpm, 55 breaths per minute and 39.3°C, respectively. All animal experiments were approved by the Animal Use Subcommittee of the Canadian Council on Animal Care at the University of Western Ontario (protocol # 2011-063).

A multispectral video of the top surface of the right ear of the piglet was captured at 11.54 fps. The underside of the ear opposite the imaging field was illuminated with a 100 W halogen lamp. The images were collected with a telecentric lens (1× magnification; 55350, Edmond Optics, NJ, USA) attached to the multispectral videography apparatus. A vessel with a nominal diameter of 200 µm was visible across the field of view. After the ICG bolus into the left ear, the bolus travelled through the systemic circulation prior to reaching the microvasculature of the right ear. In this senario, the multispectral video potentially contained contrast representative of variations in ICG signal due to the heart, analogous to pulse dye densitometry. Figure 4 (a) displays selected frames of a multispectral video of the ear before, during and after the bolus injection of ICG. Inspection of the multispectral video frames relative to the start of the experiment revealed that ICG contrast (700–762 nm and 765–824 nm) appeared within 8 ± 2 s and reached a maximum value at the 13 ± 2 s. For superficial microvasculature (i.e. visible in the color camera images), the appearance and decay of ICG contrast due to the bolus was clearly visible and appeared coregistered with the superficial microvasculature morphology. Also, the overall brightness in the first two bands decreased due to the transit of ICG through deeper microvasculature, which were not spatially resolvable due to the scattering properties of tissue. In the last two bands (824–886 nm and 886–948 nm), no presence of ICG contrast was observed; however, near surface microvasculature were observed with low contrast (see Figure 4 (a)).

Figure 4
figure 4

Multispectral videography of the microvasculature of the outer surface of a piglet ear during a bolus injection of ICG into a vein in the opposite ear.

Multispectral videos were collected with a telecentric lens attached to the nMSV apparatus and transillumination of the ear with a halogen light source. (a) Multispectral video frames comprised of the color camera image (Color), image of the NHA device (NIR) and unmixed detection bands (700–762 nm, 762–824 nm, 824–886 nm and 886–948 nm) selected to reveal major changes due to a bolus injection of 2 mM ICG (See the supplementary video for ICG flowing through the piglet's ear). (b) Region of interest analysis of entire multispectral video showing the average pixel intensities in the region of interest defined by the blue and red dashed rectangle (inset) within the second detection band (762–824 nm) over time (blue and red). A PDD was used to monitor ICG concentration independently (green). (c) Magnitude Fourier transform of a time series where each time point represents the average pixel intensity in the 762–824 nm detection band for a given frame. (d) Maps for 700–762 nm and 762–824 nm spectral bands showing transmitted light intensities at the time of the maximum presence of ICG for each pixel in the respective detection band images. (e) Maps for time to maximum presence of ICG for pixels in 700–762 nm and 762–824 nm bands. (f) Histogram of time to maximum presence of ICG in 700–762 nm (top panel) and 762–824 nm (bottom panel). (g) Maps for 700–762 nm and 762–824 nm bands showing the correlation between the PDD signal and each pixel in the detection band images. (h) Relative correlation lag for pixels in bands 700–762 nm and 762–824 nm relative to the PDD signal. (i) Histogram of correlation lags between PDD signal and each pixel in 700–762 nm (top panel) and 762–824 nm (bottom panel) bands.

Full size image

A region of interest analysis repeated on consecutive frames of the images representative of the second detection band are shown alongside the corresponding PDD readings (Figure 4 (b)). The ICG estimates obtained from the PDD device revealed an earlier arrival of ICG contrast (≈4.2 seconds lag) compared to the arrival of ICG in the vasculature on the ear. This difference can be explained by the shorter arrival time of ICG to the arterioles of the hind foot where the PDD probe was placed, compared to the arrival of the ICG bolus to the venous vasculature in the ear. The signal from the second band representative of ICG contrast was temporally wider than the PDD signal due to the dispersion of the contrast agent as it passed through the capillary bed of the ear28.

Low amplitude oscillations were apparent in the ICG time course and investigated using the second band images from the nMSV. The magnitude Fourier transform of ICG time course is shown in Figure 4 (c). Frequencies below 1 Hz were present and likely related to respiratory motion. Also, two distinct peaks were observed at 2.75 Hz and 5.5 Hz, which corresponded to the heart rate (measured independently at 165 bpm) and the second harmonic of the heart rate. The appearance of frequencies representative of the heart rate can only be explained by sensitivity of the nMSV to changes in optical pathlength caused by the pulsation of arterial blood.

Using the multispectral video, the transmitted light intensity at the ICG peak and the time to ICG peak were computed for each pixel in the 700–762 nm and 762–824 nm spectral band images; this is shown in Figures 4 (d) and 4 (e), respectively. In Figure 4(d), the signal intensity at each pixel was lower for the map represenative of the second band compared to the map of the first band due to higher absorption of ICG dye in the second band. The surface-weighted patterns of microvasculature were observed in both maps as features with lower signal intensity compared to surrounding regions. However, other features were present in the images and were likely representative of the surface irregularities of the skin. The maps of time to ICG peak clearly revealed the surface-weighted microvasculature; however, the extent of the visualized microvasculature was greater in the second band due to the higher ICG absorption and greater signal contrast (Figure 4 (e)). Based on the time to peak in the second band, the ICG contrast signal appeared later in time for the regions representative of microvasculature compared to surrounding regions, which suggested that the microvasculature features are on the venous side of the circulation. Furthermore, one can infer that the darker areas are representative of the arterial side of the circulation since the correlation lag appeared earlier in time. However, due to the scattering properties of tissue and the likelihood that arteries and arterioles are located deeper than veins, morphological features typical of vessels, other than the surface-weighted venous vessels, were not obvious in the images. Although both first and second band maps were similar in terms of the distribution of time to peak values, the second band image revealed a later time to peak compared to the first band image. The later arrival can be understood by interpreting the results within the context of the cumulative effects of the wavelength-dependent scatter in tissue and the absorption of ICG. Compared to the second spectral band, the frames related to the first spectral band were expected to have lower optical absorption due to ICG, but higher optical scattering. Therefore, in transillumination, photons detected in the first band were more likley to have propagated along longer paths compared to photons detected in the second band and hence represent optical properties for deeper tissues where the arterial vasculature is denser and ICG appears early. In contrast, the signals present in the second band are likely to be representative of the superficial venous vasulature where ICG is expected to arrive later. This effect is consistent with earlier work (e.g. see Ref. 29) that demonstrated a dependence of the arrival time of ICG on detection wavelength. Our interpretation is further supported by data presented in Figure 4 (f), which displays the histograms of the time to ICG peak measurements for the first two spectral bands. Analysis of the histograms revealed a 0.5 s time difference between the mean time to peak values of the first band and the second band. We hypothesized that the temporal shift between the two histograms was representative of measurements that were arterial-weighted in the first band and venous-weighted in the second band. In principle, the time to ICG peak extracted from each image pixel represented a temporal curve weighted by contributions from the arterial (early) and venous (late) vasculature. To test this hypothesis, maps of the correlation coefficient and correlation lag between the time-dependent PDD signal and the unmixed time series representative of each pixel were calculated (Figures 4 (g) and (h)). Similar to the time to ICG peak maps, the correlation coefficient maps exposed contrast features with low correlation values that were consistent with the superficial microvasculature observed in the color images. Furthermore, the maps contained diffuse areas of high correlation, which were likely representative of arteries and arterioles below the surface, since the PDD signal represents the arterial component. In addition, maps of the correlation lag (see Figure 4 (h)) and histograms of the correlation lag (see Figure 4 (i)) were similar to the maps (Figure 4 (e)) and histograms (Figure 4 (f)) extracted directly from the multispectral videos without prior knowledge of the PDD signal.

Furthermore, we compared the temporal signals from early, middle and late arrival of the time to ICG peak with the captured PDD signal from the hind foot of the piglet. In the analysis, motion artefacts were reduced greatly by dividing data from the first band by data from the second band on a pixel-by-pixel basis. Figure 5 displays the average ratio of pixel intensity for the first band to the second band for image region areas where early, intermediate and late time to ICG peak occurred. The PDD signal representative of the arterial component is shown for comparison. The signal from the region representative of early arrival had the highest correlation with the signal from the PDD. The early arrival signal had a narrower circulation bandwidth and had both dye recirculation trends in the signal. Therefore, the early arrival region was weighted toward the arterial component. The signal representative of the intermediate time to ICG peak region was broadened compared to the signal related to the early arrival region, but narrower than the region representative of the late arrival. Therefore, the signals related to regions of intermediate and late arrival of the ICG peak were weighted toward the venous component to a larger extent than in the early arrival region.

Figure 5
figure 5

Average signals from the first band (B1) over the second band (B2) for early (red), intermediate (blue) and late (black) arrival of the ICG peak.

The blue and black curves were offset relative to the red curve for clarity. The green curve represents the PDD signal.

Full size image

Discussion

It is evident from the piglet results that the nMSV apparatus provided a means to perform bolus tracking of an intravasculature dye within four spectral bands simultaneously. The high temporal sampling combined with simultaneous spectral measurements enabled the extraction of various temporal and spectral metrics attributed to the transit of dye, which permitted a distinction between regions based on relative weighting of the arterial and venous components. The high temporal sampling also enabled measurement of the heart rate. Moreover, spectral images facilitated the significant removal of motion artefacts to improve visualization of the bolus first-pass and the subtle dynamics of bolus recirculation.

In future, the nMSV apparatus could be employed for localizing abnormal tissue during a surgical process when ICG is injected into the body (as described in Ref. 30). The captured images from the nMSV system could enhance detectability of the abnormal tissue as well as monitoring different physiological parameters such as blood flow and heart rate. Also, the nMSV apparatus can be used to monitor oxygenated and deoxygenated blood cells in vein and blood vessels. Therefore, one can speculate that the nMSV apparatus can be applied to the detection of cancerous skin lesions due to the difference in oxygenated and deoxygenated blood concentrations in abnormal and normal skin lesions (as described in Ref. 31).

We have presented a nanofilter-based multispectral videography system capable of unmixing a complex scene into images at 4 detection bands within the range of 700 nm to 950 nm. The nMSV system utilized a tiled arrangement of identical blocks, where each block consisted of a mosaic of 4 SP-energy-matched NHAs. Each NHA in the block was fabricated with a unique periodicity selected to control its optical pass band. Multispectral videography was capable of recording the dispersal of droplets of dyes (MB+, CuSO4 and ICG) into water with specificity for the absorption properties for each dye. Furthermore, nMSV was able to capture the dynamic flow-induced changes in dye (ICG) concentration within a capillary tube. Finally, nMSV was used to track a bolus injection of dye (ICG) within the microvasculature of the ear of an anesthetized piglet. The multispectral videos were rich in features and video-based analysis provided physiological parameters such as average heart rate and venous-weighted flow dynamics of the microvasculature. Analysis of the multispectral videos in combination with supplemental PDD measurements resulted in estimates of the distribution of transit times through the microvasculature and separation of the venous and arterial components. This technique holds promise for dynamic physiological monitoring of blood flow in tissues and may have application to skin cancer detection and identification.