Introduction

With increased focus on high-speed biological processes at molecular, chemical, and biophysical levels, there has come a critical need to develop tools that can keep pace with the number of active research areas in biomedicine. Technical developments in nonlinear optical microscopy have revolutionized our ability to study biophysical and biochemical properties of tissues, both in terms of the wide range of samples, improved contrast modalities, and unprecedented speeds at which imaging is now possible. However, combining direct spatial temperature measurements alongside multiphoton modes of contrast have not been widely adopted. Multiphoton imaging approaches have enabled deep-tissue imaging across numerous model systems, gleaning important insights on the structure and function of cells and tissues1,2 while thermal imaging can yield complementary information to relate to study metabolism, circulation, and immune response. Temperature plays a crucial role in biology yet has yet to be extensively demonstrated in any substantial capacity towards biological microscopy applications. Thus, instrumentation to explore the structural, functional, and thermal properties of tissue would prove useful in studying biophysical dynamics in tissue physiology.

Multiphoton fluorescence (MPF) and second harmonic generation (SHG), are particularly useful in highlighting tissue structure and function3,4. Calcium, voltage, and molecule-sensitive reporters continue to push the boundaries of multiphoton applications in biomedical research. Vibrational spectroscopic contrast using nonlinear Raman imaging (NRI), namely coherent anti-Stokes Raman scattering (CARS) and stimulated Raman scattering (SRS), has shown particularly wide-ranging applicability. These include rapid identification of tumor margins5, high-dimensional structural labeling6, other biological processes such as drug delivery7, neurotransmitter release8, and electrophysiological dynamics9. Such imaging can highlight the molecular organization and biochemical composition of tissue over time-offering a glimpse into sub- and inter-molecular dynamics without using exogeneous labels. Multiplexing functional, biochemical, and biophysical (e.g. temperature) contrast in cells and tissues can offer a unique tool in evaluating and imaging biological and physiological processes that may not be possible with the same degree of exploratory power in separate instruments1.

Nonlinear optical microscopy is inherently suitable for multimodal imaging applications. Integrating the different modalities that fall under nonlinear microscopy can help alleviate unnecessary system complexity while providing complementary information about tissue structure and function. Notably, CARS, SRS, MPF, and SHG are all optical processes that can occur and be detected simultaneously or sequentially by selecting the appropriate filters and detectors for the techniques used. Several groups have demonstrated selective combinations of nonlinear imaging strategies to address a range of biological questions, in vitro and in vivo 10, particularly sensing fast biological processes such as neuronal action potentials and calcium activity with combined SRS and calcium-sensitive MPF9. Integration of SRS with optical coherence tomography (OCT) has been shown to augment nonspecific scattering-based contrast with vibrational specificity to image lipid distributions in excised human adipose tissue11. More recently, researchers have employed four or more nonlinear imaging modalities, including CARS, MPF, SHG, and third harmonic generation for live tissue and intravital imaging towards wound healing and cancer metastasis12,13. While most of these demonstrated approaches have been applied towards observing biological processes over the span of multiple hours at the molecular and biochemical levels, they are limited in their ability to observe snapshots of sub-second functional, biochemical, and thermal processes.

The influence of temperature in physiological and biochemical processes—particularly surrounding sample damage and physiological modulation—has recently become an area of prominence across biomedical research disciplines14,15,16. As optical imaging and perturbation technologies continue to advance biomedical research, it becomes critical to consider thermodynamic effects particularly on live specimens albeit practically difficult. A flexible imaging system that incorporates molecular, biochemical, and biophysical information from cells to tissues, in vitro to in vivo, would enable the study of rapid and dynamic biophysical processes from multiple perspectives.

Measuring temperatures in biological samples at high resolutions spatially and temporally has proven difficult and continues to be an active area of research. Groups have multiplexed blackbody thermal imaging and fluorescence microscopy at cellular resolutions in vitro 17,18, however water absorption generally obscures the ability to visualize cell morphology with thermal imaging alone. Moreover, optical components that suffice for use between 0.4 and 14 µm wavelengths to encompass visible optical and thermal infrared imaging are not readily available. Finite element heat transfer modeling and Monte Carlo simulations of photon transport in scattering media is often regarded as the benchmark approach for temperature estimation of dynamic photothermal processes in biological samples with high water concentrations19. Several indirect approaches have been demonstrated including temperature-dependent changes in fluorophore emission, and probe beam deflection microscopy16,19,20. While point-based methods (utilizing thermocouples, customized miniature sensors, or intrinsic fluorescence of rare-earth doped glass waveguides) are more direct, they sacrifice spatial information obtained with temperature mapping21. Previous work has utilized thermal imaging with laser speckle imaging to study cerebral blood flow changes during different methods of anesthesia22. However, despite the potential utility of combining thermal imaging with multimodal nonlinear microscopy, such a combination has not been previously reported. A system that can combine multimodal nonlinear microscopy with thermal imaging would offer unprecedented flexibility to study biological processes in real time.

The goal of this project is to integrate multiple nonlinear imaging methods with thermal imaging so that biophysical, biochemical, and molecular information from dynamic biological processes may be imaged with micron level spatial resolution and millisecond level temporal resolution. Towards this effort, we present an imaging platform that integrates single-band CARS, SRS, MPF, SHG, and wide field thermal microscopy (ThM) for characterizing tissues at varying time scales. Applications of our imaging system are demonstrated imaging both in vitro with neural cell cultures and tissue specimen as well as in vivo with acute craniotomy rat preparations. Our platform offers a novel approach enabling imaging techniques with otherwise incompatible optical paths (due to physical limitations of hardware) to be applied on a common sample. We demonstrate that this platform provides a robust tool for researchers to probe the physiology of dynamic biological systems, through the unique integration of vibrational spectroscopic, functional fluorescence, and thermal contrast.

System design

The framework of the Multimodal Advanced Nonlinear and Thermal Imaging System, or MANTIS, is a three-armed imaging turret (Customized Bergamo II Platform, Thorlabs Imaging Research, Sterling, VA, USA) that respectively images a sample with either nonlinear imaging, thermal imaging, or widefield white light reflectance imaging which is reconfigurable to adapt additional measurement approaches (Fig. 1A). The mechanical arrangement of distinct imaging arms allows for modalities with incompatible optical instrumentation (e.g. ultrafast near infrared imaging with endogenous shortwave infrared measurements) to be performed sequentially without disturbing or repositioning the sample. A FLIR SC8300-series high-speed indium-antimonide CCD camera equipped with a 4X germanium imaging objective (FLIR Systems Inc., Nashua, NH, USA| Fig. 1B) is attached to the thermal imaging arm of the microscope. The thermal camera relies on the endogenous black/grey body emission of a sample between 3 and 5 µm wavelength under the assumption of homogenous emissivity to infer sample temperatures.

Figure 1
figure 1

MANTIS system design. (A) The layout of MANTIS. The system is capable of performing multimodal nonlinear and thermal imaging over the same sample. Additionally, white light imaging on a third imaging arm is available for further expansion of the system. (B) Schematic of wide-field thermal microscopy, which infers the temperature of a sample based on the blackbody emission observed between wavelengths of 3 and 5 µm. (C) General optical schematic of a multimodal nonlinear imaging system capable of integrating CARS, SRS, MPF and SHG imaging. DM—dichroic mirror, BP—bandpass filter, PMT—photomultiplier tube, EOM—electro-optical modulator, InSb CCD—indium antimonide charge coupled device.

Full size image

The nonlinear imaging section of MANTIS, depicted in Fig. 1C with a simplified schematic, is coupled to two nonlinear laser sources supplying femtosecond (Insight DS + , Spectra Physics, Fremont, CA, USA) and picosecond (picoEmerald S, A.P.E, Berlin, DE) laser lines. Higher spectral bandwidth (~ 15 nm) femtosecond laser pulses provides high peak powers necessary for optimal multiphoton and higher harmonic generation in vivo 1. The picosecond laser has narrower bandwidth (~ 0.5 nm, or 10 cm-1), which is critical for maintaining the spectral resolution necessary for single-band nonlinear Raman imaging while minimizing power at the sample for safe in vivo applications (< 20 mW average power)23. Both laser systems operate with a pulse repetition rate of 80 MHz and output two beams necessary for pump and Stokes excitation of NRI contrast processes. The shorter wavelength tunable laser line undergoes a tunable path length delay while the longer wavelength laser line is intensity-modulated at 20 MHz via an electro-optical modulator (Thorlabs, Newton, NJ, USA) to facilitate SRS. Each laser’s output is spatially and temporally co-linear and directed with a series of mirrors into the scan head of the nonlinear imaging arm of MANTIS.

The imaging optics of the nonlinear imaging arm are based on conventional upright laser-scanning microscopy. The scanning optics consist of a pair of galvanometric mirrors (Thorlabs, Newton, NJ, USA), which are imaged onto the back focal plane of a commercial objective lens (Olympus XLUMPLFLN, 20X 1.0NA | Nikon CFI Apochromat NIR 60X, 1.0NA) via a 4f optical relay. This scan relay performs a fixed 4X magnification of the laser beam diameter to accommodate the back-aperture pupil size of the largest objective lenses we use relative to the entrance beam diameter (SL50-2P2 & TL200-2P2, Thorlabs, Newton, NJ, USA). Two epi-detection ports with a photomultiplier tube (CARS/MPF/SHG | GaAsP Amplified PMT, Thorlabs, USA) and a large area 50-V reverse-biased photodiode (SRS | A.P.E. GmbH, Berlin, DE) are used for imaging epi-detected contrast. Two additional detachable forward detection ports were also built to accommodate coherent imaging modalities in transparent samples in transmission mode. All reported SRS images are acquired in transmission mode with the exception of in vivo imaging data. Stimulated Raman loss for SRS was demodulated via a commercial lock-in-amplifier (A.P.E. Gmbh, Berlin, DE). All emission filters and excitation wavelengths included with the system are in Table S1 (Semrock, Brattleboro, VT, USA). Scanning and detection hardware for imaging is controlled through ThorImageLS version 2.1, 3.0, or 3.2 (Thorlabs Imaging Research, Sterling, VA, USA). All images shown are raw with linear intensity rescaling, with all analysis being performed in FIJI24.

The third imaging arm of MANTIS initially features a color complementary metal oxide semiconductor (CMOS) camera (Thorlabs Inc., Newton, NJ, USA) with a variable focal length lens (Navitar Inc., Rochester, NY, USA) for widefield white-light reflectance imaging of samples (Fig. 1A). However, this arm was built into the imaging system to readily enable integration of additional contrast modalities that may be further incompatible with multimodal nonlinear microscopy—such as interferometric contrast or stereoscopic surgical guidance. Each arm is locked into position by a custom-designed spring-loaded locking system against the rotating turret base (Fig. 1A). The maximum fields of view of each imaging arm is ~ 800 µm × 800 µm for nonlinear microscopy with a 20X objective, 3 mm × 4 mm for thermal microscopy with a 4X objective, and 50 mm × 50 mm for widefield white light reflectance imaging.

System performance

To achieve high-speed imaging at subcellular resolution, MANTIS was designed to resolve at least 1 µm lateral resolution in the nonlinear imaging arm. Performance was verified on a series of standards, (Fig. 2A), with measured axial and lateral resolutions for each modality summarized in Table S2 (with full data summary in Table S3). Images of 0.5 µm diameter polystyrene (PS, latex, Polysciences Inc. Warrington, PA, USA) beads with CARS and SRS were obtained to generate a point spread function (PSF), which is reported here as the full-width half-maximum intensity cross-section of beads fit to a Gaussian curve. Fluorescent bead calibration standards with a 250 nm diameter (Polysciences Inc. Warrington, PA, USA) were used to measure the PSF of MPF. All PSF calculations were performed using the MetroloJ plugin in FIJI25. Measured resolutions, reported in Table S2, are on par with results published in literature when imaging with numerical apertures approaching 1.03,23,26. Spectral separation of polymethyl-methacrylate (PMMA, acrylic, Polysciences Inc. Warrington, PA, USA)) and PS beads with single-band SRS are shown in Fig. 2B. Acrylic (PMMA) beads are depicted using the 2927 cm-1 band—an asymmetric CH3 resonance—(yellow), and latex (PS) beads are depicted spectrally separate with the 3053 cm-1 band—an asymmetric CH2 resonance- in blue. The signal-to-noise ratio (SNR), measured at the edge of a vegetable oil meniscus at the 2927 cm−1 band with SRS (Figs. 2C and S2), was calculated to be 34.627.

Figure 2
figure 2

MANTIS system performance. (A) Resolution of CARS (Green), SRS (Red) with a 1 µm polystyrene bead, and Multiphoton Fluorescence (Magenta) with a 500 nm yellow-fluorescent polystyrene bead. Measured resolutions summarized in Table S1. (B) Preparation of mixed beads containing 2 µm polystyrene beads (blue, 2927 cm−1) with 1–10 µm PMMA beads (yellow, 3053 cm−1). Peaks specific to each bead type (C) CARS (Green) and SRS (Red) of signal-to-noise profile measured at a vegetable oil–air meniscus at 2927 cm−1. (D) SHG imaging of a porcine mitral valve ex vivo. (E) Thermal Imaging of a 1951 U.S. Air Force Target.

Full size image

Resolution standards to verify SHG imaging resolution are not commercially available. In place of a controlled standard, a collagen-rich biological sample (a porcine mitral valve) was imaged at a high spatial sampling density (250 nm/px) and the finest resolvable fibrillar structures were measured as a proxy for lateral resolution. Sample images of mitral valve collagen are shown in Figs. 2D and S3. Example calculations of fibril diameter are show in Fig. S1. Fibrillar collagen bundles and quaternary structure are clearly visualized. While fibrillar structures vary substantially in diameter3, the resolving power of SHG on MANTIS (Table S2) is well within our targeted resolution goals being able to resolve sub-micron spatial features.

The FLIR thermal camera resolution was estimated by identifying the smallest resolvable group on a 1951 United States Air Force resolution target (Fig. 2E). Resolving group 6, element 4 equates to a lateral resolution of 6.9 µm. The depth of focus of the thermal camera is about 40 µm, based on an f/4.0 aperture stop of the objective lens in the thermal camera. Since the depth of focus of the thermal camera is substantially larger than the depth of focus of the nonlinear imaging system, registration of the focal planes axially was not an issue. The factory measured temperature resolution is specified to be 0.1 °C. The nonlinear and thermal microscopy field view were independently adjusted to centrally registered the corner of a 10 μm grid target. Switching between thermal and nonlinear microscopy fields of view was found to repeatable within 1 µm radial to imaging turret rotation and 10 µm tangent to imaging turret rotation (Fig. 1A, yellow arrows). Since field of view of the thermal camera is nearly four times that of the nonlinear microscopy field of view—making the coincidence of each modality’s field of view with each other straight-forward Fig. S5. Once aligned, the fields of view were observed to remain well registered for more than a week following initial alignment. It takes about 90–180 s to switch between nonlinear and thermal imaging arms over the same sample. The sub-millimeter repeatability of positioning multiple imaging arms readily enables multiplexing of thermal and nonlinear microscopy on the same field of view without needing to reposition the sample.

In vitro imaging

To verify the capability of MANTIS to perform in vitro imaging, cultured NG108 cells—a spiking neuron-astrocytoma hybridoma cell line—were imaged with CARS and MPF, as seen in Fig. 3D. Briefly, cells were plated on poly-D-lysine coated cover glass for 48 h prior to imaging and maintained in Dulbecco’s Modified Eagle Medium (DMEM) supplemented with 10%v/v fetal bovine serum and 5 mM L-glutamine. 12-h prior to imaging, medium was replaced with DMEM containing 3%v/v FBS to morphologically differentiate adherent cells. Performing CARS imaging at 2830 cm−1 resonance highlights a lipid-dominant CH2 symmetric stretch mode, which relates to cellular projections and lipid-rich intracellular contents such as organelles, lipid droplets, and vesicles (Fig. 3A). Autofluorescence from NADH and FAD can be used to extrapolate metabolic information relating to relative aerobic metabolism dynamics and metabolic cofactor distribution throughout the cell, Together, combined CARS and MPF demonstrate the structural and functional capacities of multimodal nonlinear imaging in vitro (Fig. 3B, C). Blackbody thermal imaging of cells in media in an upright configuration was not feasible due to the strong absorption of water in the wavelength range measured by the thermal camera (3 to 5-µm). Imaging with cellular resolution with thermal microscopy is possible in an upright configuration (Fig. S4A), with the ability to resolve cellular temperature distribution. However, once medium is introduced to the imaging field of view (Fig. S4B), cellular morphology is occluded due to strong water absorption of the sensed wavelength range. This has been addressed by others by utilizing inverted imaging configurations17,28,29, however resolvable cellular morphology were not demonstrated. Imaging of cells through glass coverslips is possible, however strong water absorption still occluded cellular morphology.

Figure 3
figure 3

In vitro imaging capabilities of MANTIS. NG108 cells observed with CARS (A) at 2830 cm−1, a protein-dominant peak, NADH autofluorescence (B), FAD autofluorescence (C), and as a composite overlay of the three channels (D). Images were acquired separately and coregistered together with a rigid transformation.

Full size image

Combined SHG and MPF was demonstrated on fresh ex vivo porcine mitral valve samples (Fig. S5). Tissue samples were harvested from porcine cardiac tissue postmortem and stored in phosphate buffered saline for 72 h prior to imaging. Samples were mounted on a standard microscopy slide and imaged through leveled and supported cover glass in contact with the tissue surface. Simultaneous autofluorescence and SHG under 930 nm illumination with the femtosecond laser source yielded high resolution microstructure of endogenous elastin and collagen, free of exogenous labeling30. Changes in collagen and elastin can dangerously disrupt the structure and function of heart valves31. Having methods to study new interventions to improve heart valve function could be valuable to cardiac biomechanics researchers. Combining SHG with SRS at 2880 cm-1 to broadly highlight lipids allows for the visualization of tissue architecture in unfixed frozen sections of murine cervix (Fig. S6). Distinct chemical and structural differences are notable between the epithelium and stroma of the cervix. The stroma is dense with collagen and low in cellular content, while the epithelium is rich in lipids and proteins due to cellularization. The lamina propria of blood vessels are also discernable with SHG, appearing as fine tubular structures with strong SHG signal, providing an indication of tissue vascularization. Visualizing the changes in vasculature, cellular, extracellular matrix protein distribution throughout cervix shows potential for studying parturition processes and cancer progression in the cervix without the need for exogenous labelling.

Figures 4, S7, and S8 demonstrate multimodal nonlinear imaging with CARS, SRS, SHG, and MPF in an ex vivo rat sciatic nerve preparation. Nerves were harvested from Sprague–Dawley rats and imaged immediately postmortem. Contrast from CARS and SRS primarily highlights myelin (Fig. S7B, C). The CARS and SRS images correspond well with the MPF images of FluoroMyelin green (ThermoFisher Inc| Figs. S7E and S8C, D) but are less prone to heterogeneous dye uptake. While myelin plays a major functional role in nerve signal conduction, collagen is a major structural component of the sciatic nerve that provides a protective exterior sheath and additional mechanical stability within the nerve. Collagen contrast from SHG can be used to visualize the epineurium and fascicle-residing collagen (Fig. S9). These four modalities together therefore offer key structural insight to the sciatic nerve and can be used to study nerve injury, regeneration, and disease. Furthermore, the ability to flexibly apply these techniques to different sample types, between live cells and excised tissue specimen, highlights a key design goal of the imaging platform.

Figure 4
figure 4

Ex vivo rat sciatic nerve samples imaged with high-speed multimodal nonlinear and thermal microscopy. (A) Combined image of B-E. Images and corresponding high-speed line scans through the centers of the fields of view with (B) CARS at 2880 cm−1. (C) SRS at 2880 cm−1, (D) FluoroMyelin Green, (E) SHG. Red lines on graphs indicate the exposure of the tissue to 1875 nm light from an optical fiber. Light exposure causes a local temperature increase over 1 s (thermal images, (F) and (G)) yielding a corresponding decrease in nonlinear signals.

Full size image

The key advancement allowed by the MANTIS platform is the ability to multiplex optically incompatible imaging techniques such as thermal imaging with nonlinear optical microscopy. Thermal imaging can be used to evaluate fast photothermal processes (Fig. 4F, G), such as infrared neural stimulation (INS)32,33, with coregistered contrast from nonlinear imaging modalities to study real time biophysical dynamics (Fig. 4A, F). To assess the multimodal capabilities of MANTIS, readout of multimodal nonlinear contrast was directly correlated with high-speed temperature changes during infrared neural stimulation in a fresh ex vivo sciatic nerve preparation (Fig. 4). Point-scanned line repeats through the center of the images fields of view were employed to obtain high speed (488 Hz) nonlinear signals during IR stimulation. Average intensities across the entire acquired line scan at each timepoint are reported for each modality. Reported temperature data is a the average temperature of the entire FOV denoted in Fig. 4G acquired at 180 Hz. Intriguingly, our results show a notable decrease in nonlinear signals from CARS (Fig. 4B), SRS (Fig. 4C), and MPF (Fig. 4D) that tracks closely with temperature increases (Fig. 4G) over the period of one second. Notably, SHG signals (Fig. 4E) did not exhibit a discernable decrease in signal, which is attributable to the lack of collagen present through where line scans were acquired (center of the field of view, Fig. 4). However, decreases in SHG signal during IR stimulation have been verified in other FOVs, suggesting this signal decrease is a more general physical phenomenon. Decreases in nonlinear signal were found to be caused by a defocusing artifact induced by the spatial thermal gradient inherent to infrared neural stimulation in the microscope’s field of view. Resolving photothermal effects of IR heating in biological tissues at high speed, such as during INS, will provide invaluable information in studying in vivo applications of INS.

In vivo imaging

All protocols using live animals were performed in accordance with and are approved by Vanderbilt University Institute for Animal Care and Use Committee (VU-IACUC, Protocol M1600084). All protocols using live animals were performed in accordance with ARRIVE guidelines for animal use. An acute craniotomy preparation in an anesthetized rat model was utilized to validate the in vivo capabilities of MANTIS. Sprague–Dawley rats were anesthetized with intraperitoneal injections of ketamine (40–80 mg/kg) and xylazine (10 mg/kg) for 40 min prior to surgery. Anesthetic depth was monitored every 10 min and maintained with follow-up half-dose anesthetic injections every 90–120 min as needed. Animals were mounted in a stereotaxic frame under the microscope objective while a 2 mm- × -2 mm section of skull and dura mater overlying the somatosensory cortex was carefully removed. Tissue hydration was maintained with sterile saline throughout imaging. Animals were sacrificed by anesthetic overdose and cervical dislocation following experiments. During imaging experiments, pixel dwell times needed to be increased to 8 µs to account for signal loss in moving the SRS detection path into an epi-detection configuration. Epi-SRS detection was implemented by replacing the dichroic mirror immediately preceding to the objective in the incident light path with a polarizing beam splitter cube and quarter wave plate23. Images were initially acquired at a high spatial sampling density (0.5 µm/px, 800 × 800 px) with a 0.2 Hz framerate. However, by reducing the spatial sampling density and overall image sizes (3 µm/px, 256 × 256 px) over a comparable field of view, we were able to achieve 2 Hz framerates. Subsequently, repeated point-based line scanning could be implemented with a 488 Hz line rate to observe faster biophysical dynamics. This flexibility emphasizes the trade-offs between spatial and temporal sampling in nonlinear microscopy approaches in vivo. Acquiring images using CARS, MPF, or SHG needed to be performed sequentially, due to the availability of only one PMT detector at the time of experiments. However, any one of the three PMT-requiring modalities can be multiplexed with SRS contrast, since SRS uses a separate detector in an adjacent detection path. Excitation and detection wavelengths are summarized in Table S1.

Examples of in vivo images can be seen in Fig. 5. A protein-dominant SRS resonance at 2930 cm-1 (Fig. 5B) and a lipid-rich CARS resonance at 2850 cm-1 (Fig. 5C) and appears to highlight superficial cellular features as well as vascular perfusion within blood vessels. Autofluorescence contrast from blue and green detection channels are most commonly used to measure endogenous NADH (Fig. 5D, G, Table S1) and FAD (Fig. 5D, E, Table S1). This diffuse NADH signal is likely arising from the neuropil at the pial surface of the cortex, while FAD tends to present more sparsely superficially in the outer cortex in vivo; this weak fluorescence is often spectrally indistinct from other biological autofluorescence34,35. Contrast from SHG (Fig. 5F) revealed unknown morphologies, which may be due to cytoskeletal structures across the pial surface, but is more likely to be residual collagen from the dura mater after surgical preparation. Thermal imaging (Fig. 5H) of the brain surface appears to reveal mostly topographical contrast—vasculature is clearly visible. The lack of optical penetration depth of measured wavelengths of the thermal camera are likely responsible for the brain surface topographical contrast. Temperature fluctuations across the surface of the brain appear to be minimal, even with breathing, heart rate, and motion artifacts. Larger vessels appear slightly warmer than smaller vessels. Focusing artifacts appeared to have some effect on the accurate approximation of the brain surface temperature within 1 °C—particularly noticeable at the edges of the field of view, which likely arise from out of focus signal from underlying bulk tissue due to the high degree of surface curvature in the sample (Fig. 5H). Cellular morphologies were not apparent in thermal images of the brain surface. Thermal imaging of the brain surface was capable of acquisition speeds approaching 180 Hz framerates with a full field of view. This imaging rate can be substantially increased to 500 Hz or more by cropping the image sensor readout without loss in spatial resolution or pixel binning without reduction in field of view. The vessel morphologies visible between thermal and CARS/SRS contrast made fine registration of thermal and nonlinear imaging fields straightforward with rigid registration of manually labelled image features.

Figure 5
figure 5

Multimodal images from a rat brain in vivo. (A) Composite image of (B) (SRS, 2930 cm−1) and (C) (CARS, 2850 cm−1) highlighting lipid and protein-dominated signals, respectively. (D) Composite image of the same field of view in (A) of (E) (Green autofluorescence), (F) (SHG signal), and (G) (Blue autofluorescence, likely NADH). (H) Thermal images of the brain surface, temperatures shown in °C. Red box indicates nonlinear imaging field of view.

Full size image

Fast multimodal nonlinear and thermal microscopy of photothermal cell damage

Infrared neural stimulation utilizes pulsed short-wave infrared laser light to transiently invoke a thermal gradient in neural tissue resulting in activation of neural cells. One of the major concerns about neuromodulation with rapid targeted thermal gradients is that the change in temperature could cause cell or tissue damage. Generally, damage from IR exposure is verified with histology, immunohistochemistry, or staining with cell damage indicators well after exposure, introducing notable lag times to results. Cells are naturally prone to damage at elevated temperatures; however, the role of temperature–time history is often underappreciated in the context of biological thermodynamics. Integrating fast nonlinear and thermal microscopy offers a particularly useful platform for studying the physical and functional impacts of INS on neural cell in the context of cell activation or damage in real time.

The ability to correlate high resolution temporal and spatial thermal information (Fig. 6E, F) with the functional information afforded by nonlinear imaging (Fig. 6A–D), available in the MANTIS platform offers a unique opportunity to visualize biochemical, biophysical, and biomolecular dynamics of cultured NG108 cells during INS. An 8 ms pulse of 1875 nm infrared light was delivered to the cells via a 400 µm diameter optical fiber at average radiant exposures spanning 0.5 (stimulating) and 3 J/cm2 (damaging) while imaging at 10 Hz framerates. Multiphoton fluorescence of cells imaged in saline containing a damage indicator, propidium iodide (PI, 1 μM concentration, Fig. 6C, Table S1), differentiated healthy and damaged cells due to IR exposure. Increases in relative fluorescence greater than 3% were presumed to be indicative of cell damage. Simultaneously, SRS imaging at a CH3 resonance (2930 cm-1, lipid/protein) provides endogenous biochemical information of cells during INS. In this study, SRS images were primarily used to segment cell morphologies to extract viability status (MPF) and interpolate thermodynamics (thermal imaging).

Figure 6
figure 6

Fast Multimodal Nonlinear and Thermal Microscopy of NG108 cells during infrared neural stimulation—Examples of an (A) SRS average intensity projection image of cells with indicated centroids and (B) pooled average SRS timeseries of live (magenta) and damaged (green) cells during IR stimulation across all observed cells. (C) Example of a normalized sum intensity projection of cell viability image timeseries used to indicate cell viability via uptake and fluorescence of propidium iodide. (D) Average pooled MPF timeseries of all observed live and damaged cells during INS. Increases in MPF signal after INS above 3% dF/F over 30 s were considered damaged/dead. (E) Thermal images of NG108 sample preparation during peak temperatures of INS. Registration of thermal and nonlinear fields of view (E, inset) allows for precise spatial temperature mapping. Cellular details are not observable due to strong absorption by the aqueous imaging medium. Coupled with high speed thermal imaging, (F) high resolution temporal and spatial thermal observations can be directly correlated with functional cellular outcomes such as cell viability. Data shows results of 10 different experiments across 3 different INS intensity conditions, n = 1144 cells.

Full size image

Thermal images during INS were acquired at 34 Hz framerates after all nonlinear imaging and stimulation experiments were completed. Temperature information is not simultaneously available with nonlinear observations due to physical limitations. Since the thermal properties of cells mounted in imaging saline behave thermodynamically like water upon INS, cell position data was registered to thermal imaging videos acquired after IR stimulation experiments while holding the stimulation geometry constant. Based on alignment against a fluorescent target performed immediately before cell imaging experiments, the translation matrix needed to register nonlinear spatial coordinate to thermal spatial coordinates was applied to cell centroid positions and used to obtain spatial thermodynamic information for each cell. Registration was performed with an iterative closest point algorithm implemented in MATLAB to register manually selected features of a fluorescent target (Chroma, Rochester, NY, USA) observable in both nonlinear and thermal imaging modes. Cell centroids were calculated from cell morphologies identified with SRS (Fig. 6A) in Fiji utilizing a seeded watershed segmentation algorithm24,36. Cell positions coordinates in the thermal camera field of view were used to interpolate temperature–time and temperature-space thermal data on a per-cell basis using built-in 2D interpolation functions in MATLAB.

Since the stimulating IR light (1875 nm) is strongly absorbed by water, the immersion medium required for nonlinear imaging had a substantial impact on the heating of cells, leading to potentially large discrepancies in thermal imaging measurements. To minimize optical absorption of 1875 nm stimulation light by the aqueous immersion medium necessary for the nonlinear microscopy objective (Olympus XLUMPLNFL 20X, 1.0NA), spectroscopic-grade deuterated water (Sigma Aldrich, St. Louis, MO, USA) was used as an immersion medium33. Cells remained immersed in normal imaging media and were separated from heavy water by the coverslip on which the cells were mounted. The reduction in IR absorption in the objective lens immersion media was enough to allow for representative spatial thermal measurements of INS on aqueous cellular samples with the thermal camera. Functional observations of cell death aligned well with peak spatial temperature maps (Fig. 6E, inset). It is expected that absolute temperature measurements at the sample are likely to differ with and without the presence of deuterated immersion medium during INS, however the spatial and temporal dynamics of IR-induced cellular thermal gradients are expected to remain consistent and comparable across all exposure conditions. With this assumption, physiological comparisons can be effectively drawn with the understanding that the observed absolute temperatures are likely slightly higher than during nonlinear image experiments where immersion medium may impact INS.

Spatially registered observations of cell functional, chemical, and temperature dynamics is only made possible by a combined and accurately coregistered nonlinear and thermal imaging platform such as the MANTIS platform. From these multimodal imaging experiments, we observed that cells that experience more rapid changes in temperatures as a function of time were more likely to be damaged (Fig. 6E, F). Damaged cells are more likely to be located near low spatial thermal gradient values, which correspond to local maxima in spatial heating profiles (Fig. 6E, inset). However, cells outside the fiber illumination would be expected to present similar spatial thermal gradients as cells at peak levels of heating—introducing some heterogeneity in the spatial thermal gradient information. Considering spatial and temporal thermal gradient information together clarifies the thermodynamic difference between heated and unheated cells. Numerous cells appear to survive rapid temperature changes while others do not, however damaged cells are far more likely to undergo rapid heating. This observation demonstrates the variability in cell physiological responses due to INS. Distinct increases in mean multiphoton fluorescence timeseries indicate cell damage within 30 s of IR exposure. Cell damage appears to coincide with stable mean SRS CH3 amplitude following IR stimulation, while live cells present a decrease in 2930 cm-1 SRS amplitude. The basis for SRS signal increase is currently unclear, though we speculate attribution to a balance between cell swelling and morphological changes triggered by cell damage. Currently, there remains to be extensive validation of changes in endogenous SRS signal in the context of live cell imaging and damage beyond lipid storage and cell membrane dynamics. Experimentally, SRS images were primarily used in Fig. 6 to identify, segment, and locate cells for temperature and viability data calculations. However, this platform readily enables such explorations into the molecular basis of SRS signal changes simultaneously with commonly used live cell fluorescence imaging probes.

Discussion

We demonstrate the first imaging system that combines CARS, SRS, MPF, SHG, and thermal imaging into a single microscope for biomedical imaging applications. The similarities in illumination and detection instrumentation needed for nonlinear excitation has been exploited in the past to design multimodal imaging platforms. But as demonstrated here, our novel system design includes a movable turret that allows for overlaying these different modalities to achieve a more complete picture of biological processes to be imaged. This logically suggests that another imaging modality that typically could not be integrated into a nonlinear optical microscopy path may be similarly incorporated. Such flexibility of optical microscopy design provides the opportunity to study physiology in unique and dynamic ways.

The proposed imaging system, MANTIS, was built with modularity and expandability in mind. On the nonlinear imaging arm, the current configuration has four optional detection ports—two each in the epi- and forward detection configurations. Additional channels can easily be added to expand simultaneous imaging capabilities based on research needs. Lock-in detection arms can be refitted with the appropriate optical filters for transient absorption imaging. Sum-frequency generation imaging can also be readily integrated for studying ordered molecular orientation and interfacial phenomena in biological samples. The widefield reflectance imaging arm may be useful for integrating laser speckle, diffuse reflectance, optical coherence tomography and microscopy, or spatial frequency domain imaging with the correct illumination optics to map tissue blood flow, oxygenation, and optical properties. Furthermore, the widefield reflectance and thermal imaging arms were built on detachable 96-mm optical rails so that other imaging methods may be integrated based on future research needs. The multi-armed imaging concept can also yield more utility out of a condensed instrument footprint, which may be useful in places where laboratory floorspace is limited.

The scanning optics used in nonlinear imaging arm can achieve 1 kHz scan rates. A galvo-scanning mirror pair was chosen over a faster galvo-resonant scanning mirror pair, which typically offer an order of magnitude increase in scanning rates. By doing so, improved control over image sampling densities can be achieved to readily span subcellular and multicellular scales. This is exemplified by the difference in sampling density observed between Figs. 3 and 5. As demonstrated in Fig. 4, our imaging system can perform nonlinear imaging with line scans approaching 0.5 kHz with detectable amounts of signal. This is enough to resolve high-speed biophysical dynamics, such as during neural modulation or optogenetic stimulation, from a functional, chemical, and physical standpoint. Kilohertz bandwidth scanning rates are sufficient to achieve visualization of neuronal action potentials as demonstrated by Lee et al. with balanced detection SRS and calcium fluorescence microscopy9.

Both femtosecond and picosecond lasers were included in the design of MANTIS for signal and spectral resolution considerations. The bandwidth of the picosecond laser source (around 0.5 nm) allows for higher spectral resolutions (around 10 cm-1) which is critical for nonlinear Raman imaging37. However, the peak powers of the picosecond source are orders of magnitude lower than that of the femtosecond laser, yielding less signal during multiphoton fluorescence and higher harmonic imaging. At the time of construction, the use of broadband nonlinear Raman techniques, such as pulse shaping and spectral focusing had yet to be realized for fast imaging in vivo38. As such, design considerations to optimize narrow band spectral resolution and imaging speed were a priority. A number of broadband spectral NRI techniques have since been described 39,40, with a handful demonstrated in vivo10. The flexibility of the existing instrumentation on MANTIS readily allows for the integration such broadband approaches.

Using blackbody thermal microscopy to characterize sample temperatures in conjunction with nonlinear imaging, or any laser-scanning microscopy approach, has yet to be previously published. Estimating temperature with a thermal camera provides a more direct measurement than other approaches, such as fluorescence-based methods. The ability to measure thermal information alongside the functional and structural information offered by nonlinear microscopy provides a unique instrument to explore new questions in biophysics, particularly ex vivo and in vivo. Practically, thermal microscopy presents some advantages and drawbacks depending on the model systems being imaged. We have found that high resolution fluorescence microscopy can be difficult to interpret in highly thermally dynamic systems with high numerical apertures at high imaging speeds, which became apparent in applying infrared neural stimulation in sciatic nerve (Fig. 4) as well as in cells. Having thermal information to contrast and systematically compensate for thermally induced effects is something that our imaging platform lends itself to accomplish. Since imaging depth with thermal microscopy is limited due to the optical penetration depth of water in the 3–5 µm wavelength spectral regime, this means that SWIR based imaging methods are ideally suited for measuring surface temperatures in water-dominant samples such as biological tissues. This is even more apparent in vitro, where imaging through any amount of aqueous medium occludes cellular morphology (Fig. S4). While others have demonstrated blackbody thermal imaging of cells via inverted microscopy through cover glass17,18, visualizing cell morphology with thermal imaging has not been previously demonstrated—our observations support the previous work. Thermal microscopy on MANTIS can be performed readily and reliably on tissues ex vivo (Fig. 4F) and in vivo (Fig. 5H). Topographical features such as blood vessels tissue surfaces are clearly visible in the brain. The pronounced topography of these anatomical features can be helpful when performing fine registration nonlinear and thermal imaging fields. However in tissue samples with limited visibility of morphology (e.g. Figure 4F) thermal imaging is limited to providing purely temperature information. Temperature at aqueous interfaces are useful in approximating spatial temperature distributions in vitro. Combining surface temperatures with finite element heat transfer modeling and precise control over sample thermodynamics can be used to render volumetric estimates of temperature in vitro18. In combination with fluorescence or other nonlinear imaging modalities, imaging the thermodynamics of adherent cells in vitro or in tissues with thermal microscopy could be quite useful. Consequently, multimodal methods almost become an essential requirement to correlate temperature and real-time cellular observations. Our microscopy platform readily allows such observation to be made and flexibly configured to address a wide range of biological and physiological questions.

As expected, cell viability at elevated temporal thermal gradients values is more likely to result in cell death than at lower gradient values (Fig. 6F, axes histograms). However, as evident by the overlap in cell viability as a function of thermodynamics in Fig. 6F, peak thermal gradients are not absolutely predictive (Fig. 6F, scatter plot). While cells heated quickly are more likely to become damaged, rapid heating is not necessarily a death sentence for cells. The variability in thermally evoked cell damage illustrates the need for tools to study underlying functional and biomolecular dynamics evoked by IR light on a cell-to-cell basis. Exposure time and time after exposure becomes a crucial dimension of cellular physiology to explore in the context of infrared neural stimulation. Endogenous lipid/protein SRS signals are stable (Fig. 6A, B) and a fluorescence marker for cell damage (Fig. 6C, D) increases following damaging levels of INS. Intriguingly, live cells show a decrease in SRS amplitude and stable fluorescence intensities. Cell swelling could cause CH3 related SRS amplitudes to decrease to the increase in cell volume, though the reasoning for this observation is unclear. Regardless, the unique ability to correlate temperature, cell function, and cell biochemistry together shows that this imaging platform can be a powerful research tool in certain research applications. Having temperature information to correlate directly with cell physiology or damage allows for more detailed characterization of safety and damage levels for INS applications. Registration of thermal and nonlinear imaging fields within a couple of microns is key to enable direct correlation of temperature–time dynamics with cell functionality since cells are not readily visible with thermal imaging.

To practically extend this imaging platform, cell viability markers like that shown in Fig. 6 can readily be substituted for calcium sensors, voltage probes, FRET constructs, or gene transcription assays. Using hyperspectral or bio-orthogonal SRS alongslide functional contrast, it is possible to correlate endogenous contrast metrics with functional information to improve optical tools in studying cellular physiology. SRS often provides relatively nonspecific biomolecular information relative to fluorescence imaging strategies; combining functional fluorescence and SRS imaging during INS can expand the current understanding of the impact of cellular physiological dynamics on SRS signals. Similar approaches can be used to understand the effects of rapid temperature changes on more specific aspects of cellular physiology to improve our general understanding of the impact of thermodynamics on cells. Such multidimensional data set can make further use of multivariate statistics and machine learning based analyses to integrate and observe the interplay of biochemical, biomolecular, and biophysical dynamics in unprecedented ways.

Beyond INS, similar approaches as demonstrated in Fig. 6 may readily be adapted to study laser tissue interactions associated with optogenetic or nanoplasmonic neuromodulation, laser preconditioning of immunological response, or photodynamic therapy of infectious or cancerous model systems where temperature changes may be involved. These types of experiments are enabled only by an imaging platform that integrates nonlinear and thermal microscopy with high enough spatial and temporal resolution capabilities to study the processes at hand. While neuroscientific questions provide a valuable benchmark in terms of evaluating imaging speed, this platform is just as applicable to other areas of biomedical research.

Undoubtedly, an imaging platform that simultaneously integrates thermal and nonlinear imaging would be practically important in answering many questions in the aforementioned disciplines. It becomes technically difficult to accommodate the disparity in optical detection wavelengths for use in a broad range of cellular, tissue, and live animal preparations. Tradeoffs between sample flexibility and optical access were major considerations in the imaging system’s three-armed design. Inherent hardware limitations prevent the design a truly simultaneous multimodal nonlinear and thermal imaging system. Nonetheless, the MANTIS platform offers a novel approach to creatively answer fundamental and translationally relevant questions about biology and thermodynamics which are applicable broadly to biomedical research.

Conclusion

We present a novel and flexible multimodal optical imaging system that combines nonlinear and thermal microscopies for biomedical imaging applications. Our platform can be used to overlay functional, structural, and biochemical information from a single specimen tracked over time at high speed and spatial resolution. Such approaches can be applied to study dynamic biophysical processes in cells and tissues, in vitro and in vivo. In conjunction with continually expanding molecular biology tools, this instrument and related imaging methods will aid in studying physiological and biochemical processes from multiple perspectives in many fields including neuroscience, cancer biology, metabolic disease, tissue biomechanics, and much more.