Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Fast wide-field upconversion luminescence lifetime thermometry enabled by single-shot compressed ultrahigh-speed imaging

Abstract

Photoluminescence lifetime imaging of upconverting nanoparticles is increasingly featured in recent progress in optical thermometry. Despite remarkable advances in photoluminescent temperature indicators, existing optical instruments lack the ability of wide-field photoluminescence lifetime imaging in real time, thus falling short in dynamic temperature mapping. Here, we report video-rate upconversion temperature sensing in wide field using single-shot photoluminescence lifetime imaging thermometry (SPLIT). Developed from a compressed-sensing ultrahigh-speed imaging paradigm, SPLIT first records wide-field luminescence intensity decay compressively in two views in a single exposure. Then, an algorithm, built upon the plug-and-play alternating direction method of multipliers, is used to reconstruct the video, from which the extracted lifetime distribution is converted to a temperature map. Using the core/shell NaGdF4:Er3+,Yb3+/NaGdF4 upconverting nanoparticles as the lifetime-based temperature indicators, we apply SPLIT in longitudinal wide-field temperature monitoring beneath a thin scattering medium. SPLIT also enables video-rate temperature mapping of a moving biological sample at single-cell resolution.

Introduction

Temperature is an important parameter associated with many physical, chemical, and biological processes1. Accurate and real-time (i.e., the actual time during which the event occurs) temperature sensing at microscopic scales is essential to both industrial applications and scientific research, including the examination of internal strains in turbine blades2, control of the synthesis of ionic liquids3, and theranostics of cancer4. In the past decade, photoluminescence lifetime imaging (PLI) has emerged as a promising approach to temperature sensing5. Because photoluminescence can be both excited and detected optically, the resulting non-contact PLI possesses a high spatial resolution6,7,8. This advantage not only overcomes the intrinsic limitation in spatial resolution of imaging thermography due to the long wavelengths of thermal radiation but also avoids heat-transfer-induced inaccuracy in conventional contact methods9. Moreover, independent of prior knowledge of samples’ physical properties (e.g., emissivity and Grüneisen coefficient10,11), PLI brings in higher flexibility in sample selection. Furthermore, PLI is less susceptible than the intensity-based measurements to inhomogeneous signal attenuation, stray light, photobleaching, light’s path length, and excitation intensity variations12,13,14,15,16. Finally, PLI does not rely on the concentration of labeling agents8, which eliminates the need for special ratiometric probes17. Overcoming many challenges in previous methods, PLI is becoming a popular choice for optical thermometry18,19,20,21,22.

The success of PLI in temperature mapping depends on two essential constituents: temperature indicators and optical imaging instruments. Recent advances in biochemistry, materials science, and molecular biology have discovered numerous labeling agents23,24,25,26 for PLI-based temperature sensing. Among them, lanthanide-doped upconverting nanoparticles (UCNPs) are ideal candidates. Leveraging the long-lived excited states provided by the lanthanide ions, UCNPs can sequentially absorb two (or more) low-energy near-infrared photons and convert them to one higher-energy photon. This upconversion process allows using excitation power densities several orders of magnitude lower than those needed for simultaneous multi-photon absorption27,28. The near-infrared excitation, with smaller tissue extinction coefficients, also gains deeper penetration29. Besides, the upconverted luminescence, particularly the Boltzmann-coupled emission bands in co-doped erbium/ytterbium (Er3+/Yb3+) systems, is highly sensitive to temperature changes30,31. Moreover, long-lived (i.e., microseconds to milliseconds) photoluminescence of UCNPs circumvents interferences from autofluorescence and scattering during image acquisition, which translates into improved imaging contrast and detection sensitivity. Finally, because of advances in their synthesis and surface functionalization coupled with the innovation of core/shell engineering, over the years, UCNPs have become much brighter, photostable, biocompatible, and non-toxic32. As a result of these salient merits, UCNPs are one of the frontrunners in temperature indicators for PLI.

Advanced optical imaging is the other indispensable constituent in PLI-based temperature mapping33. To detect photoluminescence on the time scale of microseconds to milliseconds, like that produced by UCNPs, most PLI techniques use point-scanning time-correlated single-photon counting (TCSPC)34. Although they possess high signal-to-noise ratios, the scanning operation leads to an excessively long imaging time to form a two-dimensional (2D) lifetime map because extended pixel dwell time is required to record the long-lived emission35. To accelerate data acquisition, wide-field PLI modalities based on parallel collection in time-domain and frequency-domain have been developed36. In the time domain, these techniques extend the TCSPC technique to wide-field imaging (e.g., TimepixCam37 and Tpx3Cam38). Photoluminescence decay over a 2D field of view (FOV) is synthesized from >100,000 frames, which requires the emission to be precisely repeatable. Alternatively, the frequency-domain wide-field PLI techniques39,40 use phase difference between the intensity-modulated excitation and the received photoluminescence signal to determine the 2D lifetime distribution. Nevertheless, limited by the range of frequency synthesizers, the measurable lifetimes are mostly restricted to ≤100 µs, which is shorter than the lifetimes of most UCNPs. Akin to the time-domain techniques, these systems rely on the integration over many periods of modulation intensity, during which the sample must remain stationary. Thus far, existing PLI techniques fall short in 2D temperature sensing of moving samples with a micrometer-level spatial resolution.

To surmount these limitations, we report an optical temperature mapping modality, termed single-shot photoluminescence lifetime imaging thermometry (SPLIT). Synergistically combining dual-view optical streak imaging with compressed sensing41, SPLIT records wide-field luminescence decay of Er3+, Yb3+ co-doped NaGdF4 UCNPs in real time, from which a lifetime-based 2D temperature map is obtained in a single exposure. Largely advancing existing optical thermometry techniques in detection capabilities, SPLIT enables longitudinal 2D temperature monitoring beneath a thin scattering medium and dynamic temperature tracking of a moving biological sample at single-cell resolution.

Results

Operating principle of SPLIT

The schematic of the SPLIT system is shown in Fig. 1. A 980-nm continuous-wave laser (BWT, DS3-11312-113-LD) is used as the light source. The laser beam passes through a 4f system consisting of two 50-mm focal length lenses (L1 and L2, Thorlabs, LA1255). An optical chopper (Scitec Instruments, 300CD) is placed at the back focal plane of lens L1 to generate 50-µs optical pulses. Then, the pulse passes through a 100-mm focal length lens (L3, Thorlabs, AC254-100-B) and is reflected by a short-pass dichroic mirror (Edmund Optics, 69-219) to generate a focus on the back focal plane of an objective lens (Nikon, CF Achro 4×, 0.1 numerical aperture, 11-mm field number). This illumination scheme produces wide-field illumination (1.5 × 1.5 mm2 FOV) to UCNPs at the object plane.

Fig. 1: Schematic of the SPLIT system.
figure 1

The illustration shows data acquisition and image reconstruction of luminescence intensity decay in a letter “C”. L1–L5, lens.

The near-infrared excited UCNPs emit light in the visible spectral range. The decay of light intensity over the 2D FOV is a dynamic scene, denoted by \(I(x,{y},{t})\). The emitted light is collected by the same objective lens, transmits through the dichroic mirror, and is filtered by a band-pass filter (Thorlabs, MF542-20 or Semrock, FF01-660/30-25). Then, a beam splitter (Thorlabs, BS013) equally divides the light into two components. The reflected component is imaged by a CMOS camera (FLIR, GS3-U3-23S6M-C) with a camera lens (Fujinon, HF75SA1) via spatiotemporal integration (denoted as the operator \({{{{{\bf{T}}}}}}\)) as View 1, whose optical energy distribution is denoted by \({E}_{1}\left({x}_{1},{y}_{1}\right)\).

The transmitted component forms an image of the dynamic scene on a transmissive encoding mask with a pseudo-random binary pattern (Fineline Imaging, 50% transmission ratio; 60-µm encoding pixel size). This process of spatial encoding is denoted by the operator \({{{{{\bf{C}}}}}}\). Then, the spatially encoded scene is imaged by a mechanical streak camera. In particular, the scene is relayed to the sensor plane of an electron-multiplying (EM) CCD camera (Nüvü Camēras, HNü 1024) by a 4f imaging system consisting of two 100-mm focal length lenses (L4 and L5, Thorlabs, AC254-100-A). A galvanometer scanner (Cambridge Technology, 6220H), placed at the Fourier plane of the 4f imaging system, temporally shears the spatially encoded frames linearly to different spatial locations along the \({x}_{2}\) axis of the EMCCD camera according to their time of arrival. This process of temporal shearing is denoted by the operator \({{{{{\bf{S}}}}}}\). Finally, the spatially encoded and temporally sheared dynamic scene is recorded by the EMCCD camera via spatiotemporal integration to form View 2, whose optical energy distribution is denoted by \({E}_{2}\left({x}_{2},{y}_{2}\right)\).

By combining the image formation of \({E}_{1}\left({x}_{1},{y}_{1}\right)\) and \({E}_{2}\left({x}_{2},{y}_{2}\right)\), the data acquisition of SPLIT is expressed by

$$E={{{{{\bf{T}}}}}}{{{{{\bf{M}}}}}}I,$$
(1)

where \(E\) denotes the concatenation of measurements \({\left[{E}_{1},{\alpha E}_{2}\right]}^{T}\) (the superscript T denotes the transpose), \({{{{{\bf{M}}}}}}\) denotes the linear operator \({\left[{{{{{\bf{1}}}}}},\alpha {{{{{\bf{SC}}}}}}\right]}^{T}\), and \(\alpha\) is a scalar factor introduced to balance the energy ratio between the two views during measurement42. The hardware of the SPLIT system is synchronized for capturing both views (detailed in “Methods”) that are calibrated before data acquisition (detailed in Supplementary Note 1 and Supplementary Fig. 1).

After data acquisition, \(E\) is processed by an algorithm that retrieves the datacube of the dynamic scene by leveraging the spatiotemporal sparsity of the dynamic scene and the prior knowledge of each operator43,44. Developed from the plug-and-play alternating direction method of multipliers (PnP-ADMM) framework45,46, the reconstruction algorithm of SPLIT solves the minimization problem of

$$\hat{I}=\mathop{{{{{{\rm{argmin}}}}}}}\limits_{\,I}\left\{\frac{1}{2}\Vert{{{{{\bf{T}}}}}}{{{{{\bf{M}}}}}}I-{E}\Vert_{2}^{2}+R(I)+{{{{{{\bf{I}}}}}}}_{+}(I)\right\}.$$
(2)

Here, \({\Vert\!\!\cdot\!\!\Vert }_{2}\) represents the l2 norm. The fidelity term, \(\frac{1}{2}\Vert{{{{{\bf{T}}}}}}{{{{{\bf{M}}}}}}I-{E}\Vert_ {2}^{2}\), represents the similarity between the measurement and the estimated result. \(R(\bullet )\) is the implicit regularizer that promotes sparsity in the dynamic scene. \({{{{{{\bf{I}}}}}}}_{+}(\cdot )\) represents a non-negative intensity constraint. Compared to existing reconstruction schemes47,48,49, PnP-ADMM implements a variable splitting strategy with a state-of-the-art denoiser to obtain fast and closed-form solutions to each sub-optimization problem, which produces a high image quality in reconstruction (see Supplementary Notes 2 and 3 and Supplementary Fig. 2). The retrieved datacube of the dynamic scene has a sequence depth (i.e., the number of frames in a reconstructed movie) of 12–100 frames, each containing 460 × 460 \((x,y)\) pixels. The imaging speed is tunable from 4 to 33 thousand frames per second (kfps) (detailed in “Methods”).

The reconstructed datacube is then converted to a photoluminescence lifetime map. In particular, for each (\(x,y\)) point, the area under the normalized intensity decay curve is integrated to report the value of the photoluminescence lifetime50. Finally, using the approximately linear relationship between the UCNPs’ lifetime and the physiologically relevant temperature range (20–46 °C in this work)51,52, the 2D temperature distribution, \(T\left(x,y\right)\), is calculated by

$$T\left(x,y\right)={c}_{{{{{{\rm{t}}}}}}}+\frac{1}{{S}_{{{{{{\rm{a}}}}}}}}\int \frac{\hat{I}\left(x,y,t\right)}{\hat{I}\left(x,y,0\right)}{{{{{\rm{d}}}}}t},$$
(3)

where \({c}_{{{{{{\rm{t}}}}}}}\) is a constant, and \({S}_{{{{{{\rm{a}}}}}}}\) is the absolute temperature sensitivity33. The derivation of Eq. (3) is detailed in Supplementary Note 4. Leveraging the intrinsic frame rate of the EMCCD camera, the SPLIT system can generate lifetime-determined temperature maps at a video rate of 20 Hz.

Quantification of the system’s performance of SPLIT

We prepared a series of core/shell UCNP samples to showcase SPLIT’s capabilities. These UCNPs shared the same NaGdF4: 2 mol% Er3+, 20 mol% Yb3+ active core of 14.6 nm in size, while differed by the thickness of their undoped NaGdF4 passive shell of 1.9, 3.5, and 5.6 nm (Fig. 2a and detailed in Supplementary Note 5). All of the UCNP samples were of pure hexagonal crystal phase (Supplementary Fig. 3). Under the 980-nm excitation, upconversion emission bands of all samples were measured at around 525/545 nm and 660 nm, which correspond to the 2H11/2/4S3/2 → 4I15/2 and 4F9/2 → 4I15/2 radiative transitions, respectively (Fig. 2b–c).

Fig. 2: Quantification of the performance of the SPLIT system.
figure 2

a Images of core/shell UCNPs acquired with a transmission electron microscope. Scale bar: 25 nm. b Normalized upconversion spectra of UCNPs shown in (a). c Simplified energy level diagram of Yb3+-Er3+ energy transfer upconversion excitation and emission. d Temporally projected image of photoluminescence intensity decay of the 5.6 nm-thick-shell UCNPs covered by a negative resolution target. e Comparison of averaged light fluence distribution along the horizontal bars (blue) and vertical bars (orange) of Element 5 in Group 4 on the resolution target. Error bar: standard deviation. f Lifetime images of UCNPs with the shell thicknesses of 1.9, 3.5, and 5.6 nm covered by transparencies of letters “C”, “A”, and “N” in green emission. g Time-lapse averaged emission intensities of the samples. h Histograms of photoluminescence lifetimes in the letters shown in (f).

To characterize SPLIT’s spatial resolution, we covered the 5.6 nm-thick-shell UCNP sample with a negative USAF resolution target (Edmund Optics, 55-622). Operating at 33 kfps, SPLIT recorded the photoluminescence decay (Supplementary Movie 1). The temporally projected datacube reveals that the intensity and contrast in the reconstructed image degrade with the decreased spatial feature sizes, eventually leading to the loss of structure whose size approaches that of the encoding pixel (Fig. 2d). The effective spatial resolution was thus determined to be 20 µm (Fig. 2e). Under these experimental conditions, the minimum power density for the SPLIT system was quantified to be 0.06 W mm−2 (detailed in Supplementary Note 6 and Supplementary Fig. 4).

To demonstrate SPLIT’s ability to distinguish different lifetimes, we imaged the UCNPs with shell thicknesses of 1.9, 3.5, and 5.6 nm, covered by transparencies of letters “C”, “A”, and “N”, respectively, using a single laser pulse (Supplementary Movie 2). The lifetime maps of these samples are shown in Fig. 2f, which reveals the averaged lifetimes for the 4S3/2 excited state of samples “C”, “A”, and “N” to be 142, 335, and 478 µs, respectively (Fig. 2g–h). These results were verified by using the standard TCSPC method (detailed in Supplementary Note 7 and Supplementary Fig. 5).

SPLIT’s reconstruction algorithm shows a superb performance to existing mainstream algorithms popularly used in single-shot compressed ultrafast imaging41,42,47,48. By using the experimental data, the comparison demonstrates that the dual-view PnP-ADMM algorithm used by SPLIT is more powerful in preserving spatial features while maintaining a low background, which enables a more accurate lifetime quantification and the ensuing temperature mapping (detailed in Supplementary Note 8 and Supplementary Fig. 6).

Single-shot temperature mapping using SPLIT

We used the 5.6 nm-thick-shell UCNPs as the temperature indicator for SPLIT. The UCNPs’ temperature was controlled by a heating plate placed behind the sample. To image the green (4S3/2) and red (4F9/2) upconversion emissions, the sample was covered by transparencies of a lily flower and a maple leaf, respectively. The temperature of the entire sample was measured with both a Type K thermocouple (Omega, HH306A) and a thermal camera (FLIR, E4) as references. The reconstructed lifetime images in the 20–46 °C temperature range are shown in Fig. 3a–b (see the full evolution in Supplementary Movie 3). Plotted in Fig. 3c–d, the time-lapse averaged intensity over the entire FOV shows that the averaged lifetimes of green and red emissions decrease from 489 to 440 µs and from 458 to 398 µs, which is due to their enhanced multi-phonon deactivation at higher temperatures. We further plotted the relationship between the temperatures and lifetimes for both emission channels (Fig. 3e). Finally, the temperature sensitivities in the preset temperature range were calculated to be \({S}_{{{{{{\rm{a}}}}}}}=-1.90\,{\upmu }{{{{{\rm{s}}}}}}\,{^\circ {{{{{\rm{C}}}}}}}^{-1}\,\)for the green emission and \({S}_{{{{{{\rm{a}}}}}}}=-2.40\,{\upmu }{{{{{\rm{s}}}}}}\,{^\circ {{{{{\rm{C}}}}}}}^{-1}\) for the red emission (see detailed calculation and further analysis in Supplementary Note 9 and Supplementary Fig. 7). Compared to the green emission, the higher temperature sensitivity of the red emission results from the greater energy separation between its emitting state and the adjacent lower-laying excited state (Fig. 2c). Since multi-phonon relaxation rate depends exponentially on the number of phonons necessary to deactivate an excited state to the one below it, the increase in phonon energies at higher temperatures has greater influence over the states with a larger energy gap between them53. These results establish lifetime-temperature calibration curves [i.e., Eq. (3)] for ensuing thermometry experiments.

Fig. 3: Single-shot temperature mapping using SPLIT.
figure 3

a, b Lifetime images of green (a) and red (b) upconversion emission bands under different temperatures. c, d Normalized photoluminescence decay curves of green (c) and red (d) emission bands at different temperatures, averaged over the entire field of view. e Relationship between temperature and mean lifetimes of green and red emissions with linear fitting. Error bar: standard deviation from three independent measurements. f Normalized contrast versus chicken tissue thickness for green and red emission bands with single-component exponential fitting. g Longitudinal temperature monitoring through 0.5  mm-thick fresh chicken tissue.

To demonstrate SPLIT’s feasibility in a biological environment, we conducted longitudinal temperature monitoring under a phantom, made by using the 5.6 nm-thick-shell UCNPs covered by lift-out grids (Ted Pella, 460-2031-S), overlaid by fresh chicken breast tissue. We investigated SPLIT’s imaging depth with varied tissue thicknesses of up to 1 mm (Fig. 3f, Supplementary Note 10, Supplementary Fig. 8, and Supplementary Movie 4). The chicken tissue of 0.5 mm thickness, where both the green and red emissions produced images with full spatial features of the lift-out grid, was used in the following imaging experiments. Subsequently, we cycled the temperature of the sample between 20 and 46 °C. The lifetime distributions of both green and red emissions and their corresponding temperature maps were monitored every 20 and 23 min, respectively, for ~4 h (see the full evolution in Supplementary Fig. 9 and Supplementary Movie 5). As shown in Fig. 3g, the results are in good agreement with the temperature change preset by the heating plate, and decisively showcase how SPLIT can map 2D temperatures over time with high accuracy beneath biological tissue.

We also demonstrated SPLIT using a fresh beef phantom as a scattering medium, where both light scattering and absorption are present (detailed in Supplementary Note 10 and Supplementary Fig. 10). The results reveal better penetration of the red emission over the green counterpart due to its weaker scattering and absorption. More importantly, the results confirm the independence of the measured photoluminescence lifetime of UCNPs to tissue thickness and hence the excitation light power density used in our work (≤0.4 W mm−2).

Single-cell dynamic temperature tracking using SPLIT

To apply SPLIT to dynamic single-cell temperature mapping, we tested a single-layer onion epidermis sample labeled by the 5.6 nm-thick-shell UCNPs (detailed in Supplementary Note 11 and Supplementary Fig. 11). Furthermore, to generate non-repeatable photoluminescent dynamics, the sample was moved across the FOV at a speed of 1.18 mm s−1 by a translation stage. In the 3-second measurement window, the SPLIT system continuously recorded 60 lifetime/temperature maps. Four representative time-integrated images and their corresponding lifetime maps are shown in Fig. 4a–b (see dynamic lifetime mapping in Supplementary Movie 6). Figure 4c shows intensity decay curves from four selected regions with varied intensities in the onion cell sample at 0.05 s. The photoluminescence lifetimes and hence the temperatures remain stable, showing SPLIT’s resilience to spatial intensity variation. We also tracked the time histories of the average emitted fluence and lifetime-indicated temperatures of these four regions during the sample’s translational moving (Fig. 4d). In this measurement time window, the emitted photoluminescence fluences have varied in each selected region. In contrast, the measured temperatures show a small fluctuation of ±0.35 °C, which validates the advantage of PLI thermometry in handling temporal intensity variation.

Fig. 4: Dynamic single-cell temperature mapping using SPLIT.
figure 4

a Representative time-integrated images of a moving onion epidermis cell sample labeled by UCNPs. b Lifetime images corresponding to (a). c Photoluminescence decay profiles at four selected areas [marked by the solid boxes in the first panel of (a)] with varied intensities. d Time histories of averaged fluence and corresponding temperature in the four selected regions during the sample’s translational motion.

Discussion

In summary, we have developed SPLIT for wide-field dynamic temperature sensing in real time. In data acquisition, SPLIT compressively records the photoluminescence emission over a 2D FOV in two views. Then, the dual-view PnP-ADMM algorithm reconstructs spatially resolved intensity decay traces, from which a photoluminescence lifetime distribution and the corresponding temperature map are extracted. Used with core/shell NaGdF4:Er3+,Yb3+/NaGdF4 UCNPs, SPLIT has enabled temperature mapping with high sensitivity for both green and red upconversion emission bands with a 20-µm spatial resolution in a 1.5 × 1.5 mm2 FOV at a video rate of 20 Hz. SPLIT is demonstrated in longitudinal temperature monitoring of a phantom beneath fresh chicken tissue. SPLIT is also applied to dynamic single-cell temperature mapping of a moving single-layer onion epidermis sample.

SPLIT advances the technical frontier of optical instrumentation in PLI. The high parallelism in SPLIT’s data acquisition drastically improves the overall light throughput. The resulting system, featuring single-shot temperature sensing over a 2D FOV, solves the long-standing issue in scanning-based techniques (see Supplementary Note 12 and Supplementary Figs. 1213). In particular, SPLIT improves the measurement accuracy by avoiding artifacts generated from the scanning-induced motion blur and the excitation intensity fluctuation. More importantly, as shown in Fig. 4, SPLIT extends the application scope of PLI to observing non-repeatable 2D temperature dynamics. Its high tunability of imaging speeds also accommodates a variety of UCNPs with a wide lifetime span (from hundreds of nanoseconds to milliseconds). Among existing single-shot 2D ultrafast imaging modalities based on streak cameras, SPLIT is well suited for dynamic PLI of UCNPs in terms of the targeted imaging speed, detection sensitivity, spatial resolution, and cost efficiency (detailed in Supplementary Note 12 and Supplementary Table 1). Finally, the SPLIT system by itself records only the lifetime images; yet, when using UCNPs as contrast agents, those images carry temperature information in situ, where the UCNPs reside. Thus, compared to thermal imaging cameras, SPLIT supplies superior temperature mapping results with higher image contrast and better resilience to background interference (detailed in Supplementary Note 13 and Supplementary Fig. 14).

From the perspective of system design, both the dual-view data acquisition and the PnP-ADMM algorithm support high imaging quality in SPLIT. In particular, View 1 preserves the spatial information in the dynamic scene54. Meanwhile, View 2 retains temporal information by optical streaking via time-to-space conversion. Altogether, both views maximally keep rich spatiotemporal information. In software, the dual-view PnP-ADMM algorithm provides a powerful modular structure, which allows separated optimization of individual sub-optimization problems with an advanced denoising algorithm to generate high-quality image restoration results.

SPLIT offers a versatile PLI temperature-sensing platform. In materials characterization, it could be used in the stress analysis of metal fatigue in turbine blades55. In biomedicine, it could be implemented for accurate sub-cutaneous temperature monitoring for theranostics of skin diseases (e.g., micro-melanoma)56,57. SPLIT’s microscopic temperature mapping ability could also be exploited for the studies of temperature-regulated cellular signaling58. Finally, the operation of SPLIT could be extended to Stokes emission in lanthanide-doped nanoparticles and spectrally resolved temperature mapping. All of these topics are promising research directions in the future.

Methods

Synchronization of the SPLIT system

The optical chopper outputs a transistor-transistor logic (TTL) signal that is synchronized with the generated optical pulses. This TTL signal is input to a delay generator (Stanford Research Systems, DG 645), which then generates three synchronized TTL signals at 20 Hz. The first two signals are used to trigger the 3-ms exposure of the EMCCD and CMOS cameras. The last one is used to trigger a function generator (Rigol, DG1022Z) that outputs a 20-Hz sinusoidal waveform under the external burst mode to control the rotation of the galvanometer scanner (GS).

Calculation of SPLIT’s key parameters

The GS, placed at the Fourier plane of the 4\(f\) imaging system consisting of lenses L4 and L5 (Fig. 1), deflects temporal information to different spatial positions. Rotating during the data acquisition, the GS changes the reflection angles of the spatial frequency spectra of individual frames with different time-of-arrival. After the Fourier transformation by Lens 5, this angular difference is converted to the lateral shift in space on the EMCCD camera, which results in temporal shearing. An illustration with a simple example is provided in Supplementary Fig. 15.

The imaging speed is determined by the data acquisition for View 2. In particular, the reconstructed movie has a frame rate of59

$${{{{{\rm{r}}}}}}=\frac{{\gamma }_{{{{{{\rm{a}}}}}}}{V}_{{{{{{\rm{g}}}}}}}{f}_{5}}{{t}_{{{{{{\rm{s}}}}}}}d}.$$
(4)

Here \({V}_{{{{{{\rm{g}}}}}}}\) is the voltage added onto the GS. \({\gamma }_{{{{{{\rm{a}}}}}}}\) is a constant that links \({V}_{{{{{{\rm{g}}}}}}}\) with GS’s deflection angle with the consideration of the input waveform. \({f}_{5}\)=100 mm is the focal length of lens L5, \({t}_{{{{{{\rm{s}}}}}}}=50{{{{{\rm{ms}}}}}}\) is the period of the sinusoidal voltage waveform added to the GS, and \(d\) = 13 µm is the EMCCD sensor’s pixel size. In this work, we used the voltage from \({V}_{{{{{{\rm{g}}}}}}}=\) 0.2–1.7 V. The imaging speed of SPLIT ranged from 4 to 33 kfps. In addition, we used \({t}_{{{{{{\rm{e}}}}}}}=3\,{{{{{\rm{ms}}}}}}\) as the exposure time of the EMCCD and CMOS cameras. The sequence depth, \({N}_{t}\), is determined by

$${N}_{t}={{{{{\rm{r}}}}}}{t}_{{{{{{\rm{e}}}}}}}.$$
(5)

In the experiments presented in this work, \({N}_{t}\) ranged from 12 to 100 frames.

Data availability

All data needed to evaluate the findings of this study are present in the paper and Supplementary Information. The raw data for Fig. 2 can be downloaded via the following link: https://figshare.com/articles/figure/SPLIT_Fig2/16703413. All other raw data are available from the corresponding authors upon reasonable request.

Code availability

The image reconstruction algorithm is described in detail in Supplementary Information. The custom computer code is not publicly available because it is proprietary and included in a patent application.

References

  1. 1.

    Inada, N., Fukuda, N., Hayashi, T. & Uchiyama, S. Temperature imaging using a cationic linear fluorescent polymeric thermometer and fluorescence lifetime imaging microscopy. Nat. Protoc. 14, 1293–1321 (2019).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  2. 2.

    Wood, M. & Ozanyan, K. Simultaneous temperature, concentration, and pressure imaging of water vapor in a turbine engine. IEEE Sens. J. 15, 545–551 (2015).

    ADS  Article  Google Scholar 

  3. 3.

    Obermayer, D. & Kappe, C. On the importance of simultaneous infrared/fiber-optic temperature monitoring in the microwave-assisted synthesis of ionic liquids. Org. Biomol. Chem. 8, 114–121 (2010).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  4. 4.

    Zhang, Z., Wang, J. & Chen, C. Near‐infrared light‐mediated nanoplatforms for cancer thermo‐chemotherapy and optical imaging. Adv. Mater. 25, 3869–3880 (2013).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  5. 5.

    Chen, Z. et al. Phosphorescent polymeric thermometers for in vitro and in vivo temperature sensing with minimized background interference. Adv. Funct. Mater. 26, 4386–4396 (2016).

    CAS  Article  Google Scholar 

  6. 6.

    Jaque, D. & Vetrone, F. Luminescence nanothermometry. Nanoscale 4, 4301–4326 (2012).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

  7. 7.

    Datta, R. et al. Fluorescence lifetime imaging microscopy: fundamentals and advances in instrumentation, analysis, and applications. J. Biomed. Opt. 25, 071203 (2020).

    ADS  CAS  PubMed Central  Article  Google Scholar 

  8. 8.

    Kurokawa, H. et al. High resolution imaging of intracellular oxygen concentration by phosphorescence lifetime. Sci. Rep. 5, 1–13 (2015).

    Article  CAS  Google Scholar 

  9. 9.

    Childs, P., Greenwood, J. & Long, C. Review of temperature measurement. Rev. Sci. Instrum. 71, 2959–2978 (2000).

    ADS  CAS  Article  Google Scholar 

  10. 10.

    Gao, L. et al. Single-cell photoacoustic thermometry. J. Biomed. Opt. 18, 026003 (2013).

    ADS  PubMed Central  Article  CAS  Google Scholar 

  11. 11.

    Snyder, W., Wan, Z., Zhang, Y. & Feng, Y. Classification-based emissivity for land surface temperature measurement from space. Int. J. Remote Sens. 19, 2753–2774 (1998).

    Article  Google Scholar 

  12. 12.

    Suhling, K. et al. Fluorescence lifetime imaging (FLIM): basic concepts and some recent developments. Med. Photonics 27, 3–40 (2015).

    Article  Google Scholar 

  13. 13.

    Chelushkin, P. & Tunik, S. Phosphorescence lifetime imaging (PLIM): state of the art and perspectives. Chelushkin, P. & Tunik, S. In Progress in Photon Science. (eds Yamanouchi, K., Tunik, S. and Makarov, V.) (Springer Nature, 2019).

  14. 14.

    Labrador-Páez, L. et al. Reliability of rare-earth-doped infrared luminescent nanothermometers. Nanoscale 10, 22319–22328 (2018).

    PubMed  Article  PubMed Central  Google Scholar 

  15. 15.

    Pickel, A. et al. Apparent self-heating of individual upconverting nanoparticle thermometers. Nat. Commun. 9, 1–12 (2018).

    CAS  Article  Google Scholar 

  16. 16.

    Shen, Y. et al. In vivo spectral distortions of infrared luminescent nanothermometers compromise their reliability. ACS nano 14, 4122–4133 (2020).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  17. 17.

    Becker, W. Fluorescence lifetime imaging–techniques and applications. J. Microsc. 247, 119–136 (2012).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  18. 18.

    Bolek, P. et al. Ga-modified YAG: Pr3+ dual-mode tunable luminescence thermometers. Chem. Eng. J. 421, 129764 (2021).

    CAS  Article  Google Scholar 

  19. 19.

    Gao, H. et al. A simple yet effective AIE-based fluorescent nano-thermometer for temperature mapping in living cells using fluorescence lifetime imaging microscopy. Nanoscale Horiz. 5, 488–494 (2020).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

  20. 20.

    Maciejewska, K., Bednarkiewicz, A. & Marciniak, L. NIR Luminescence lifetime nanothermometry based on phonon assisted Yb3+-Nd3+ energy transfer. Nanoscale Adv. 3, 4918–4925 (2021).

    ADS  CAS  Article  Google Scholar 

  21. 21.

    Zhang, H. et al. Dual-emissive phosphorescent polymer probe for accurate temperature sensing in living cells and zebrafish using ratiometric and phosphorescence lifetime imaging microscopy. ACS Appl. Mater. Interfaces 10, 17542–17550 (2018).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  22. 22.

    Tan, M. et al. Accurate in vivo nanothermometry through NIR‐II lanthanide luminescence lifetime. Small 16, 2004118 (2020).

    CAS  Article  Google Scholar 

  23. 23.

    Allison, S. W. et al. Nanoscale thermometry via the fluorescence of YAG: Ce phosphor particles: measurements from 7 to 77 °C. Nanotechnology 14, 859 (2003).

    ADS  CAS  Article  Google Scholar 

  24. 24.

    Benninger, R. et al. Quantitative 3D mapping of fluidic temperatures within microchannel networks using fluorescence lifetime imaging. Anal. Chem. 78, 2272–2278 (2006).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  25. 25.

    Graham, E. M. et al. Quantitative mapping of aqueous microfluidic temperature with sub-degree resolution using fluorescence lifetime imaging microscopy. Lab Chip 10, 1267–1273 (2010).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  26. 26.

    Schlegel, G. et al. Fluorescence decay time of single semiconductor nanocrystals. Phys. Rev. Lett. 88, 137401 (2002).

    ADS  PubMed  Article  CAS  PubMed Central  Google Scholar 

  27. 27.

    Auzel, F. Upconversion and anti-stokes processes with f and d ions in solids. Chem. Rev. 104, 139–174 (2004).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  28. 28.

    Skripka, A. et al. Spectral characterization of LiYbF4 upconverting nanoparticles. Nanoscale 12, 17545–17554 (2020).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  29. 29.

    Jacques, S. Optical properties of biological tissues: a review. Phys. Med. Biol. 58, R37 (2013).

    ADS  PubMed  Article  PubMed Central  Google Scholar 

  30. 30.

    Vetrone, F. et al. Temperature sensing using fluorescent nanothermometers. ACS Nano 4, 3254–3258 (2010).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  31. 31.

    Brites, D. S. C. et al. Thermometry at the nanoscale. Nanoscale 4, 4799–4829 (2012).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

  32. 32.

    Rostami, I., Alanagh, H., Hu, Z. & Shahmoradian, S. H. Breakthroughs in medicine and bioimaging with up-conversion nanoparticles. Int. J. Nanomed. 14, 7759 (2019).

    CAS  Article  Google Scholar 

  33. 33.

    Zhou, J., Del Rosal, B., Jaque, D., Uchiyama, S. & Jin, D. Advances and challenges for fluorescence nanothermometry. Nat. Methods 17, 967–980 (2020).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  34. 34.

    Qin, H. et al. Tuning the upconversion photoluminescence lifetimes of NaYF4: Yb3+, Er3+ through lanthanide Gd3+ doping. Sci. Rep. 8, 12683 (2018).

    ADS  PubMed  PubMed Central  Article  CAS  Google Scholar 

  35. 35.

    Howard, S., Straub, A., Horton, N., Kobat, D. & Xu, C. Frequency-multiplexed in vivo multiphoton phosphorescence lifetime microscopy. Nat. Photonics 7, 33–37 (2013).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

  36. 36.

    Suhling, K. et al. Wide-field TCSPC-based fluorescence lifetime imaging (FLIM). Microsc. SPIE proc. 9858, 98580J (2016).

    Article  Google Scholar 

  37. 37.

    Hirvonen, L., Fisher, M., Suhling, K. & Nomerotski, A. Photon counting phosphorescence lifetime imaging with TimepixCam. Rev. Sci. Instrum. 88, 013104 (2017).

    ADS  PubMed  Article  CAS  PubMed Central  Google Scholar 

  38. 38.

    Sen, R. et al. New luminescence lifetime macro-imager based on a Tpx3Cam optical camera. Biomed. Opt. Express 11, 77–88 (2020).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  39. 39.

    Franke, R. & Holst, G. Frequency-domain fluorescence lifetime imaging system (pco. flim) based on a in-pixel dual tap control CMOS image sensor. SPIE proc. 9328, 93281K (2015).

    ADS  Article  Google Scholar 

  40. 40.

    Xiong, B. & Fang, Q. Luminescence lifetime imaging using a cellphone camera with an electronic rolling shutter. Opt. Lett. 45, 81–84 (2020).

    ADS  CAS  Article  Google Scholar 

  41. 41.

    Liang, J. Punching holes in light: recent progress in single-shot coded-aperture optical imaging. Rep. Prog. Phys. 83, 116101 (2020).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

  42. 42.

    Liang, J. et al. Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse. Sci. Adv. 3, e1601814 (2017).

    ADS  PubMed  PubMed Central  Article  Google Scholar 

  43. 43.

    Liu, Y., Yuan, X., Suo, J., Brady, D. J. & Dai, Q. Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell. 41, 2990–3006 (2018).

    PubMed  Article  PubMed Central  Google Scholar 

  44. 44.

    Yuan, X., Liu, Y., Suo, J. & Dai, Q. Plug-and-play algorithms for large-scale snapshot compressive imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1447–1457 (2020).

  45. 45.

    Chan, S., Wang, X. & Elgendy, O. Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Trans. Comput. Imag. 3, 84–98 (2016).

    MathSciNet  Article  Google Scholar 

  46. 46.

    Yuan, X. et al. Compressive hyperspectral imaging with side information. IEEE J. Sel. Top. Signal Process 9, 964–976 (2015).

    ADS  Article  Google Scholar 

  47. 47.

    Liang, J., Zhu, L. & Wang, L. Single-shot real-time femtosecond imaging of temporal focusing. Light Sci. Appl. 7, 42 (2018).

    ADS  PubMed  PubMed Central  Article  CAS  Google Scholar 

  48. 48.

    Wang, P., Liang, J. & Wang, L. Single-shot ultrafast imaging attaining 70 trillion frames per second. Nat. Commun. 11, 2091 (2020).

    ADS  CAS  PubMed  PubMed Central  Article  Google Scholar 

  49. 49.

    Liang, J. et al. Single-shot stereo-polarimetric compressed ultrafast photography for light-speed observation of high-dimensional optical transients with picosecond resolution. Nat. Commun. 11, 5252 (2020).

    ADS  CAS  PubMed  PubMed Central  Article  Google Scholar 

  50. 50.

    May, P. & Berry, M. Tutorial on the acquisition, analysis, and interpretation of upconversion luminescence data. Methods Appl. Fluoresc. 7, 023001 (2019).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

  51. 51.

    Zhang, H. et al. Dual-emissive phosphorescent polymer probe for accurate temperature sensing in living cells and zebrafish using ratiometric and phosphorescence lifetime imaging microscopy. ACS Appl. Mater. Interfaces 10, 17542–17550 (2018).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  52. 52.

    Dos Santos, P., De Araujo, M., Gouveia-Neto, A., Medeiros Neto, J. & Sombra, A. Optical temperature sensing using upconversion fluorescence emission in Er 3+/Yb 3+-codoped chalcogenide glass. Appl. Phys. Lett. 73, 578–580 (1998).

    ADS  Article  Google Scholar 

  53. 53.

    Miller, M. & Wright, J. Multiphonon and energy transfer relaxation in charge compensated crystals. J. Chem. Phys. 71, 324–338 (1979).

    ADS  CAS  Article  Google Scholar 

  54. 54.

    Liu, X., Zhang, S., Yurtsever, A. & Liang, J. Single-shot real-time sub-nanosecond electron imaging aided by compressed sensing: analytical modeling and simulation. Micron 117, 47–54 (2019).

    PubMed  Article  PubMed Central  Google Scholar 

  55. 55.

    Wang, R. et al. Thermomechanical fatigue experiment and failure analysis on a nickel-based superalloy turbine blade. Eng. Fail. Anal. 102, 35–45 (2019).

    CAS  Article  Google Scholar 

  56. 56.

    Jung, H. et al. Organic molecule-based photothermal agents: an expanding photothermal therapy universe. Chem. Soc. Rev. 47, 2280–2297 (2018).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  57. 57.

    Shen, Y. et al. Ag2S nanoheaters with multiparameter sensing for reliable thermal feedback during in vivo tumor therapy. Adv. Funct. Mater. 30, 2002730 (2020).

    CAS  Article  Google Scholar 

  58. 58.

    Wang, C. et al. Determining intracellular temperature at single-cell level by a novel thermocouple method. Cell Res. 21, 1517–1519 (2011).

    PubMed  PubMed Central  Article  Google Scholar 

  59. 59.

    Liu, X., Liu, J., Jiang, C., Vetrone, F. & Liang, J. Single-shot compressed optical-streaking ultra-high-speed photography. Opt. Lett. 44, 1387–1390 (2019).

    ADS  CAS  PubMed  Article  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors thank Professor Aycan Yurtsever and Wanting He for experimental assistance and fruitful discussion. Natural Sciences and Engineering Research Council of Canada (RGPIN-2017-05959, RGPAS-2017-507845, I2IPJ-555593-20, RGPIN-2018-06217, RGPAS-2018-522650); Canada Foundation for Innovation and Ministère de l'Économie et de l’Innovation du Québec (37146); Canadian Cancer Society (707056); New Frontier in Research Fund (NFRFE-2020-00267); Fonds de Recherche du Québec–Nature et Technologies (2019-NC-252960); Fonds de Recherche du Québec–Santé (267406, 280229).

Author information

Affiliations

Authors

Contributions

X.L. designed and built the system, conducted the experiments, developed the reconstruction algorithm, and analyzed the data. A.S. prepared the UCNPs, conducted some experiments, and analyzed the data. Y.L. contributed to the algorithm development. C.J. and J. Liu conducted some experiments. F.V. and J. Liang initiated the project. J. Liang proposed the concept, contributed to experimental design, and supervised the project. All authors wrote and revised the manuscript.

Corresponding authors

Correspondence to Fiorenzo Vetrone or Jinyang Liang.

Ethics declarations

Competing interests

The authors disclose the following patent applications: WO 2020/154806 A1 (J. Liang, F.V., and X.L.) and US Provisional 63/260,511 (J. Liang, F.V., X.L., and A.S.).

Additional information

Peer review information Nature Communications thanks the anonymous reviewers for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Skripka, A., Lai, Y. et al. Fast wide-field upconversion luminescence lifetime thermometry enabled by single-shot compressed ultrahigh-speed imaging. Nat Commun 12, 6401 (2021). https://doi.org/10.1038/s41467-021-26701-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41467-021-26701-1

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing