Abstract
Efforts to map atomicscale chemistry at low doses with minimal noise using electron microscopes are fundamentally limited by inelastic interactions. Here, fused multimodal electron microscopy offers high signaltonoise ratio (SNR) recovery of material chemistry at nano and atomicresolution by coupling correlated information encoded within both elastic scattering (highangle annular darkfield (HAADF)) and inelastic spectroscopic signals (electron energy loss (EELS) or energydispersive xray (EDX)). By linking these simultaneously acquired signals, or modalities, the chemical distribution within nanomaterials can be imaged at significantly lower doses with existing detector hardware. In many cases, the dose requirements can be reduced by over one order of magnitude. This high SNR recovery of chemistry is tested against simulated and experimental atomic resolution data of heterogeneous nanomaterials.
Similar content being viewed by others
Introduction
Modern scanning transmission electron microscopes (STEM) can focus subangstrom electron beams on and between atoms to quantify structure and chemistry in real space from elastic and inelastic scattering processes. The chemical composition of specimens is revealed by spectroscopic techniques produced from inelastic interactions in the form of energydispersive Xrays (EDX)^{1,2} or electron energy loss (EELS)^{3,4}. Unfortunately, highresolution chemical imaging requires high doses (e.g., >10^{6} e/Å^{2}) that often exceed the specimen limits—resulting in chemical maps that are noisy or missing entirely^{5,6}. Substantial effort and cost to improve detector hardware have brought the field closer to the measurement limits set by inelastic processes^{7,8}. Direct interpretation of atomic structure at higherSNR is provided by elastically scattered electrons collected in a highangle annular darkfield detector (HAADF); however, this signal underdescribes the chemistry^{9}. Reaching the lowest doses at the highest SNR ultimately requires fusing both elastic and inelastic scattering modalities.
Currently, detector signals—such as HAADF and EDX/EELS—are analyzed separately for insight into structural, chemical, or electronic properties^{10}. Correlative imaging disregards shared information between structure and chemistry and misses opportunities to recover useful information. Data fusion, popularized in satellite imaging, goes further than correlation by linking the separate signals to reconstruct new information and improve measurement accuracy^{11,12,13}. Successful data fusion designs an analytical model that faithfully represents the relationship between modalities, and yields a meaningful combination without imposing any artificial connections^{14}.
Here we introduce fused multimodal electron microscopy, a technique offering high SNR recovery of nanomaterial chemistry by linking correlated information encoded within both HAADF and EDX/EELS. We recover chemical maps by reformulating the inverse problem as a nonlinear optimization that seeks solutions that accurately match the actual chemical distribution in a material. Our approach substantially improves SNRs for chemical maps, often around 300–500%, and can reduce doses over one order of magnitude while remaining consistent with original measurements. We demonstrate EDX/EELS datasets at subnanometer and atomic resolution. Moreover, fused multimodal electron microscopy recovers a specimen’s relative concentration, allowing researchers to measure local stoichiometry with less than 15% error without any knowledge of the inelastic crosssections. Convergence and uncertainty estimates are identified along with simulations that provide a groundtruth assessment of when and how this approach can fail.
Results
Principles of multimodal electron microscopy
Fused multimodal electron microscopy recovers chemical maps by solving an optimization problem seeking a solution that strongly correlates with (1) the HAADF modality containing high SNR, (2) the chemically sensitive spectroscopic modality (EELS and/or EDX), and (3) encourages sparsity in the gradient domain producing solutions with reduced spatial variation. The overall optimization function is as follows:
where λ are regularization parameters, b_{H} is the measured HAADF, b_{i} and x_{i} are the measured and reconstructed chemical maps for element i, ε herein prevents log(0) issues but can also account for background, \({{\mathrm{log}}}\,\) is applied elementwise to its arguments, superscript T denotes vector transpose, and 1 denotes the vector of n_{x}n_{y} ones, where n_{x} × n_{y} is the image size.
The three terms in Eq. (1) define our multimodal approach to surpass traditional dose limits for chemical imaging. First, we assume a forward model where the simultaneous HAADF is a linear combination of elemental distributions (\({{{{\boldsymbol{x}}}}}_{i}^{\gamma }\) where γ ∈ [1.4, 2]). The incoherent linear imaging approximation for elastic scattering scales with atomic number as \({Z}_{i}^{\gamma }\) where γ is typically around 1.7^{15,16,17}. This γ is bounded between 2 for Rutherford scattering from bare nuclear potentials to 4/3 as described by Lenz–Wentzel expressions for electrons experiencing a screened Coulombic potential^{18,19}. Second, we ensure the recovered signals maintain a high degree of data fidelity with the initial measurements by using maximum negative loglikelihood for spectroscopic measurements dominated by lowcount Poisson statistics^{20,21}. In a higher count regime, this term can be substituted with a simple leastsquares error. Lastly, we utilize channelwise total variation (TV) regularization to enforce a sparse gradient magnitude, which reduces noise by promoting image smoothness while preserving sharp features^{22}. This sparsity constraint, popularized by the field of compressed sensing (CS), is powerful yet minimal prior to recovering structured data^{23,24}. When implementing, each of these three terms can and should be weighted by appropriately selected coefficients that balance their contributions. All three terms are necessary for accurate recovery (Supplementary Fig. 1).
HighSNR recovery of nanomaterial chemistry
Figure 1 demonstrates highSNR recovery for EDX signals of commercial cobalt sulfide (CoS) nanocatalysts for oxygenreduction applications—a unique class with the highest activity among nonprecious metals^{25}. Figure 1a illustrates the model that links the two modalities (EDX and HAADF) simultaneously collected in the electron microscope. The low detection rate for characteristic Xrays is due to minimal emission (e.g., over 50% for Z > 32 and below 2% for Z < 11) and collection yield (<9%)^{26}. For highresolution EDX, the low count rate yields a sparse chemical image dominated by shot noise (Fig. 1b). However, noise in the fused multimodal chemical map is virtually eliminated (Fig. 1d) and recovers chemical structure without a loss of resolution—including the nanoparticle core and oxide shell interface. The chemical maps produced by fused multimodal EM quantitatively agree with the expected stoichiometry—the specimen core contains a relative concentration of 39 ± 1.6%, 42 ± 2.5%, and 13 ± 2.4% and exterior shell composition of 26 ± 2.8%, 11 ± 2.0%, 54 ± 1.3% for Co, S, O, respectively. The dose for this dataset was ~10^{5} e Å^{−2} and a 0.7 sr EDX detector was used; however, these quantitative estimates remained consistent when the dose was reduced to ~10^{4} e Å^{−2}.
Fused multimodal electron microscopy accurately recovers chemical structure down to atomic length scales—demonstrated here for EELS spectroscopic signals. EELSderived chemical maps for Co_{3−x}Mn_{x}O_{4} (x = 1.49) highperforming supercapacitor nanoparticles are substantially improved by fused multimodal electron microscopy in Fig. 2. This composite Co–Mn oxide was designed to achieve a synergy between cobalt oxide’s high specific capacitance and manganese oxide’s long life cycle^{27,28}. While the Co_{3−x}Mn_{x}O_{4} nanoparticle appears chemically homogeneous in the HAADF projection image along the [100] direction (Fig. 2c), core–shell distinctions are hinted at in the raw EELS maps (Fig. 2b). Specifically, these nanoparticles contain an Mnrich center with a Co shell and homogeneous distribution of O. However the raw EELS maps are excessively degraded by noise, preventing analysis beyond the rough assessment of specimen morphology. The multimodal reconstructions (Fig. 2d) confirm the crystalline Corich shell and map the Co/Mn interface in greater detail (Fig. 2e). In the presence of cobalt and manganese, the HAADF image lacks noticeable contrast from oxygen; the resulting oxygen map lacks detail and benefits mostly from regularization.
Figure 3 exhibits fused multimodal electron microscopy at an atomic resolution on copper–sulfur heterostructured nanocrystals with zinc sulfide caps with potential applications in photovoltaic devices or battery electrodes^{29}. The copper sulfide properties are sensitive to the Cu–S stoichiometry and crystal structure at the interface between ZnS and Cu_{0.64}S_{0.36}. Figure 3 shows highresolution HAADF and EELS characterization of a heterostructure Cu_{0.64}S_{0.36}–ZnS interface. Fused multimodal electron microscopy maps out the atomically sharp Cu_{0.64}S_{0.36}–ZnS interface and reveals step edges between the two layers. The labeled points on the RGB chemical overlay (Fig. 3d) show the chemical ratios produced by multimodal EM for the Cu_{0.64}S_{0.36} and ZnS regions—values which are consistent with the reported growth conditions. Figure 3e shows the algorithm convergence for each of the three terms in the optimization function (Eq. (1))—smooth and asymptotic decay is an indicator of reliable reconstruction. Refer to Supplementary Fig. 2 for an additional demonstration at the atomic scale on an ordered manganite system.
Fused multimodal imaging of Fe and Pt distributions from inelastic multislice simulations (Fig. 4) provide ground truth solutions to validate recovery at atomic resolution under multiple scattering conditions of an onaxis ~8 nm nanoparticle. Here, we applied Poisson noise (Fig. 4b) containing electron doses of ~10^{9} e Å^{−2}, to produce chemical maps with noise levels resembling experimental atomicresolution EELS datasets (SNR ≃ 5). We estimated SNR improvements by measuring peakSNR for the noisy and recovered chemical maps^{30}. Qualitatively, the recovered chemical distributions (Fig. 4c) match the original images. Figure 4d illustrates the agreement of the line profiles as the atom column positions and relative peak intensities between the ground truth and multimodal reconstruction are almost identical.
Simulating EELS chemical maps is computationally demanding as every inelastic scattering event requires propagation of an additional wavefunction^{31,32}—scaling faster than the cube of the number of beams, \(O({N}^{3}{{\mathrm{log}}}\,N)\). Inelastic transition potentials of interest (in this case the L_{2,3} Fe and M_{4,5} Pt edges) were calculated from density function theory (see the “Methods” section). Long computation times (nearly 4000 corehours) result from a large number of outgoing scattering channels corresponding to the many possible excitations in a sample. For this reason, there is little precedence for inelastic image simulations. We relaxed the runtime by utilizing the PRISM STEMEELS approximation, achieving over a tenfold speedup (see the “Methods” section)^{33}. Future work may explore the effects of smaller ADF collection angles with increased coherence lengths and crystallographic contrast^{15,34}, or thicker specimens where electron channeling becomes more concerning^{35,36}.
Quantifying chemical concentration
Fused multimodal electron microscopy can produce stoichiometrically meaningful chemical maps without specific knowledge of inelastic crosssections. Here, the ratio of pixel values in the reconstructed maps quantifies elemental concentration. We demonstrate quantifiable chemistry on experimental metal oxide thin films with known stoichiometry: NiO^{37} and ZrO_{2}. A histogram of intensities from the recovered chemical maps is fitted with Gaussian distributions to determine the average concentration. The recovered pixel values highlighted in Fig. 5 followed a single Gaussian distribution where the Zr and Ni concentrations are centered at about 35 ± 5.8% and 50 ± 2.9%. In both cases, the average Ni and Zr relative concentrations are approximately equivalent to the expected ratio from the crystal stoichiometry: 33% and 50%. The CoS nanoparticle in Fig. 1 follows a bimodal distribution for the core and shell phases (Supplementary Fig. 5). We found measuring stoichiometry is robust across a range of γ values close to 1.7. In cases where γ is far off (e.g., γ = 1.0), the quantification is systematically incorrect (Supplementary Fig. 6).
We further validate stoichiometric recovery on a synthetic gallium oxide crystal (Fig. 5) where two overlapping Ga and O thin films of equal thickness have a stoichiometery of Ga_{2}O_{3}. The simulated HAADF signal is proportional to \({\sum }_{i}{({{{{\boldsymbol{x}}}}}_{i}{Z}_{i})}^{\gamma }\) where x_{i} is the concentration for element i and Z_{i} is the atomic number. As shown by the histogram, the simulated results agree strongly with the prior knowledge and successfully recover the relative Ga concentration. The Gaussian distribution is centered about 40 ± 0.4% when the ground truth is 40%. The inset shows convergence plots.
We estimate a stoichiometric error of <15% for most materials based on the relative concentration’s standard deviation (±7%) added in quadrature with the variation of solutions (±6%). Although the algorithm shows stable convergence, the overall quantitative conclusions are slightly sensitive to the selection of hyperparameters. We estimate incorrect selection of hyperparameters could result in variation of roughly ±6% from the correct prediction in stoichiometry even when the algorithm converges (convergence shown in Supplementary Figs. 8 and 9). This error is comparable to estimating chemical concentrations directly from EELS/EDX spectral maps from the ratio of scattering crosssection against coreloss intensity^{38}. However, traditional approaches require accurate knowledge of all experimental parameters (e.g., beam energy, specimenthickness, collection angles) and accurate calculation of the inelastic crosssection typically to provide errors roughly between 5% and 10%^{39}.
Influence of electron dose
To better understand the accuracy of fused multimodal electron microscopy at low doses, we performed a quantitative study of normalized rootmeansquare error (RMSE) concentrations for a simulated 3D core–shell nanoparticle (CoS core, CoO shell). Figure 6 shows the fused multimodal reconstruction accuracy across a wide range of HAADF and chemical SNR. The simulated projection images were generated by a simple linear incoherent imaging model of the 3D chemical compositions highlighted in Fig. 6d—here the probe’s depth of focus is much larger than the object. Random Poisson noise corresponding to different electron dose levels was applied to vary the SNR across each pixel.
Overall, the RMSE simulation map (Fig. 6a) shows the core–shell nanoparticle chemical maps are accurately recovered at low doses (HAADF SNR ≳ 4 and chemical SNR ≳ 2); however, they become less accurate at extremely low doses. The RMSE map for multimodal reconstruction shows a predictably continuous degradation in recovery as signals diminish. The degraded and reconstructed chemical maps for various noise levels are highlighted in Fig. 6b. The Co map closely mirrors the Zcontrast observed in HAADF (not shown) simply because it is the heaviest element present. Usually, researchers will perform spectroscopic experiments in the top right corner of Fig. 6a (e.g., HAADF SNR > 20, chemical SNR > 3), which for this simulation, provides accurate recovery.
In actual experiments, the ground truth is unknown and RMSE cannot be calculated to assess fused multimodal electron microscopy. However, we can estimate accuracy by calculating an average standard error of our recovered image from the Hessian of our model (see the “Methods” section). The standard error reflects uncertainty at each pixel in a recovered chemical map by quantifying the neighborhood size for similar solutions (Supplementary Fig. 10). The average standard error across all pixels in a fused multimodal image provides a single value metric of the reconstruction accuracy (see the “Methods” section). Figure 6c shows that RMSE and average standard error correlate, especially at higher doses (SNR > 10).
Discussion
While this paper highlights the advantages of multimodal electron microscopy, the technique is not a blackbox solution. Step sizes for convergence and weights on the terms in the cost function (Eq. (1)) must be reasonably selected. This manuscript illustrates approaches to assess the validity of concentration measurements using confidence estimation demonstrated across several simulated and experimental material classes. Standard spectroscopic preprocessing methods become ever more critical in combination with multimodal fusion. Improper background subtraction of EELS spectra or overlapping characteristic Xray peaks that normally cause inaccurate stoichiometric quantification also reduce the accuracy of fused multimodal imaging.
Fused multimodal electron microscopy offers little advantage in recovering chemical maps for elements with insignificant contrast in the HAADF modality. This property is limiting for analyzing specimens with lowZ elements in the presence of heavy elements (e.g., oxygen and lutetium). Future efforts could resolve this challenge by incorporating an additional complementary elastic imaging mode where light elements are visible, such as an annular brightfield (ABF)^{40}. However, in some instances, fused multimodal electron microscopy may recover useful information for underdetermined chemical signals. For example, in a Bi_{0.35}Sr_{0.18}Ca_{0.47}MnO_{3} (BSCMO) system^{41}, only the Ca, Mn, and O EELS maps were obtained, yet multimodality remarkably improves the SNR of measured maps despite missing two elements (Supplementary Fig. 2).
Although fused multimodal chemical mapping appears quite robust at nanometer or subnanometer resolution, we found atomicresolution reconstructions can be challenged by spurious atom artifacts which require attention. However, this is easily remedied by downsampling to frequencies below the first Bragg peaks and analyzing a lower resolution chemical map. Alternatively, recovery with minimal spurious atom artifacts is achieved when lower resolution reconstructions are used as an initial guess (Supplementary Fig. 11).
In summary, we present a modeldriven data fusion algorithm that substantially improves the quality of electron microscopy spectroscopic maps at nanometer to atomic resolutions by using both elastic and inelastic signals. From these signals or modalities, each atom’s chemical identity and coordination provides essential information about the performance of nanomaterials across a wide range of applications from clean energy, batteries, and optoelectronics, among many others. In both synthetic and experimental datasets, multimodal electron microscopy shows quantitatively accurate chemical maps with values that reflect stoichiometry. This approach not only improves SNR but opens a pathway for lowdose chemical imaging of radiationsensitive materials. Although demonstrated herein for common STEM detectors (HAADF, EDX, and EELS), this approach can be extended to many other modalities—including pixel array detectors, annular bright field, ptychography, lowloss EELS, etc. One can imagine a future where all scattered and emitted signals in an electron microscope are collected and fused for maximally efficiently atomic characterization of matter.
Methods
Electron microscopy
Simultaneously acquired EELS and HAADF datasets were collected on a fifthorder aberrationcorrected Nion UltraSTEM microscope operated at 100 keV with a probe semiangle of roughly 30 mrad and collection semiangle of 80–240 and 0–60 mrad for HAADF and EELS, respectively. Both specimens were imaged at 30 pA, for a dwell time of 10 ms (Fig. 3) and 15 ms (Fig. 2) receiving a total dose of 3.25 × 10^{4} and 7.39 × 10^{4} e/Å^{2}. The EELS signals were obtained by integration over the core loss edges, all of which were done after background subtraction. The background EELS spectra were modeled using a linear combination of power laws implemented using the opensource Cornell Spectrum Imager software^{6}.
Simultaneously acquired EDX and HAADF datasets were collected on a Thermo Fisher Scientific Titan Themis G2 at 200 keV with a probe semiangle of roughly 25 mrad, HAADF collection semiangle of 73–200 mrad, and 0.7 sr EDX solid angle. The CoS specimen was imaged at 100 pA and 40 μs dwell time for 50 frames receiving a total dose of ~2 × 10^{5} e/Å^{2}. The initial chemical distributions were generated from EDX maps using commercial Velox software that produced initial net count estimates (however atomic percent estimates are also suitable).
Fused multimodal recovery
Here, fused multimodal electron microscopy is framed as an inverse problem expressed in the following form: \(\hat{{{{\boldsymbol{x}}}}}=\arg \mathop{\min }\limits_{{{{\boldsymbol{x}}}}\ge 0}{{{\Psi }}}_{1}({{{\boldsymbol{x}}}})+{\lambda }_{1}{{{\Psi }}}_{2}({{{\boldsymbol{x}}}})+{\lambda }_{2}{{{\rm{TV}}}}({{{\boldsymbol{x}}}})\), where \(\hat{{{{\boldsymbol{x}}}}}\) is the final reconstruction, and the three terms are described in the main manuscript (Eq. (1)). When implementing an algorithm to solve this problem, we concatenate the multielement spectral variables (x_{i}, b_{i}) as a single vector: \({{{\boldsymbol{x}}}},\,{{{\boldsymbol{b}}}}\,\in \,{{\mathbb{R}}}^{{n}_{x}{n}_{y}{n}_{i}}\), where n_{i} denotes the total number of reconstructed elements.
The optimization problem is solved by a combination of gradient descent with total variation regularization. We solve this cost function by descending along with the negative gradient directions for the first two terms and subsequently evaluate the isotropic TV proximal operator to denoise the chemical maps^{42}. The gradients of the first two terms are:
where ⊘ denotes pointwise division. Here, the first term in the cost function, relating the elastic and inelastic modalities, has been equivalently rewritten as \({{{\Psi }}}_{1}=\frac{1}{2}{\left\Vert {{{{\boldsymbol{b}}}}}_{\mathrm {{H}}}{{{\boldsymbol{A}}}}{{{{\boldsymbol{x}}}}}^{\gamma }\right\Vert }_{2}^{2}\), where \({{{\boldsymbol{A}}}}\,\in \,{{\mathbb{R}}}^{{n}_{x}{n}_{y}\times {n}_{x}{n}_{y}{n}_{i}}\) expresses the summation of all elements as matrix–vector multiplication. Evaluation for the TV proximal operator is in itself another iterative algorithm. In addition, we impose a nonnegativity constraint since negative concentrations are unrealistic. We initialize the first iterate with the measured data (\({{{{\boldsymbol{x}}}}}_{i}^{0}={{{{\boldsymbol{b}}}}}_{i}\)), an ideal starting point as it is a local minima for Ψ_{2}.
The inverse of the Lipschitz constant (1/L) is an upper bound of the step size that can theoretically guarantee convergence. From Lipschitz continuity, we estimated the step size for the model term’s gradient (∇ Ψ_{1}) as 1/\({L}_{\nabla {{{\Psi }}}_{1}}\le 1/\left(\parallel {{{\boldsymbol{A}}}}{\parallel }_{1}\parallel {{{\boldsymbol{A}}}}{\parallel }_{\infty }\right)=1/{n}_{i}\). The gradient of the Poisson negative loglikelihood (Ψ_{2}) is not Lipschitz continuous, so its descent parameter cannot be precomputed^{43}. We heuristically determined the regularization parameters starting with values with a similar order of magnitude to \(1/{L}_{\nabla {{{\Psi }}}_{1}}\), then iteratively reduce until the cost function exhibits stable convergence. The regularization parameters were manually selected, however, future work may allow automated optimization by the Lcurve method or crossvalidation^{44}.
Estimating standard error of recovered chemical maps
Using estimation theory, we can approximate the uncertainty in a recovered chemical image for unbiased estimators with the model’s (Eq. (1)) Hessian expressed as: \({{{\boldsymbol{H}}}}({{{\boldsymbol{x}}}})={\nabla }_{{{{\boldsymbol{x}}}}}^{2}{{{\Psi }}}_{1}({{{\boldsymbol{x}}}})+{\nabla }_{{{{\boldsymbol{x}}}}}^{2}{{{\Psi }}}_{2}({{{\boldsymbol{x}}}})\), where
Calculation of standard error follows the Cramer–Rao inequality, which provides a lower bound given by: \(\left({{{\bf{Var}}}}({\hat{x}}_{j})\ge {[{{{{\boldsymbol{H}}}}}^{1}(\hat{{{{\boldsymbol{x}}}}})]}_{jj}\right)\)^{45}, where \({{{\bf{Var}}}}(\hat{{{{\boldsymbol{x}}}}})\) are variance maps for the recovered chemical distributions (\(\hat{{{{\boldsymbol{x}}}}}\)) and subscript jj denotes indices along with the diagonal elements. We determined this lower bound from an empirical derivation of the Fisher Information Matrix. From the variance, we thus extract standard error maps: \({{{\bf{Standard}}}}\ {{{\bf{Error}}}}=\sqrt{{{{\bf{Var}}}}(\hat{{{{\boldsymbol{x}}}}})}\) as demonstrated in Supplementary Fig. 10. The average standard error denotes the mean value of all pixels in Standard Error. Note, the TV regularizer reduces noise and may introduce bias due to smoothing, so the standard error measurements could potentially be lower; our Fisher information derivation provides an upper bound on uncertainty.
Inelastic scattering simulations for atomic imaging
The inelastic scattering simulations for the FePt nanoparticle structure (Fig. 4) were performed using the abTEM simulation code^{46}, using the algorithm described in ref. ^{33}. In this algorithm, the initial STEM probe is propagated and transmitted to some depth into the specimen using the scattering matrix method described in the PRISM algorithm^{47}. Next, the inelastic transition potentials of interest (in this case the L_{2,3} Fe and M_{4,5} Pt edges) were calculated and applied using the methods given in refs. ^{48,49}, using the GPAW density functional theory code^{50}. Finally, a second scattering matrix is used to propagate the inelastically scattered electrons through the sample and to the plane of the EELS entrance aperture. The elastic signal channels were calculated with the conventional PRISM method using the same parameters.
The atomic structure used in the simulations was a portion of the FePt nanoparticle structure determined from atomic electron tomography^{51}. After cropping out 1/4 of nanoparticle coordinates, the boundaries were padded by 5 Å total vacuum. The STEM probe’s convergence semiangle was set to 20 mrad and the voltage to 200 kV. The multislice steps used slice thicknesses of 2 Å, the wavefunction sampling size was 0.15 Å, and the projected potentials were computed using the infinite Kirkland parameterization^{52}. The EELS detector had a semiangle of 30 mrad, and the STEM probe positions were Nyquist sampled at a step size of 0.31 Å. After completion, we convolved the simulated images with a 0.2 Å Gaussian to account for source size. These simulation parameters required ~4 days of calculation time using the CPU mode of abTEM on a workstation with a 40 core Xeon processor clocked at 2.0 GHz.
Data availability
The datasets and codes that support the finding of this study are available from the corresponding author upon reasonable request.
References
D’Alfonso, A., Freitag, B., Klenov, D. & Allen, L. Atomicresolution chemical mapping using energydispersive xray spectroscopy. Phys. Rev. B 81, 100101 (2010).
Kothleitner, G. et al. Quantitative elemental mapping at atomic resolution using xray spectroscopy. Phys. Rev. Lett. 112, 085501 (2014).
Spence, J. & Lynch, J. Stem microanalysis by transmission electron energy loss spectroscopy in crystals. Ultramicroscopy 9, 267–276 (1982).
Muller, D. et al. Atomicscale chemical imaging of composition and bonding by aberrationcorrected microscopy. Science 319, 1073–1076 (2008).
Hart, J. L. et al. Direct detection electron energyloss spectroscopy: a method to push the limits of resolution and sensitivity. Sci. Rep. 7, 1–14 (2017).
Cueva, P., Hovden, R., Mundy, J., Xin, H. & Muller, D. Data processing for atomic resolution electron energy loss spectroscopy. Microsc. Microanal. 18, 667–675 (2012).
McMullan, G., Faruqi, A., Clare, D. & Henderson, R. Comparison of optimal performance at 300kev of three direct electron detectors for use in low dose electron microscopy. Ultramicroscopy 147, 156–163 (2014).
Kotula, P., Klenov, D. & Harrach, S. Challenges to quantitative multivariate statistical analysis of atomicresolution xray spectral. Microsc. Microanal. 18, 691–698 (2012).
LeBeau, J. M., Findlay, S. D., Allen, L. J. & Stemmer, S. Quantitative atomic resolution scanning transmission electron microscopy. Phys. Rev. Lett. 100, 206101 (2008).
Su, Y. et al. Multidimensional correlative imaging of subcellular events: combining the strengths of light and electron microscopy. Biophys. Rev. 2, 121–135 (2010).
Hall, D. L. & Llinas, J. An introduction to multisensor data fusion. Proc. IEEE 85, 6–23 (1997).
Lahat, D., Adali, T. & Jutten, C. Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103, 1449–1477 (2015).
Di, Z. W., Leyffer, S. & Wild, S. Optimizationbased approach for joint xray fluorescence and transmission tomographic inversion. SIAM J. Imaging Sci. 9, 1–23 (2016).
Calhoun, V. & Sui, J. Multimodal fusion of brain imaging data: a key to finding the missing link(s) in complex mental illness. Biol. Psychiatry 1, 230–244 (2016).
Hartel, P., Rose, H. & Dinges, C. Conditions and reasons for incoherent imaging in stem. Ultramicroscopy 63, 93–114 (1996).
Krivanek, O. L. et al. Atombyatom structural and chemical analysis by annular darkfield electron microscopy. Nature 464, 571–574 (2010).
Hovden, R. & Muller, D. A. Efficient elastic imaging of single atoms on ultrathin supports in a scanning transmission electron microscope. Ultramicroscopy 123, 59–65 (2012).
Crewe, A., Wall, J. & Langmore, J. Visibility of single atoms. Science 168, 1338–1340 (1970).
Wall, J., Isaacson, M. & Langmore, J. The collection of scattered electrons in dark field electron microscopy. Optik 39, 359–374 (1974).
Di, Z. W. et al. Joint reconstruction of xray fluorescence and transmission tomography. Opt. Express 25, 13107–13124 (2017).
Odstrčil, M., Menzel, A. & GuizarSicairos, M. Iterative leastsquares solver for generalized maximumlikelihood ptychography. Opt. Express 26, 3108–3123 (2018).
Rudin, L., Osher, S. & Fatemi, E. Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992).
Donoho, D. Compressed sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006).
Candès, E., Romberg, J. & Tao, T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52, 489–509 (2006).
Rozeveld, S., Lee, W.S. & Longo, P. Characterization of cobalt sulfide catalysts. Microsc. Microanal. 26, 1248–1250 (2020).
Schlossmacher, P., Klenov, D., Freitag, B. & von Harrach, H. Enhanced detection sensitivity with a new windowless XEDS system for AEM based on silicon drift detector technology. Microsc. Today 18, 14–20 (2010).
Perera, S. et al. Enhanced supercapacitor performance for equal comn stoichiometry in colloidal Co_{3−x}Mn_{x}O_{4} nanoparticles, in additivefree electrodes. Chem. Mater. 27, 7861–7873 (2015).
Bhargava, A. et al. Mn cations control electronic transport in spinel Co_{x}Mn_{3−x}O_{4} nanoparticles. Chem. Mater. 31, 4228–4233 (2019).
Ha, D.H. et al. Solidsolid phase transformations induced through cation exchange and strain in 2d heterostructured copper sulfide nanocrystals. Nano Lett. 14, 7090–7099 (2014).
Horé, A. & Ziou, D. Image quality metrics: PSNR and SSIM. In 2010 20th International Conference on Pattern Recognition, (ed. Erçil, A.) 2366–2369 (IEEE, 2010).
Dwyer, C. The role of symmetry in the theory of inelastic highenergy electron scattering and its application to atomicresolution coreloss imaging. Ultramicroscopy 151, 68–77 (2015).
Allen, L. J. et al. Modelling the inelastic scattering of fast electrons. Ultramicroscopy 151, 11–22 (2015).
Brown, H., Ciston, J. & Ophus, C. Linearscaling algorithm for rapid computation of inelastic transition of multiple electron scattering. Phys. Rev. Res. 1, 033186 (2019).
Zhang, Z., De Backer, A., Lobato, I., Van Aert, S. & Nellist, P. Combining ADFEDX scattering crosssections for elemental quantification of nanostructures. Microsc. Microanal. 27, 600–602 (2021).
Anstis, G., Cai, D. & Cockayne, D. Limitations on the sstate approach to the interpretation of subangstrom resolution electron microscope images and microanalysis. Ultramicroscopy 94, 309–327 (2003).
Hovden, R., Xin, H. L. & Muller, D. A. Channeling of a subangstrom electron beam in a crystal mapped to twodimensional molecular orbitals. Phys. Rev. B 86, 195415 (2012).
Egerton, R. & Cheng, S. Characterization of an analytical electron microscope with a NiO test specimen. Ultramicroscopy 55, 43–54 (1994).
Rez, P. Crosssections for energy loss spectroscopy. Ultramicroscopy 9, 283–288 (1982).
Egerton, R. Formulae for lightelement microanalysis by electron energyloss spectrometry. Ultramicroscopy 3, 243–251 (1978).
Findlay, S. et al. Dynamics of annular bright field imaging in scanning transmission electron microscopy. Ultramicroscopy 110, 903–923 (2010).
Savitzky, B. et al. Bending and breaking of stripes in charge ordered manganite. Nat. Commun. 8, 1883 (2017).
Beck, D. & Teboulle, M. Fast gradientbased algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Proc. 18, 2419–2434 (2009).
Dupé, F.X., Fadili, J. & Starck, J.L. A proximal iteration for deconvolving Poisson noisy images using sparse representations. IEEE Trans. Image Proc. 18, 310–321 (2009).
Vogel, C. Computational Methods for Inverse Problems, chap 7, 97–127 (SIAM, 2002).
Wei, X., Urbach, H. P. & Coene, W. Cramér–Rao lower bound and maximumlikelihood estimation in ptychography with Poisson noise. Phys. Rev. A 102, P043516 (2020).
Madsen, J. & Susi, T. The abTEM code: transmission electron microscopy from first principles. Open Res. Eur. 1, 1–24 (2021).
Ophus, C. A fast image simulation algorithm for scanning transmission electron microscopy. Adv. Struct. Chem. Imag. 3, 13 (2017).
Saldin, D. & Rez, P. The theory of the excitation of atomic innershells in crystals by fast electrons. Philos. Mag. B 55, 481–489 (1987).
Dwyer, C., Findlay, S. & Allen, L. Multiple elastic scattering of coreloss electrons in atomic resolution imaging. Phys. Rev. B 100, 206101 (2008).
Enkovaara, J. et al. Electronic structure calculations with GPAW: a realspace implementation of the projector augmentedwave method. J. Phys.: Condens. Matter 22, 253202 (2010).
Yang, Y. et al. Deciphering chemical order/disorder and material properties at the singleatom level. Nature 542, 75–79 (2017).
Kirkland, E. Advanced Computing in Electron Microscopy (Springer Nature, 2010).
Acknowledgements
R.H. and J.S. acknowledge support from the Army Research Office, Computing Sciences (W911NF17S0002) and Dow Chemical Company. Work at the Molecular Foundry was supported by the Office of Basic Energy Sciences, of the U.S. Department of Energy under Contract no. DEAC0205CH11231.
Author information
Authors and Affiliations
Contributions
J.S., R.H., Z.W.D., and Y.J. conceived the idea. J.S. and R.H. implemented the multimodal reconstruction algorithms and performed the analysis. J.F. and Z.W.D. assisted with the algorithm formulation. C.O. designed and ran the inelastic multislice simulations. S.R. and A.J.F. conducted the EDX experiments. R.H. and I.E.B. conducted the EELS experiments. R.R., D. H., S.D.P., synthesized the Co_{3−x}Mn_{x}O and ZnSCu nanoparticles. J.S. and R.H. wrote the manuscript. All authors reviewed and commented on the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Schwartz, J., Di, Z.W., Jiang, Y. et al. Imaging atomicscale chemistry from fused multimodal electron microscopy. npj Comput Mater 8, 16 (2022). https://doi.org/10.1038/s41524021006925
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41524021006925
This article is cited by

Imaging 3D chemistry at 1 nm resolution with fused multimodal electron tomography
Nature Communications (2024)

Machine learning for automated experimentation in scanning transmission electron microscopy
npj Computational Materials (2023)