## Abstract

Efforts to map atomic-scale chemistry at low doses with minimal noise using electron microscopes are fundamentally limited by inelastic interactions. Here, fused multi-modal electron microscopy offers high signal-to-noise ratio (SNR) recovery of material chemistry at nano- and atomic-resolution by coupling correlated information encoded within both elastic scattering (high-angle annular dark-field (HAADF)) and inelastic spectroscopic signals (electron energy loss (EELS) or energy-dispersive x-ray (EDX)). By linking these simultaneously acquired signals, or modalities, the chemical distribution within nanomaterials can be imaged at significantly lower doses with existing detector hardware. In many cases, the dose requirements can be reduced by over one order of magnitude. This high SNR recovery of chemistry is tested against simulated and experimental atomic resolution data of heterogeneous nanomaterials.

## Introduction

Modern scanning transmission electron microscopes (STEM) can focus sub-angstrom electron beams on and between atoms to quantify structure and chemistry in real space from elastic and inelastic scattering processes. The chemical composition of specimens is revealed by spectroscopic techniques produced from inelastic interactions in the form of energy-dispersive X-rays (EDX)^{1,2} or electron energy loss (EELS)^{3,4}. Unfortunately, high-resolution chemical imaging requires high doses (e.g., >10^{6} e/Å^{2}) that often exceed the specimen limits—resulting in chemical maps that are noisy or missing entirely^{5,6}. Substantial effort and cost to improve detector hardware have brought the field closer to the measurement limits set by inelastic processes^{7,8}. Direct interpretation of atomic structure at higher-SNR is provided by elastically scattered electrons collected in a high-angle annular dark-field detector (HAADF); however, this signal under-describes the chemistry^{9}. Reaching the lowest doses at the highest SNR ultimately requires fusing both elastic and inelastic scattering modalities.

Currently, detector signals—such as HAADF and EDX/EELS—are analyzed separately for insight into structural, chemical, or electronic properties^{10}. Correlative imaging disregards shared information between structure and chemistry and misses opportunities to recover useful information. Data fusion, popularized in satellite imaging, goes further than correlation by linking the separate signals to reconstruct new information and improve measurement accuracy^{11,12,13}. Successful data fusion designs an analytical model that faithfully represents the relationship between modalities, and yields a meaningful combination without imposing any artificial connections^{14}.

Here we introduce fused multi-modal electron microscopy, a technique offering high SNR recovery of nanomaterial chemistry by linking correlated information encoded within both HAADF and EDX/EELS. We recover chemical maps by reformulating the inverse problem as a nonlinear optimization that seeks solutions that accurately match the actual chemical distribution in a material. Our approach substantially improves SNRs for chemical maps, often around 300–500%, and can reduce doses over one order of magnitude while remaining consistent with original measurements. We demonstrate EDX/EELS datasets at sub-nanometer and atomic resolution. Moreover, fused multi-modal electron microscopy recovers a specimen’s relative concentration, allowing researchers to measure local stoichiometry with less than 15% error without any knowledge of the inelastic cross-sections. Convergence and uncertainty estimates are identified along with simulations that provide a ground-truth assessment of when and how this approach can fail.

## Results

### Principles of multi-modal electron microscopy

Fused multi-modal electron microscopy recovers chemical maps by solving an optimization problem seeking a solution that strongly correlates with (1) the HAADF modality containing high SNR, (2) the chemically sensitive spectroscopic modality (EELS and/or EDX), and (3) encourages sparsity in the gradient domain producing solutions with reduced spatial variation. The overall optimization function is as follows:

where *λ* are regularization parameters, *b*_{H} is the measured HAADF, *b*_{i} and *x*_{i} are the measured and reconstructed chemical maps for element *i*, *ε* herein prevents log(0) issues but can also account for background, \({{\mathrm{log}}}\,\) is applied element-wise to its arguments, superscript T denotes vector transpose, and **1** denotes the vector of *n*_{x}*n*_{y} ones, where *n*_{x} × *n*_{y} is the image size.

The three terms in Eq. (1) define our multi-modal approach to surpass traditional dose limits for chemical imaging. First, we assume a forward model where the simultaneous HAADF is a linear combination of elemental distributions (\({{{{\boldsymbol{x}}}}}_{i}^{\gamma }\) where *γ* ∈ [1.4, 2]). The incoherent linear imaging approximation for elastic scattering scales with atomic number as \({Z}_{i}^{\gamma }\) where *γ* is typically around 1.7^{15,16,17}. This *γ* is bounded between 2 for Rutherford scattering from bare nuclear potentials to 4/3 as described by Lenz–Wentzel expressions for electrons experiencing a screened Coulombic potential^{18,19}. Second, we ensure the recovered signals maintain a high degree of data fidelity with the initial measurements by using maximum negative log-likelihood for spectroscopic measurements dominated by low-count Poisson statistics^{20,21}. In a higher count regime, this term can be substituted with a simple least-squares error. Lastly, we utilize channel-wise total variation (TV) regularization to enforce a sparse gradient magnitude, which reduces noise by promoting image smoothness while preserving sharp features^{22}. This sparsity constraint, popularized by the field of compressed sensing (CS), is powerful yet minimal prior to recovering structured data^{23,24}. When implementing, each of these three terms can and should be weighted by appropriately selected coefficients that balance their contributions. All three terms are necessary for accurate recovery (Supplementary Fig. 1).

### High-SNR recovery of nanomaterial chemistry

Figure 1 demonstrates high-SNR recovery for EDX signals of commercial cobalt sulfide (CoS) nano-catalysts for oxygen-reduction applications—a unique class with the highest activity among non-precious metals^{25}. Figure 1a illustrates the model that links the two modalities (EDX and HAADF) simultaneously collected in the electron microscope. The low detection rate for characteristic X-rays is due to minimal emission (e.g., over 50% for *Z* > 32 and below 2% for *Z* < 11) and collection yield (<9%)^{26}. For high-resolution EDX, the low count rate yields a sparse chemical image dominated by shot noise (Fig. 1b). However, noise in the fused multi-modal chemical map is virtually eliminated (Fig. 1d) and recovers chemical structure without a loss of resolution—including the nanoparticle core and oxide shell interface. The chemical maps produced by fused multi-modal EM quantitatively agree with the expected stoichiometry—the specimen core contains a relative concentration of 39 ± 1.6%, 42 ± 2.5%, and 13 ± 2.4% and exterior shell composition of 26 ± 2.8%, 11 ± 2.0%, 54 ± 1.3% for Co, S, O, respectively. The dose for this dataset was ~10^{5} e Å^{−2} and a 0.7 sr EDX detector was used; however, these quantitative estimates remained consistent when the dose was reduced to ~10^{4} e Å^{−2}.

Fused multi-modal electron microscopy accurately recovers chemical structure down to atomic length scales—demonstrated here for EELS spectroscopic signals. EELS-derived chemical maps for Co_{3−x}Mn_{x}O_{4} (*x* = 1.49) high-performing super-capacitor nanoparticles are substantially improved by fused multi-modal electron microscopy in Fig. 2. This composite Co–Mn oxide was designed to achieve a synergy between cobalt oxide’s high specific capacitance and manganese oxide’s long life cycle^{27,28}. While the Co_{3−x}Mn_{x}O_{4} nanoparticle appears chemically homogeneous in the HAADF projection image along the [100] direction (Fig. 2c), core–shell distinctions are hinted at in the raw EELS maps (Fig. 2b). Specifically, these nanoparticles contain an Mn-rich center with a Co shell and homogeneous distribution of O. However the raw EELS maps are excessively degraded by noise, preventing analysis beyond the rough assessment of specimen morphology. The multi-modal reconstructions (Fig. 2d) confirm the crystalline Co-rich shell and map the Co/Mn interface in greater detail (Fig. 2e). In the presence of cobalt and manganese, the HAADF image lacks noticeable contrast from oxygen; the resulting oxygen map lacks detail and benefits mostly from regularization.

Figure 3 exhibits fused multi-modal electron microscopy at an atomic resolution on copper–sulfur heterostructured nanocrystals with zinc sulfide caps with potential applications in photovoltaic devices or battery electrodes^{29}. The copper sulfide properties are sensitive to the Cu–S stoichiometry and crystal structure at the interface between ZnS and Cu_{0.64}S_{0.36}. Figure 3 shows high-resolution HAADF and EELS characterization of a heterostructure Cu_{0.64}S_{0.36}–ZnS interface. Fused multi-modal electron microscopy maps out the atomically sharp Cu_{0.64}S_{0.36}–ZnS interface and reveals step edges between the two layers. The labeled points on the RGB chemical overlay (Fig. 3d) show the chemical ratios produced by multi-modal EM for the Cu_{0.64}S_{0.36} and ZnS regions—values which are consistent with the reported growth conditions. Figure 3e shows the algorithm convergence for each of the three terms in the optimization function (Eq. (1))—smooth and asymptotic decay is an indicator of reliable reconstruction. Refer to Supplementary Fig. 2 for an additional demonstration at the atomic scale on an ordered manganite system.

Fused multi-modal imaging of Fe and Pt distributions from inelastic multislice simulations (Fig. 4) provide ground truth solutions to validate recovery at atomic resolution under multiple scattering conditions of an on-axis ~8 nm nanoparticle. Here, we applied Poisson noise (Fig. 4b) containing electron doses of ~10^{9} e Å^{−2}, to produce chemical maps with noise levels resembling experimental atomic-resolution EELS datasets (SNR ≃ 5). We estimated SNR improvements by measuring peak-SNR for the noisy and recovered chemical maps^{30}. Qualitatively, the recovered chemical distributions (Fig. 4c) match the original images. Figure 4d illustrates the agreement of the line profiles as the atom column positions and relative peak intensities between the ground truth and multi-modal reconstruction are almost identical.

Simulating EELS chemical maps is computationally demanding as every inelastic scattering event requires propagation of an additional wavefunction^{31,32}—scaling faster than the cube of the number of beams, \(O({N}^{3}{{\mathrm{log}}}\,N)\). Inelastic transition potentials of interest (in this case the L_{2,3} Fe and M_{4,5} Pt edges) were calculated from density function theory (see the “Methods” section). Long computation times (nearly 4000 core-hours) result from a large number of outgoing scattering channels corresponding to the many possible excitations in a sample. For this reason, there is little precedence for inelastic image simulations. We relaxed the runtime by utilizing the PRISM STEM-EELS approximation, achieving over a ten-fold speedup (see the “Methods” section)^{33}. Future work may explore the effects of smaller ADF collection angles with increased coherence lengths and crystallographic contrast^{15,34}, or thicker specimens where electron channeling becomes more concerning^{35,36}.

### Quantifying chemical concentration

Fused multi-modal electron microscopy can produce stoichiometrically meaningful chemical maps without specific knowledge of inelastic cross-sections. Here, the ratio of pixel values in the reconstructed maps quantifies elemental concentration. We demonstrate quantifiable chemistry on experimental metal oxide thin films with known stoichiometry: NiO^{37} and ZrO_{2}. A histogram of intensities from the recovered chemical maps is fitted with Gaussian distributions to determine the average concentration. The recovered pixel values highlighted in Fig. 5 followed a single Gaussian distribution where the Zr and Ni concentrations are centered at about 35 ± 5.8% and 50 ± 2.9%. In both cases, the average Ni and Zr relative concentrations are approximately equivalent to the expected ratio from the crystal stoichiometry: 33% and 50%. The CoS nanoparticle in Fig. 1 follows a bi-modal distribution for the core and shell phases (Supplementary Fig. 5). We found measuring stoichiometry is robust across a range of *γ* values close to 1.7. In cases where *γ* is far off (e.g., *γ* = 1.0), the quantification is systematically incorrect (Supplementary Fig. 6).

We further validate stoichiometric recovery on a synthetic gallium oxide crystal (Fig. 5) where two overlapping Ga and O thin films of equal thickness have a stoichiometery of Ga_{2}O_{3}. The simulated HAADF signal is proportional to \({\sum }_{i}{({{{{\boldsymbol{x}}}}}_{i}{Z}_{i})}^{\gamma }\) where *x*_{i} is the concentration for element *i* and *Z*_{i} is the atomic number. As shown by the histogram, the simulated results agree strongly with the prior knowledge and successfully recover the relative Ga concentration. The Gaussian distribution is centered about 40 ± 0.4% when the ground truth is 40%. The inset shows convergence plots.

We estimate a stoichiometric error of <15% for most materials based on the relative concentration’s standard deviation (±7%) added in quadrature with the variation of solutions (±6%). Although the algorithm shows stable convergence, the overall quantitative conclusions are slightly sensitive to the selection of hyperparameters. We estimate incorrect selection of hyperparameters could result in variation of roughly ±6% from the correct prediction in stoichiometry even when the algorithm converges (convergence shown in Supplementary Figs. 8 and 9). This error is comparable to estimating chemical concentrations directly from EELS/EDX spectral maps from the ratio of scattering cross-section against core-loss intensity^{38}. However, traditional approaches require accurate knowledge of all experimental parameters (e.g., beam energy, specimen-thickness, collection angles) and accurate calculation of the inelastic cross-section typically to provide errors roughly between 5% and 10%^{39}.

### Influence of electron dose

To better understand the accuracy of fused multi-modal electron microscopy at low doses, we performed a quantitative study of normalized root-mean-square error (RMSE) concentrations for a simulated 3D core–shell nanoparticle (CoS core, CoO shell). Figure 6 shows the fused multi-modal reconstruction accuracy across a wide range of HAADF and chemical SNR. The simulated projection images were generated by a simple linear incoherent imaging model of the 3D chemical compositions highlighted in Fig. 6d—here the probe’s depth of focus is much larger than the object. Random Poisson noise corresponding to different electron dose levels was applied to vary the SNR across each pixel.

Overall, the RMSE simulation map (Fig. 6a) shows the core–shell nanoparticle chemical maps are accurately recovered at low doses (HAADF SNR ≳ 4 and chemical SNR ≳ 2); however, they become less accurate at extremely low doses. The RMSE map for multimodal reconstruction shows a predictably continuous degradation in recovery as signals diminish. The degraded and reconstructed chemical maps for various noise levels are highlighted in Fig. 6b. The Co map closely mirrors the *Z*-contrast observed in HAADF (not shown) simply because it is the heaviest element present. Usually, researchers will perform spectroscopic experiments in the top right corner of Fig. 6a (e.g., HAADF SNR > 20, chemical SNR > 3), which for this simulation, provides accurate recovery.

In actual experiments, the ground truth is unknown and RMSE cannot be calculated to assess fused multi-modal electron microscopy. However, we can estimate accuracy by calculating an average standard error of our recovered image from the Hessian of our model (see the “Methods” section). The standard error reflects uncertainty at each pixel in a recovered chemical map by quantifying the neighborhood size for similar solutions (Supplementary Fig. 10). The average standard error across all pixels in a fused multi-modal image provides a single value metric of the reconstruction accuracy (see the “Methods” section). Figure 6c shows that RMSE and average standard error correlate, especially at higher doses (SNR > 10).

## Discussion

While this paper highlights the advantages of multi-modal electron microscopy, the technique is not a black-box solution. Step sizes for convergence and weights on the terms in the cost function (Eq. (1)) must be reasonably selected. This manuscript illustrates approaches to assess the validity of concentration measurements using confidence estimation demonstrated across several simulated and experimental material classes. Standard spectroscopic pre-processing methods become ever more critical in combination with multi-modal fusion. Improper background subtraction of EELS spectra or overlapping characteristic X-ray peaks that normally cause inaccurate stoichiometric quantification also reduce the accuracy of fused multi-modal imaging.

Fused multi-modal electron microscopy offers little advantage in recovering chemical maps for elements with insignificant contrast in the HAADF modality. This property is limiting for analyzing specimens with low-*Z* elements in the presence of heavy elements (e.g., oxygen and lutetium). Future efforts could resolve this challenge by incorporating an additional complementary elastic imaging mode where light elements are visible, such as an annular bright-field (ABF)^{40}. However, in some instances, fused multi-modal electron microscopy may recover useful information for under-determined chemical signals. For example, in a Bi_{0.35}Sr_{0.18}Ca_{0.47}MnO_{3} (BSCMO) system^{41}, only the Ca, Mn, and O EELS maps were obtained, yet multimodality remarkably improves the SNR of measured maps despite missing two elements (Supplementary Fig. 2).

Although fused multi-modal chemical mapping appears quite robust at nanometer or sub-nanometer resolution, we found atomic-resolution reconstructions can be challenged by spurious atom artifacts which require attention. However, this is easily remedied by down-sampling to frequencies below the first Bragg peaks and analyzing a lower resolution chemical map. Alternatively, recovery with minimal spurious atom artifacts is achieved when lower resolution reconstructions are used as an initial guess (Supplementary Fig. 11).

In summary, we present a model-driven data fusion algorithm that substantially improves the quality of electron microscopy spectroscopic maps at nanometer to atomic resolutions by using both elastic and inelastic signals. From these signals or modalities, each atom’s chemical identity and coordination provides essential information about the performance of nanomaterials across a wide range of applications from clean energy, batteries, and optoelectronics, among many others. In both synthetic and experimental datasets, multi-modal electron microscopy shows quantitatively accurate chemical maps with values that reflect stoichiometry. This approach not only improves SNR but opens a pathway for low-dose chemical imaging of radiation-sensitive materials. Although demonstrated herein for common STEM detectors (HAADF, EDX, and EELS), this approach can be extended to many other modalities—including pixel array detectors, annular bright field, ptychography, low-loss EELS, etc. One can imagine a future where all scattered and emitted signals in an electron microscope are collected and fused for maximally efficiently atomic characterization of matter.

## Methods

### Electron microscopy

Simultaneously acquired EELS and HAADF datasets were collected on a fifth-order aberration-corrected Nion UltraSTEM microscope operated at 100 keV with a probe semi-angle of roughly 30 mrad and collection semi-angle of 80–240 and 0–60 mrad for HAADF and EELS, respectively. Both specimens were imaged at 30 pA, for a dwell time of 10 ms (Fig. 3) and 15 ms (Fig. 2) receiving a total dose of 3.25 × 10^{4} and 7.39 × 10^{4} e/Å^{2}. The EELS signals were obtained by integration over the core loss edges, all of which were done after background subtraction. The background EELS spectra were modeled using a linear combination of power laws implemented using the open-source Cornell Spectrum Imager software^{6}.

Simultaneously acquired EDX and HAADF datasets were collected on a Thermo Fisher Scientific Titan Themis G2 at 200 keV with a probe semi-angle of roughly 25 mrad, HAADF collection semi-angle of 73–200 mrad, and 0.7 sr EDX solid angle. The CoS specimen was imaged at 100 pA and 40 μs dwell time for 50 frames receiving a total dose of ~2 × 10^{5} e/Å^{2}. The initial chemical distributions were generated from EDX maps using commercial Velox software that produced initial net count estimates (however atomic percent estimates are also suitable).

### Fused multi-modal recovery

Here, fused multi-modal electron microscopy is framed as an inverse problem expressed in the following form: \(\hat{{{{\boldsymbol{x}}}}}=\arg \mathop{\min }\limits_{{{{\boldsymbol{x}}}}\ge 0}{{{\Psi }}}_{1}({{{\boldsymbol{x}}}})+{\lambda }_{1}{{{\Psi }}}_{2}({{{\boldsymbol{x}}}})+{\lambda }_{2}{{{\rm{TV}}}}({{{\boldsymbol{x}}}})\), where \(\hat{{{{\boldsymbol{x}}}}}\) is the final reconstruction, and the three terms are described in the main manuscript (Eq. (1)). When implementing an algorithm to solve this problem, we concatenate the multi-element spectral variables (*x*_{i}, *b*_{i}) as a single vector: \({{{\boldsymbol{x}}}},\,{{{\boldsymbol{b}}}}\,\in \,{{\mathbb{R}}}^{{n}_{x}{n}_{y}{n}_{i}}\), where *n*_{i} denotes the total number of reconstructed elements.

The optimization problem is solved by a combination of gradient descent with total variation regularization. We solve this cost function by descending along with the negative gradient directions for the first two terms and subsequently evaluate the isotropic TV proximal operator to denoise the chemical maps^{42}. The gradients of the first two terms are:

where ⊘ denotes point-wise division. Here, the first term in the cost function, relating the elastic and inelastic modalities, has been equivalently re-written as \({{{\Psi }}}_{1}=\frac{1}{2}{\left\Vert {{{{\boldsymbol{b}}}}}_{\mathrm {{H}}}-{{{\boldsymbol{A}}}}{{{{\boldsymbol{x}}}}}^{\gamma }\right\Vert }_{2}^{2}\), where \({{{\boldsymbol{A}}}}\,\in \,{{\mathbb{R}}}^{{n}_{x}{n}_{y}\times {n}_{x}{n}_{y}{n}_{i}}\) expresses the summation of all elements as matrix–vector multiplication. Evaluation for the TV proximal operator is in itself another iterative algorithm. In addition, we impose a non-negativity constraint since negative concentrations are unrealistic. We initialize the first iterate with the measured data (\({{{{\boldsymbol{x}}}}}_{i}^{0}={{{{\boldsymbol{b}}}}}_{i}\)), an ideal starting point as it is a local minima for Ψ_{2}.

The inverse of the Lipschitz constant (1/*L*) is an upper bound of the step size that can theoretically guarantee convergence. From Lipschitz continuity, we estimated the step size for the model term’s gradient (∇ Ψ_{1}) as 1/\({L}_{\nabla {{{\Psi }}}_{1}}\le 1/\left(\parallel {{{\boldsymbol{A}}}}{\parallel }_{1}\parallel {{{\boldsymbol{A}}}}{\parallel }_{\infty }\right)=1/{n}_{i}\). The gradient of the Poisson negative log-likelihood (Ψ_{2}) is not Lipschitz continuous, so its descent parameter cannot be pre-computed^{43}. We heuristically determined the regularization parameters starting with values with a similar order of magnitude to \(1/{L}_{\nabla {{{\Psi }}}_{1}}\), then iteratively reduce until the cost function exhibits stable convergence. The regularization parameters were manually selected, however, future work may allow automated optimization by the *L*-curve method or cross-validation^{44}.

### Estimating standard error of recovered chemical maps

Using estimation theory, we can approximate the uncertainty in a recovered chemical image for unbiased estimators with the model’s (Eq. (1)) Hessian expressed as: \({{{\boldsymbol{H}}}}({{{\boldsymbol{x}}}})={\nabla }_{{{{\boldsymbol{x}}}}}^{2}{{{\Psi }}}_{1}({{{\boldsymbol{x}}}})+{\nabla }_{{{{\boldsymbol{x}}}}}^{2}{{{\Psi }}}_{2}({{{\boldsymbol{x}}}})\), where

Calculation of standard error follows the Cramer–Rao inequality, which provides a lower bound given by: \(\left({{{\bf{Var}}}}({\hat{x}}_{j})\ge {[{{{{\boldsymbol{H}}}}}^{-1}(\hat{{{{\boldsymbol{x}}}}})]}_{jj}\right)\)^{45}, where \({{{\bf{Var}}}}(\hat{{{{\boldsymbol{x}}}}})\) are variance maps for the recovered chemical distributions (\(\hat{{{{\boldsymbol{x}}}}}\)) and subscript *j**j* denotes indices along with the diagonal elements. We determined this lower bound from an empirical derivation of the Fisher Information Matrix. From the variance, we thus extract standard error maps: \({{{\bf{Standard}}}}\ {{{\bf{Error}}}}=\sqrt{{{{\bf{Var}}}}(\hat{{{{\boldsymbol{x}}}}})}\) as demonstrated in Supplementary Fig. 10. The average standard error denotes the mean value of all pixels in **Standard Error**. Note, the TV regularizer reduces noise and may introduce bias due to smoothing, so the standard error measurements could potentially be lower; our Fisher information derivation provides an upper bound on uncertainty.

### Inelastic scattering simulations for atomic imaging

The inelastic scattering simulations for the FePt nanoparticle structure (Fig. 4) were performed using the abTEM simulation code^{46}, using the algorithm described in ref. ^{33}. In this algorithm, the initial STEM probe is propagated and transmitted to some depth into the specimen using the scattering matrix method described in the PRISM algorithm^{47}. Next, the inelastic transition potentials of interest (in this case the L_{2,3} Fe and M_{4,5} Pt edges) were calculated and applied using the methods given in refs. ^{48,49}, using the GPAW density functional theory code^{50}. Finally, a second scattering matrix is used to propagate the inelastically scattered electrons through the sample and to the plane of the EELS entrance aperture. The elastic signal channels were calculated with the conventional PRISM method using the same parameters.

The atomic structure used in the simulations was a portion of the FePt nanoparticle structure determined from atomic electron tomography^{51}. After cropping out 1/4 of nanoparticle coordinates, the boundaries were padded by 5 Å total vacuum. The STEM probe’s convergence semiangle was set to 20 mrad and the voltage to 200 kV. The multislice steps used slice thicknesses of 2 Å, the wavefunction sampling size was 0.15 Å, and the projected potentials were computed using the infinite Kirkland parameterization^{52}. The EELS detector had a semiangle of 30 mrad, and the STEM probe positions were Nyquist sampled at a step size of 0.31 Å. After completion, we convolved the simulated images with a 0.2 Å Gaussian to account for source size. These simulation parameters required ~4 days of calculation time using the CPU mode of abTEM on a workstation with a 40 core Xeon processor clocked at 2.0 GHz.

## Data availability

The datasets and codes that support the finding of this study are available from the corresponding author upon reasonable request.

## References

D’Alfonso, A., Freitag, B., Klenov, D. & Allen, L. Atomic-resolution chemical mapping using energy-dispersive x-ray spectroscopy.

*Phys. Rev. B***81**, 100101 (2010).Kothleitner, G. et al. Quantitative elemental mapping at atomic resolution using x-ray spectroscopy.

*Phys. Rev. Lett.***112**, 085501 (2014).Spence, J. & Lynch, J. Stem microanalysis by transmission electron energy loss spectroscopy in crystals.

*Ultramicroscopy***9**, 267–276 (1982).Muller, D. et al. Atomic-scale chemical imaging of composition and bonding by aberration-corrected microscopy.

*Science***319**, 1073–1076 (2008).Hart, J. L. et al. Direct detection electron energy-loss spectroscopy: a method to push the limits of resolution and sensitivity.

*Sci. Rep.***7**, 1–14 (2017).Cueva, P., Hovden, R., Mundy, J., Xin, H. & Muller, D. Data processing for atomic resolution electron energy loss spectroscopy.

*Microsc. Microanal.***18**, 667–675 (2012).McMullan, G., Faruqi, A., Clare, D. & Henderson, R. Comparison of optimal performance at 300kev of three direct electron detectors for use in low dose electron microscopy.

*Ultramicroscopy***147**, 156–163 (2014).Kotula, P., Klenov, D. & Harrach, S. Challenges to quantitative multivariate statistical analysis of atomic-resolution x-ray spectral.

*Microsc. Microanal.***18**, 691–698 (2012).LeBeau, J. M., Findlay, S. D., Allen, L. J. & Stemmer, S. Quantitative atomic resolution scanning transmission electron microscopy.

*Phys. Rev. Lett.***100**, 206101 (2008).Su, Y. et al. Multi-dimensional correlative imaging of subcellular events: combining the strengths of light and electron microscopy.

*Biophys. Rev.***2**, 121–135 (2010).Hall, D. L. & Llinas, J. An introduction to multisensor data fusion.

*Proc. IEEE***85**, 6–23 (1997).Lahat, D., Adali, T. & Jutten, C. Multimodal data fusion: an overview of methods, challenges, and prospects.

*Proc. IEEE***103**, 1449–1477 (2015).Di, Z. W., Leyffer, S. & Wild, S. Optimization-based approach for joint x-ray fluorescence and transmission tomographic inversion.

*SIAM J. Imaging Sci.***9**, 1–23 (2016).Calhoun, V. & Sui, J. Multimodal fusion of brain imaging data: a key to finding the missing link(s) in complex mental illness.

*Biol. Psychiatry***1**, 230–244 (2016).Hartel, P., Rose, H. & Dinges, C. Conditions and reasons for incoherent imaging in stem.

*Ultramicroscopy***63**, 93–114 (1996).Krivanek, O. L. et al. Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy.

*Nature***464**, 571–574 (2010).Hovden, R. & Muller, D. A. Efficient elastic imaging of single atoms on ultrathin supports in a scanning transmission electron microscope.

*Ultramicroscopy***123**, 59–65 (2012).Crewe, A., Wall, J. & Langmore, J. Visibility of single atoms.

*Science***168**, 1338–1340 (1970).Wall, J., Isaacson, M. & Langmore, J. The collection of scattered electrons in dark field electron microscopy.

*Optik***39**, 359–374 (1974).Di, Z. W. et al. Joint reconstruction of x-ray fluorescence and transmission tomography.

*Opt. Express***25**, 13107–13124 (2017).Odstrčil, M., Menzel, A. & Guizar-Sicairos, M. Iterative least-squares solver for generalized maximum-likelihood ptychography.

*Opt. Express***26**, 3108–3123 (2018).Rudin, L., Osher, S. & Fatemi, E. Nonlinear total variation based noise removal algorithms.

*Physica D***60**, 259–268 (1992).Donoho, D. Compressed sensing.

*IEEE Trans. Inf. Theory***52**, 1289–1306 (2006).Candès, E., Romberg, J. & Tao, T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information.

*IEEE Trans. Inf. Theory***52**, 489–509 (2006).Rozeveld, S., Lee, W.-S. & Longo, P. Characterization of cobalt sulfide catalysts.

*Microsc. Microanal.***26**, 1248–1250 (2020).Schlossmacher, P., Klenov, D., Freitag, B. & von Harrach, H. Enhanced detection sensitivity with a new windowless XEDS system for AEM based on silicon drift detector technology.

*Microsc. Today***18**, 14–20 (2010).Perera, S. et al. Enhanced supercapacitor performance for equal co-mn stoichiometry in colloidal Co

_{3−x}Mn_{x}O_{4}nanoparticles, in additive-free electrodes.*Chem. Mater.***27**, 7861–7873 (2015).Bhargava, A. et al. Mn cations control electronic transport in spinel Co

_{x}Mn_{3−x}O_{4}nanoparticles.*Chem. Mater.***31**, 4228–4233 (2019).Ha, D.-H. et al. Solid-solid phase transformations induced through cation exchange and strain in 2d heterostructured copper sulfide nanocrystals.

*Nano Lett.***14**, 7090–7099 (2014).Horé, A. & Ziou, D. Image quality metrics: PSNR and SSIM. In

*2010 20th International Conference on Pattern Recognition*, (ed. Erçil, A.) 2366–2369 (IEEE, 2010).Dwyer, C. The role of symmetry in the theory of inelastic high-energy electron scattering and its application to atomic-resolution core-loss imaging.

*Ultramicroscopy***151**, 68–77 (2015).Allen, L. J. et al. Modelling the inelastic scattering of fast electrons.

*Ultramicroscopy***151**, 11–22 (2015).Brown, H., Ciston, J. & Ophus, C. Linear-scaling algorithm for rapid computation of inelastic transition of multiple electron scattering.

*Phys. Rev. Res.***1**, 033186 (2019).Zhang, Z., De Backer, A., Lobato, I., Van Aert, S. & Nellist, P. Combining ADF-EDX scattering cross-sections for elemental quantification of nanostructures.

*Microsc. Microanal.***27**, 600–602 (2021).Anstis, G., Cai, D. & Cockayne, D. Limitations on the s-state approach to the interpretation of sub-angstrom resolution electron microscope images and microanalysis.

*Ultramicroscopy***94**, 309–327 (2003).Hovden, R., Xin, H. L. & Muller, D. A. Channeling of a subangstrom electron beam in a crystal mapped to two-dimensional molecular orbitals.

*Phys. Rev. B***86**, 195415 (2012).Egerton, R. & Cheng, S. Characterization of an analytical electron microscope with a NiO test specimen.

*Ultramicroscopy***55**, 43–54 (1994).Rez, P. Cross-sections for energy loss spectroscopy.

*Ultramicroscopy***9**, 283–288 (1982).Egerton, R. Formulae for light-element microanalysis by electron energy-loss spectrometry.

*Ultramicroscopy***3**, 243–251 (1978).Findlay, S. et al. Dynamics of annular bright field imaging in scanning transmission electron microscopy.

*Ultramicroscopy***110**, 903–923 (2010).Savitzky, B. et al. Bending and breaking of stripes in charge ordered manganite.

*Nat. Commun.***8**, 1883 (2017).Beck, D. & Teboulle, M. Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems.

*IEEE Trans. Image Proc.***18**, 2419–2434 (2009).Dupé, F.-X., Fadili, J. & Starck, J.-L. A proximal iteration for deconvolving Poisson noisy images using sparse representations.

*IEEE Trans. Image Proc.***18**, 310–321 (2009).Vogel, C.

*Computational Methods for Inverse Problems*, chap 7, 97–127 (SIAM, 2002).Wei, X., Urbach, H. P. & Coene, W. Cramér–Rao lower bound and maximum-likelihood estimation in ptychography with Poisson noise.

*Phys. Rev. A***102**, P043516 (2020).Madsen, J. & Susi, T. The abTEM code: transmission electron microscopy from first principles.

*Open Res. Eur.***1**, 1–24 (2021).Ophus, C. A fast image simulation algorithm for scanning transmission electron microscopy.

*Adv. Struct. Chem. Imag.***3**, 13 (2017).Saldin, D. & Rez, P. The theory of the excitation of atomic inner-shells in crystals by fast electrons.

*Philos. Mag. B***55**, 481–489 (1987).Dwyer, C., Findlay, S. & Allen, L. Multiple elastic scattering of core-loss electrons in atomic resolution imaging.

*Phys. Rev. B***100**, 206101 (2008).Enkovaara, J. et al. Electronic structure calculations with GPAW: a real-space implementation of the projector augmented-wave method.

*J. Phys.: Condens. Matter***22**, 253202 (2010).Yang, Y. et al. Deciphering chemical order/disorder and material properties at the single-atom level.

*Nature***542**, 75–79 (2017).Kirkland, E.

*Advanced Computing in Electron Microscopy*(Springer Nature, 2010).

## Acknowledgements

R.H. and J.S. acknowledge support from the Army Research Office, Computing Sciences (W911NF-17-S-0002) and Dow Chemical Company. Work at the Molecular Foundry was supported by the Office of Basic Energy Sciences, of the U.S. Department of Energy under Contract no. DE-AC02-05CH11231.

## Author information

### Authors and Affiliations

### Contributions

J.S., R.H., Z.W.D., and Y.J. conceived the idea. J.S. and R.H. implemented the multi-modal reconstruction algorithms and performed the analysis. J.F. and Z.W.D. assisted with the algorithm formulation. C.O. designed and ran the inelastic multi-slice simulations. S.R. and A.J.F. conducted the EDX experiments. R.H. and I.E.B. conducted the EELS experiments. R.R., D. H., S.D.P., synthesized the Co_{3−x}Mn_{x}O and ZnSCu nanoparticles. J.S. and R.H. wrote the manuscript. All authors reviewed and commented on the manuscript.

### Corresponding author

## Ethics declarations

### Competing interests

The authors declare no competing interests.

## Additional information

**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Supplementary information

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Schwartz, J., Di, Z.W., Jiang, Y. *et al.* Imaging atomic-scale chemistry from fused multi-modal electron microscopy.
*npj Comput Mater* **8**, 16 (2022). https://doi.org/10.1038/s41524-021-00692-5

Received:

Accepted:

Published:

DOI: https://doi.org/10.1038/s41524-021-00692-5