Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Primer
  • Published:

Machine learning in scanning transmission electron microscopy

Nature Reviews Methods Primers volume 2, Article number: 11 (2022) Cite this article

Subjects

Abstract

Scanning transmission electron microscopy (STEM) has emerged as a uniquely powerful tool for structural and functional imaging of materials on the atomic level. Driven by advances in aberration correction, STEM now allows the routine imaging of structures with single-digit picometre-level precision for localization of atomic units. This Primer focuses on the opportunities emerging at the interface between STEM and machine learning (ML) methods. We review the primary STEM imaging methods, including structural imaging, electron energy loss spectroscopy and its momentum-resolved modalities and 4D-STEM. We discuss the quantification of STEM structural data as a necessary step towards meaningful ML applications and its analysis in terms of the relevant physics and chemistry. We show examples of the opportunities offered by structural STEM imaging in elucidating the chemistry and physics of complex materials and how the latter connect to first-principles and phase-field models to yield consistent interpretation of generative physics. We present the critical infrastructural needs for the broad adoption of ML methods in the STEM community, including the storage of data and metadata to allow the reproduction of experiments. Finally, we discuss the application of ML to automating experiments and novel scanning modes.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: STEM as a quantitative tool.
Fig. 2: Angle-dependent STEM-EELS, 4D-STEM and beam engineering.
Fig. 3: Identifying positions of atoms and atomic columns from STEM images.
Fig. 4: Application of a variational auto encoder to graphene.
Fig. 5: HAADF imaging of grain boundaries.
Fig. 6: STEM examples in chemistry.
Fig. 7: Bottom-up structural analysis for a ferroelectric material.

Similar content being viewed by others

References

  1. Keyse, R. J., Garratt-Reed, A. J., Goodhew, P. J. & Lorimer, G. W. Introduction to Scanning Transmission Electron Microscopy (Routledge, 2018).

  2. Shibata, N. et al. Electric field imaging of single atoms. Nat. Commun. 8, 15631 (2017).

    Article  ADS  Google Scholar 

  3. Campanini, M., Nasi, L., Albertini, F. & Erni, R. Disentangling nanoscale electric and magnetic fields by time-reversal operation in differential phase-contrast STEM. Appl. Phys. Lett. 117, 154102 (2020).

    Article  ADS  Google Scholar 

  4. Müller, K. et al. Atomic electric fields revealed by a quantum mechanical approach to electron picodiffraction. Nat. Commun. 5, 5653 (2014).

    Article  ADS  Google Scholar 

  5. Nellist, P. D., McCallum, B. C. & Rodenburg, J. M. Resolution beyond the information limit in transmission electron-microscopy. Nature 374, 630–632 (1995).

    Article  ADS  Google Scholar 

  6. Hwang, J., Zhang, J. Y., D’Alfonso, A. J., Allen, L. J. & Stemmer, S. Three-dimensional imaging of individual dopant atoms in SrTiO3. Phys. Rev. Lett. 111, 266101 (2013).

    Article  ADS  Google Scholar 

  7. Xin, H. L. & Muller, D. A. Aberration-corrected ADF-STEM depth sectioning and prospects for reliable 3D imaging in S/TEM. J. Electron. Microsc. 58, 157–165 (2009).

    Article  Google Scholar 

  8. Benthem, K. V. et al. Three-dimensional imaging of individual hafnium atoms inside a semiconductor device. Appl. Phys. Lett. 87, 034104 (2005).

    Article  ADS  Google Scholar 

  9. Borisevich, A. Y., Lupini, A. R. & Pennycook, S. J. Depth sectioning with the aberration-corrected scanning transmission electron microscope. Proc. Natl Acad. Sci. USA 103, 3044–3048 (2006).

    Article  ADS  Google Scholar 

  10. Nellist, P. D. & Wang, P. Optical sectioning and confocal imaging and analysis in the transmission electron microscope. Annu. Rev. Mater. Res. 42, 125–143 (2012).

    Article  ADS  Google Scholar 

  11. Wang, P. et al. Bright-field scanning confocal electron microscopy using a double aberration-corrected transmission electron microscope. Ultramicroscopy 111, 877–886 (2011).

    Article  Google Scholar 

  12. Ishikawa, R. et al. Single atom visibility in STEM optical depth sectioning. Appl. Phys. Lett. 109, 163102 (2016).

    Article  ADS  Google Scholar 

  13. Levin, B. D. A. et al. Nanomaterial datasets to advance tomography in scanning transmission electron microscopy. Sci. Data 3, 160041 (2016).

    Article  Google Scholar 

  14. Alania, M. et al. Depth sectioning combined with atom-counting in HAADF STEM to retrieve the 3D atomic structure. Ultramicroscopy 177, 36–42 (2017).

    Article  Google Scholar 

  15. Dahmen, T. et al. Combined scanning transmission electron microscopy tilt- and focal series. Microsc. Microanal. 20, 548–560 (2014).

    Article  ADS  Google Scholar 

  16. Hovden, R. et al. Breaking the Crowther limit: combining depth-sectioning and tilt tomography for high-resolution, wide-field 3D reconstructions. Ultramicroscopy 140, 26–31 (2014).

    Article  Google Scholar 

  17. Krivanek, O. L. et al. Vibrational spectroscopy in the electron microscope. Nature 514, 209–212 (2014).

    Article  ADS  Google Scholar 

  18. Idrobo, J. C. et al. Temperature measurement by a nanoscale electron probe using energy gain and loss spectroscopy. Phys. Rev. Lett. 120, 095901 (2018).

    Article  ADS  Google Scholar 

  19. Bonnet, N. Multivariate statistical methods for the analysis of microscope image series: applications in materials science. J. Microscopy 190, 2–18 (1998).

    Article  Google Scholar 

  20. Trebbia, P. & Bonnet, N. EELS elemental mapping with unconventional methods I. Theoretical basis: image analysis with multivariate statistics and entropy concepts. Ultramicroscopy 34, 165–178 (1990).

    Article  Google Scholar 

  21. Trebbia, P. & Mory, C. EELS elemental mapping with unconventional methods II. Applications to biological specimens. Ultramicroscopy 34, 179–203 (1990).

    Article  Google Scholar 

  22. Dan, J., Zhao, X. & Pennycook, S. J. A machine perspective of atomic defects in scanning transmission electron microscopy. InfoMat 1, 359–375 (2019).

    Article  Google Scholar 

  23. Maksov, A. et al. Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2. npj Comput. Mater. 5, 12 (2019).

    Article  ADS  Google Scholar 

  24. Roccapriore, K. M., Kalinin, S. V. & Ziatdinov, M. Physics discovery in nanoplasmonic systems via autonomous experiments in scanning transmission electron microscopy. Preprint at https://arxiv.org/abs/2108.03290 (2021).

  25. Susi, T. et al. Towards atomically precise manipulation of 2D nanostructures in the electron microscope. 2D Mater. 4, 042004 (2017).

    Article  Google Scholar 

  26. Dyck, O., Kim, S., Kalinin, S. V. & Jesse, S. Placing single atoms in graphene with a scanning transmission electron microscope. Appl. Phys. Lett. 111, 113104 (2017).

    Article  ADS  Google Scholar 

  27. Dyck, O. et al. Electron-beam introduction of heteroatomic Pt–Si structures in graphene. Carbon 161, 750–757 (2020).

    Article  Google Scholar 

  28. Oxley, M. P., Lupini, A. R. & Pennycook, S. J. Ultra-high resolution electron microscopy. Rep. Prog. Phys. 80, 026101 (2016).

    Article  ADS  Google Scholar 

  29. Loth, S., Etzkorn, M., Lutz, C. P., Eigler, D. M. & Heinrich, A. J. Measurement of fast electron spin relaxation times with atomic resolution. Science 329, 1628–1630 (2010).

    Article  ADS  Google Scholar 

  30. Giessibl, F. J. AFM’s path to atomic resolution. Mater. Today 8, 32–41 (2005).

    Article  Google Scholar 

  31. Park, J. Y., Maier, S., Hendriksen, B. & Salmeron, M. Sensing current and forces with SPM. Mater. Today 13, 38–45 (2010).

    Article  Google Scholar 

  32. Abbe, E. Beiträge zur Theorie des Mikroskops und der mikroskopischen Wahrnehmung. Arch. Mikrosk. Anat. 9, 413–468 (1873).

    Article  Google Scholar 

  33. Scherzer, O. Uber einige Fehler von Elektonenlinsen. Zeit. Phys. 101, 593–603 (1936).

    Article  ADS  MATH  Google Scholar 

  34. Hawkes, P. W. in Science of Microscopy (eds Hawkes, P. W. & Spence, J. C.) 696–747 (Springer, 2007).

  35. Dellby, N., Krivanek, O. L., Nellist, P. D., Batson, P. E. & Lupini, A. R. Progress in aberration-corrected scanning transmission electron microscopy. J. Electron. Microsc. 50, 177–185 (2001).

    Google Scholar 

  36. Krivanek, O. L., Dellby, N. & Lupini, A. R. Towards sub-ångstrom electron beams. Ultramicroscopy 78, 1–11 (1999).

    Article  Google Scholar 

  37. Haider, M. et al. A spherical-aberration-corrected 200 kV transmission electron microscope. Ultramicroscopy 75, 53–60 (1998).

    Article  Google Scholar 

  38. Haider, M. et al. Electron microscopy image enhanced. Nature 392, 768–769 (1998).

    Article  ADS  Google Scholar 

  39. Rose, H. Elektronenoptische Aplanate. Optik 34, 285–311 (1971).

    Google Scholar 

  40. Zach, J. & Haider, M. Aberration correction in a low voltage SEM by a multipole corrector. Nucl. Instrum. Methods Phys. Res. A 363, 316–325 (1995).

    Article  ADS  Google Scholar 

  41. Batson, P. E. Aberration correction results in the IBM STEM instrument. Ultramicroscopy 96, 239–249 (2003).

    Article  Google Scholar 

  42. Nellist, P. D. et al. Direct sub-ångstrom imaging of a crystal lattice. Science 305, 1741–1741 (2004).

    Article  Google Scholar 

  43. Kisielowski, C. et al. Detection of single atoms and buried defects in three dimensions by aberration-corrected electron microscope with 0.5-ångstrom information limit. Microsc. Microanal. 14, 469–477 (2008).

    Article  ADS  Google Scholar 

  44. Shibata, N. et al. Differential phase-contrast microscopy at atomic resolution. Nat. Phys. 8, 611–615 (2012).

    Article  Google Scholar 

  45. Shibata, N. et al. New area detector for atomic-resolution scanning transmission electron microscopy. J. Electron. Microscopy 59, 473–479 (2010).

    Article  Google Scholar 

  46. Ophus, C. Four-dimensional scanning transmission electron microscopy (4D-STEM): from scanning nanodiffraction to ptychography and beyond. Microsc. Microanal. 25, 563–582 (2019).

    Article  ADS  Google Scholar 

  47. Egerton, R. F. Electron Energy-Loss Spectroscopy in the Electron Microscope (Springer, 2011).

  48. Hillier, J. & Baker, R. F. Microanalysis by means of electrons. J. Appl. Phys. 15, 663 (1944).

    Article  ADS  Google Scholar 

  49. Gubbens, A. J., Kraus, B., Krivanek, O. L. & Mooney, P. An imaging filter for high voltage electron microscopy. Ultramicroscopy 59, 255–265 (1995).

    Article  Google Scholar 

  50. Silcox, J. Core-loss EELS. Curr. Opin. Solid. State Mater. Sci. 3, 336–342 (1998).

    Article  ADS  Google Scholar 

  51. Nelayah, J. et al. Mapping surface plasmons on a single metallic nanoparticle. Nat. Phys. 3, 348–353 (2007).

    Article  Google Scholar 

  52. Zhou, W. et al. Atomically localized plasmon enhancement in monolayer graphene. Nat. Nanotechnol. 7, 161–165 (2012).

    Article  ADS  Google Scholar 

  53. Suenaga, K. & Koshino, M. Atom-by-atom spectroscopy at graphene edge. Nature 468, 1088–1090 (2010).

    Article  ADS  Google Scholar 

  54. Colliex, C., Kociak, M. & Stéphan, O. Electron energy loss spectroscopy imaging of surface plasmons at the nanometer scale. Ultramicroscopy 162, A1–A24 (2016).

    Article  Google Scholar 

  55. El-Sherif, H., Jovanovic, S., Preston, J. & Basim, N. Electron microscopy and interface plasmons characterization of cadmium telluride thin film grown incommensurately with weak bonding on sapphire. Microsc. Microanal. 26, 3116–3118 (2020).

    Article  ADS  Google Scholar 

  56. Yang, H., Garfunkel, E. L. & Batson, P. E. Probing free carrier plasmons in doped semiconductors using spatially resolved electron energy loss spectroscopy. Phys. Rev. B 102, 205427 (2020).

    Article  ADS  Google Scholar 

  57. Bonnet, N., Brun, N. & Colliex, C. Extracting information from sequences of spatially resolved EELS spectra using multivariate statistical analysis. Ultramicroscopy 77, 97–112 (1999).

    Article  Google Scholar 

  58. Jeanguillaume, C. & Colliex, C. Spectrum-image: the next step in EELS digital acquisition and processing. Ultramicroscopy 28, 252–257 (1989).

    Article  Google Scholar 

  59. Polman, A., Kociak, M. & García de Abajo, F. J. Electron-beam spectroscopy for nanophotonics. Nat. Mater. 18, 1158–1171 (2019).

    Article  ADS  Google Scholar 

  60. Hachtel, J., Lupini, A. & Idrobo, J. Exploring the capabilities of monochromated electron energy loss spectroscopy in the infrared regime. Sci. Rep. 8, 5637 (2018).

    Article  ADS  Google Scholar 

  61. Krivanek, O. et al. Towards sub-10 meV energy resolution STEM-EELS. J. Phys.Conf. Ser. 522, 012023 (2014).

    Article  Google Scholar 

  62. Hage, F. S. et al. Nanoscale momentum-resolved vibrational spectroscopy. Sci. Adv. 4, eaar7495 (2018).

    Article  ADS  Google Scholar 

  63. Hage, F. S., Radtke, G., Kepaptsoglou, D. M., Lazzeri, M. & Ramasse, Q. M. Single-atom vibrational spectroscopy in the scanning transmission electron microscope. Science 367, 1124 (2020).

    Article  ADS  Google Scholar 

  64. Lagos, M. J., Trügler, A., Hohenester, U. & Batson, P. E. Mapping vibrational surface and bulk modes in a single nanocube. Nature 543, 529–532 (2017).

    Article  ADS  Google Scholar 

  65. Boersch, H., Geiger, J. & Stickel, W. Interaction of 25-keV electrons with lattice vibrations in LiF. Experimental evidence for surface modes of lattice vibration. Phys. Rev. Lett. 17, 379–381 (1966).

    Article  ADS  Google Scholar 

  66. Egerton, R. F. Electron energy-loss spectroscopy in the TEM. Rep. Prog. Phys. 72, 25 (2009).

    Article  Google Scholar 

  67. Leapman, R. D., Fejes, P. L. & Silcox, J. Orientation dependence of core edges from anisotropic materials determined by inelastic-scattering of fast electrons. Phys. Rev. B 28, 2361–2373 (1983).

    Article  ADS  Google Scholar 

  68. Botton, G. A. A new approach to study bonding anisotropy with EELS. J. Electron. Spectrosc. Relat. Phenom. 143, 129–137 (2005).

    Article  Google Scholar 

  69. Hage, F. S. et al. Topologically induced confinement of collective modes in multilayer graphene nanocones measured by momentum-resolved STEM-VEELS. Phys. Rev. B 88, 12 (2013).

    Article  Google Scholar 

  70. Schattschneider, P. Exchange of angular momentum in EMCD experiments. Ultramicroscopy 109, 91–95 (2008).

    Article  Google Scholar 

  71. Wang, Y. Y. et al. Evolution of the low-energy excitations and dielectric function of Ba1-xKxBiO3 (0 <= x <= 0.50). Phys. Rev. B 47, 14503–14509 (1993).

    Article  ADS  Google Scholar 

  72. Liou, S. C. et al. Plasmons dispersion and nonvertical interband transitions in single crystal Bi2Se3 investigated by electron energy-loss spectroscopy. Phys. Rev. B 87, 6 (2013).

    Article  Google Scholar 

  73. Schuster, R., Kraus, R., Knupfer, M., Berger, H. & Buchner, B. Negative plasmon dispersion in the transition-metal dichalcogenide 2H-TaSe2. Phys. Rev. B 79, 5 (2009).

    Article  Google Scholar 

  74. Gloter, A., Chu, M. W., Kociak, M., Chen, C. H. & Colliex, C. Probing non-dipole allowed excitations in highly correlated materials with nanoscale resolution. Ultramicroscopy 109, 1333–1337 (2009).

    Article  Google Scholar 

  75. Senga, R. et al. Position and momentum mapping of vibrations in graphene nanostructures. Nature 573, 247 (2019).

    Article  ADS  Google Scholar 

  76. Egerton, R. F. Limits to the spatial, energy and momentum resolution of electron energy-loss spectroscopy. Ultramicroscopy 107, 575–586 (2007).

    Article  Google Scholar 

  77. Wang, Y. Y., Cheng, S. C., Dravid, V. P. & Zhang, F. C. Momentum-transfer resolved electron-energy-loss spectroscopy of solids — problems, solutions and applications. Ultramicroscopy 59, 109–119 (1995).

    Article  Google Scholar 

  78. Plotkin-Swing, B. et al. Hybrid pixel direct detector for electron energy loss spectroscopy. Ultramicroscopy 217, 10 (2020).

    Article  Google Scholar 

  79. Pogany, A. P. & Turner, P. S. Reciprocity in electron diffraction and microscopy. Acta Crystallogr. A 24, 103–109 (1968).

    Article  ADS  Google Scholar 

  80. Frank, J. Envelope of electron-microscopic transfer-functions for partially coherent illumination. Optik 38, 519–536 (1973).

    Google Scholar 

  81. Cowley, J. M. Electron nanodiffraction. Microsc. Res. Tech. 46, 75–97 (1999).

    Article  Google Scholar 

  82. Cowley, J. M. Adjustment of a STEM instrument by use of shadow images. Ultramicroscopy 4, 413–418 (1979).

    Article  Google Scholar 

  83. Cowley, J. M. High-resolution electron-microscopy and microdiffraction. Ultramicroscopy 18, 11–17 (1985).

    Article  Google Scholar 

  84. Cowley, J. M. in Advances in Electronics and Electron Physics Vol. 46 1–53 (Elsevier, 1978).

  85. Ronchi, V. Forty years of history of a grating interferometer. Appl. Opt. 3, 437–451 (1964).

    Article  ADS  Google Scholar 

  86. Lupini, A. R. & Pennycook, S. J. Rapid autotuning for crystalline specimens from an inline hologram. J. Electron. Microsc. 57, 195–201 (2008).

    Google Scholar 

  87. Lupini, A. R., Wang, P., Nellist, P. D., Kirkland, A. I. & Pennycook, S. J. Aberration measurement using the Ronchigram contrast transfer function. Ultramicroscopy 110, 891–898 (2010).

    Article  Google Scholar 

  88. Sawada, H., Allen, C. S., Wang, S., Warner, J. H. & Kirkland, A. I. Aberration measurement of the probe-forming system of an electron microscope using two-dimensional materials. Ultramicroscopy 182, 195–204 (2017).

    Article  Google Scholar 

  89. Ramasse, Q. M. Twenty years after: how “aberration correction in the STEM” truly placed a “a synchrotron in a microscope”. Ultramicroscopy 180, 41–51 (2017).

    Article  Google Scholar 

  90. Ramasse, Q. M. & Bleloch, A. L. Diagnosis of aberrations from crystalline samples in scanning transmission electron microscopy. Ultramicroscopy 106, 37–56 (2005).

    Article  Google Scholar 

  91. Sawada, H. et al. Measurement method of aberration from Ronchigram by autocorrelation function. Ultramicroscopy 108, 1467–1475 (2008).

    Article  Google Scholar 

  92. Lin, J. A. & Cowley, J. M. Calibration of the operating parameters for an HB5 stem instrument. Ultramicroscopy 19, 31–42 (1986).

    Article  Google Scholar 

  93. Dwyer, C., Erni, R. & Etheridge, J. Method to measure spatial coherence of subångstrom electron beams. Appl. Phys. Lett. 93, 021115 (2008).

  94. Dwyer, C., Erni, R. & Etheridge, J. Measurement of effective source distribution and its importance for quantitative interpretation of STEM images. Ultramicroscopy 110, 952–957 (2010).

    Article  Google Scholar 

  95. Lupini, A. R., Chi, M. & Jesse, S. Rapid aberration measurement with pixelated detectors. J. Microsc. 263, 43–50 (2016).

    Article  Google Scholar 

  96. Lupini, A. R. et al. Fast aberration measurement in multi-dimensional STEM. Microsc. Microanal. 22, 252–253 (2016).

    Article  Google Scholar 

  97. Pennycook, T. J. et al. Efficient phase contrast imaging in STEM using a pixelated detector. Part 1: experimental demonstration at atomic resolution. Ultramicroscopy 151, 160–167 (2015).

    Article  Google Scholar 

  98. Krajnak, M. & Etheridge, J. A symmetry-derived mechanism for atomic resolution imaging. Proc. Natl Acad. Sci. USA 117, 27805–27810 (2020).

    Article  ADS  Google Scholar 

  99. Dekkers, N. H. & de Lqng, H. Differential phase contrast in a STEM. Optik 41, 452–456 (1974).

    Google Scholar 

  100. Lazić, I., Bosch, E. G. T. & Lazar, S. Phase contrast STEM for thin samples: integrated differential phase contrast. Ultramicroscopy 160, 265–280 (2016).

    Article  Google Scholar 

  101. Tsuda, K. & Tanaka, M. Direct observation of the symmetry breaking of the nanometer-scale local structure in the paraelectric cubic phase of BaTiO3 using convergent-beam electron diffraction. Appl. Phys. Express 9, 071501 (2016).

  102. Jesse, S. et al. Big data analytics for scanning transmission electron microscopy ptychography. Sci. Rep. 6, 26348 (2016).

    Article  ADS  Google Scholar 

  103. Nord, M. et al. Strain anisotropy and magnetic domains in embedded nanomagnets. Small 15, 6 (2019).

    Google Scholar 

  104. Pekin, T. C., Gammer, C., Ciston, J., Minor, A. M. & Ophus, C. Optimizing disk registration algorithms for nanobeam electron diffraction strain mapping. Ultramicroscopy 176, 170–176 (2017).

    Article  Google Scholar 

  105. Grieb, T. et al. Strain analysis from nano-beam electron diffraction: influence of specimen tilt and beam convergence. Ultramicroscopy 190, 45–57 (2018).

    Article  Google Scholar 

  106. Han, Y. M. et al. Strain mapping of two-dimensional heterostructures with subpicometer precision. Nano Lett. 18, 3746–3751 (2018).

    Article  ADS  Google Scholar 

  107. Nord, M. et al. Three-dimensional subnanoscale imaging of unit cell doubling due to octahedral tilting and cation modulation in strained perovskite thin films. Phys. Rev. Mater. 3, 063605 (2019).

    Article  Google Scholar 

  108. Hoppe, W. Trace structure analysis, ptychography, phase tomography. Ultramicroscopy 10, 187–198 (1982).

    Article  Google Scholar 

  109. Gerchbeg, R. W. & Saxton, W. O. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35, 227–246 (1972).

    Google Scholar 

  110. Cowley, J. M. Comments on ultra-high-resolution STEM. Ultramicroscopy 87, 1–4 (2001).

    Article  Google Scholar 

  111. Nellist, P. D. & Rodenburg, J. M. Beyond the conventional information limit — the relevant coherence function. Ultramicroscopy 54, 61–74 (1994).

    Article  Google Scholar 

  112. Rodenburg, J. M. & Bates, R. H. T. The theory of superresolution electron-microscopy via Wigner-distribution deconvolution. Phil. Trans. R. Soc. Lond. A 339, 521–553 (1992).

    Article  ADS  Google Scholar 

  113. Lupini, A. R., Oxley, M. P. & Kalinin, S. V. Pushing the limits of electron ptychography. Science 362, 399 (2018).

    Article  ADS  Google Scholar 

  114. Yang, H. et al. Simultaneous atomic-resolution electron ptychography and Z-contrast imaging of light and heavy elements in complex nanostructures. Nat. Commun. 7, 12532 (2016).

    Article  ADS  Google Scholar 

  115. Van Dyck, D., Jinschek, J. R. & Chen, F.-R. ‘Big Bang’ tomography as a new route to atomic-resolution electron tomography. Nature 486, 243–246 (2012).

    Article  ADS  Google Scholar 

  116. Verbeeck, J., Tian, H. & Schattschneider, P. Production and application of electron vortex beams. Nature 467, 301–304 (2010).

    Article  ADS  Google Scholar 

  117. McMorran, B. J. et al. Electron vortex beams with high quanta of orbital angular momentum. Science 331, 192–195 (2011).

    Article  ADS  Google Scholar 

  118. Grillo, V. et al. Generation of nondiffracting electron Bessel beams. Phys. Rev. X 4, 011013 (2014).

    Google Scholar 

  119. Uchida, M. & Tonomura, A. Generation of electron beams carrying orbital angular momentum. Nature 464, 737–739 (2010).

    Article  ADS  Google Scholar 

  120. Rotunno, E. et al. Electron-beam shaping in the transmission electron microscope: control of electron-beam propagation along atomic columns. Phys. Rev. Appl. 11, 044072 (2019).

    Article  ADS  Google Scholar 

  121. Nguyen, D. T., Findlay, S. D. & Etheridge, J. A menu of electron probes for optimising information from scanning transmission electron microscopy. Ultramicroscopy 184, 143–155 (2018).

    Article  Google Scholar 

  122. Zheng, C. L. et al. Axicon lens for electrons using a magnetic vortex: the efficient generation of a Bessel beam. Phys. Rev. Lett. 119, 174801 (2017).

    Article  ADS  Google Scholar 

  123. Verbeeck, J. et al. Demonstration of a 2×2 programmable phase plate for electrons. Ultramicroscopy 190, 58–65 (2018).

    Article  Google Scholar 

  124. Tavabi, A. H. et al. Generation of electron vortices using nonexact electric fields. Phys. Rev. Res. 2, 013185 (2020).

    Article  Google Scholar 

  125. Tavabi, A. H. et al. Experimental demonstration of an electrostatic orbital angular momentum sorter for electrons. Phys. Rev. Lett. 126 094802 (2021).

    Article  ADS  Google Scholar 

  126. Grillo, V. et al. Observation of nanoscale magnetic fields using twisted electron beams. Nat. Commun. 8, 689 (2017).

    Article  Google Scholar 

  127. Edstrom, A., Lubk, A. & Rusz, J. Quantum mechanical treatment of atomic-resolution differential phase contrast imaging of magnetic materials. Phys. Rev. B 99, 174428 (2019).

  128. Pohl, D. et al. Atom size electron vortex beams with selectable orbital angular momentum. Sci. Rep. 7, 934 (2017).

    Article  ADS  Google Scholar 

  129. Rusz, J. & Bhowmick, S. Boundaries for efficient use of electron vortex beams to measure magnetic properties. Phys. Rev. Lett. 111, 105504 (2013).

  130. Ophus, C. et al. Efficient linear phase contrast in scanning transmission electron microscopy with matched illumination and detector interferometry. Nat. Commun. 7, 10719 (2016).

    Article  ADS  Google Scholar 

  131. Guzzinati, G. et al. Probing the symmetry of the potential of localized surface plasmon resonances with phase-shaped electron beams. Nat. Commun. 8, 14999 (2017).

    Article  ADS  Google Scholar 

  132. Zeltmann, S. E. et al. Patterned probes for high precision 4D-STEM Bragg measurements. Ultramicroscopy 209, 112890 (2020).

    Article  Google Scholar 

  133. Guzzinati, G. et al. Electron Bessel beam diffraction for precise and accurate nanoscale strain mapping. Appl. Phys. Lett. 114, 243501 (2019).

  134. Stoger-Pollach, M., Schachinger, T., Biedermann, K. & Beyer, V. Valence EELS below the limit of inelastic delocalization using conical dark field EFTEM or Bessel beams. Ultramicroscopy 173, 24–30 (2017).

    Article  Google Scholar 

  135. Nellist, P. D. & Pennycook, S. J. Incoherent imaging using dynamically scattered coherent electrons. Ultramicroscopy 78, 111–124 (1999).

    Article  Google Scholar 

  136. Grillo, V. et al. Measuring the orbital angular momentum spectrum of an electron beam. Nat. Commun. 8, 15536 (2017).

    Article  ADS  Google Scholar 

  137. Rotunno, E. et al. Alignment of electron optical beam shaping elements using a convolutional neural network. Ultramicroscopy 228, 113338 (2021).

    Article  Google Scholar 

  138. Shahriari, B., Swersky, K., Wang, Z., Adams, R. P. & Freitas, N. D. Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104, 148–175 (2016).

    Article  Google Scholar 

  139. Silver, D. et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 1140–1144 (2018).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  140. Wilkinson, M. D. et al. The FAIR guiding principles for scientific data management and stewardship. Sci. Data 3, 160018 (2016).

    Article  Google Scholar 

  141. Collins, T. J. ImageJ for microscopy. Biotechniques 43, 1S (2007).

    Article  MathSciNet  Google Scholar 

  142. Roels, J. et al. An interactive ImageJ plugin for semi-automated image denoising in electron microscopy. Nat. Commun. 11, 771 (2020).

    Article  ADS  Google Scholar 

  143. Somnath, S. et al. USID and Pycroscopy–Open frameworks for storing and analyzing spectroscopic and imaging data. Preprint at https://arxiv.org/abs/1903.09515 (2019).

  144. Clausen, A. et al. LiberTEM: Software platform for scalable multidimensional data processing in transmission electron microscopy. J. Open. Source Softw. 5, 2006 (2020).

    Article  ADS  Google Scholar 

  145. Savitzky, B. H. et al. py4DSTEM: Open Source Software for 4D-STEM Data Analysis. Micros. Microanal. 25, 124–125 (2019).

    Article  Google Scholar 

  146. Koch, C. T. Determination of core structure periodicity and point defect density along dislocations. PhD thesis, Arizona State University https://www.physik.hu-berlin.de/en/sem/images/koch02_phdthesis.pdf (2002).

  147. Allen, L. J., D’Alfonso, A. J. & Findlay, S. D. Modelling the inelastic scattering of fast electrons. Ultramicroscopy 151, 11–22 (2015).

    Article  Google Scholar 

  148. Barthel, J. Dr. Probe: a software for high-resolution STEM image simulation. Ultramicroscopy 193, 1–11 (2018).

    Article  Google Scholar 

  149. Lobato, I. & Van Dyck, D. MULTEM: A new multislice program to perform accurate and fast electron diffraction and imaging simulations using graphics processing units with CUDA. Ultramicroscopy 156, 9–17 (2015).

    Article  Google Scholar 

  150. Oelerich, J. O. et al. STEMsalabim: a high-performance computing cluster friendly code for scanning transmission electron microscopy image simulations of thin specimens. Ultramicroscopy 177, 91–96 (2017).

    Article  Google Scholar 

  151. Madsen, J. & Susi, T. abTEM: ab initio transmission electron microscopy image simulation. Microsc. Microanal. 26, 448–450 (2020).

    Article  ADS  Google Scholar 

  152. Pryor, A., Ophus, C. & Miao, J. A streaming multi-GPU implementation of image simulation algorithms for scanning transmission electron microscopy. Adv. Struct. Chem. Imag. 3, 15 (2017).

    Article  Google Scholar 

  153. Mortensen, J. J., Hansen, L. B. & Jacobsen, K. W. Real-space grid implementation of the projector augmented wave method. Phys. Rev. B 71, 035109 (2005).

    Article  ADS  Google Scholar 

  154. Sakaguchi, N., Matsumoto, S., Kunisada, Y. & Ueda, M. Interaction of localized surface plasmons of a silver nanosphere dimer embedded in a uniform medium: scanning transmission electron microscopy electron energy-loss spectroscopy and discrete dipole approximation simulation. J. Phys. Chem. C 123, 6735–6744 (2019).

    Article  Google Scholar 

  155. Mendis, B. G. An inelastic multislice simulation method incorporating plasmon energy losses. Ultramicroscopy 206, 112816 (2019).

    Article  Google Scholar 

  156. Brown, H. G., Ciston, J. & Ophus, C. Linear-scaling algorithm for rapid computation of inelastic transitions in the presence of multiple electron scattering. Phys. Rev. Res. 1, 033186 (2019).

    Article  Google Scholar 

  157. Tessmer, J., Singh, S., Gu, Y., El-Awady, J. A. & Graef, M. D. Scanning transmission electron microscopy image simulations of complex dislocation structures generated by discrete dislocation dynamics. Ultramicroscopy 219, 113124 (2020).

    Article  Google Scholar 

  158. Thomas, P. J. & Midgley, P. A. Image-spectroscopy — II. The removal of plural scattering from extended energy-filtered series by Fourier deconvolution. Ultramicroscopy 88, 187–194 (2001).

    Article  Google Scholar 

  159. Leapman, R. D., Rez, P. & Mayers, D. F. K, L, and M shell generalized oscillator strengths and ionization cross sections for fast electron collisions. J. Chem. Phys. 72, 1232–1243 (1980).

    Article  ADS  Google Scholar 

  160. Ahn, C. C. & Rez, P. Inner shell edge profiles in electron energy loss spectroscopy. Ultramicroscopy 17, 105–115 (1985).

    Article  Google Scholar 

  161. Cueva, P., Hovden, R., Mundy, J. A., Xin, H. L. & Muller, D. A. Data processing for atomic resolution electron energy loss spectroscopy. Microsc. Microanal. 18, 667–675 (2012).

    Article  ADS  Google Scholar 

  162. Varela, M. et al. Atomic-resolution imaging of oxidation states in manganites. Phys. Rev. B 79, 085117 (2009).

    Article  ADS  Google Scholar 

  163. Verbeeck, J., Van Aert, S. & Bertoni, G. Model-based quantification of EELS spectra: Including the fine structure. Ultramicroscopy 106, 976–980 (2006).

    Article  Google Scholar 

  164. Bosman, M. et al. Two-dimensional mapping of chemical information at atomic resolution. Phys. Rev. Lett. 99, 086102 (2007).

    Article  ADS  Google Scholar 

  165. Bosman, M., Watanabe, M., Alexander, D. & Keast, V. Mapping chemical and bonding information using multivariate analysis of electron energy-loss spectrum images. Ultramicroscopy 106, 1024–1032 (2006).

    Article  Google Scholar 

  166. Thersleff, T. et al. Chemically and morphologically distinct grain boundaries in Ge-doped Cu2ZnSnSe4 solar cells revealed with STEM-EELS. Mater. Des. 122, 102–109 (2017).

    Article  Google Scholar 

  167. Potapov, P. & Lubk, A. Extraction of physically meaningful endmembers from STEM spectrum-images combining geometrical and statistical approaches. Micron 145, 103068 (2021).

    Article  Google Scholar 

  168. Bobynko, J., MacLaren, I. & Craven, A. J. Spectrum imaging of complex nanostructures using DualEELS: I. Digital extraction replicas. Ultramicroscopy 149, 9–20 (2015).

    Article  Google Scholar 

  169. Craven, A. J., Sala, B., Bobynko, J. & MacLaren, I. Spectrum imaging of complex nanostructures using DualEELS: II. Absolute quantification using standards. Ultramicroscopy 186, 66–81 (2018).

    Article  Google Scholar 

  170. Shiga, M. et al. Sparse modeling of EELS and EDX spectral imaging data by nonnegative matrix factorization. Ultramicroscopy 170, 43–59 (2016).

    Article  Google Scholar 

  171. Nicoletti, O. et al. Three-dimensional imaging of localized surface plasmon resonances of metal nanoparticles. Nature 502, 80–84 (2013).

    Article  ADS  Google Scholar 

  172. Lichtert, S. & Verbeeck, J. Statistical consequences of applying a PCA noise filter on EELS spectrum images. Ultramicroscopy 125, 35–42 (2013).

    Article  Google Scholar 

  173. LeBeau, J. M., Findlay, S. D., Allen, L. J. & Stemmer, S. Quantitative atomic resolution scanning transmission electron microscopy. Phys. Rev. Lett. 100, 4 (2008).

    Article  Google Scholar 

  174. LeBeau, J. M., Findlay, S. D., Allen, L. J. & Stemmer, S. Standardless atom counting in scanning transmission electron microscopy. Nano Lett. 10, 4405–4408 (2010).

    Article  ADS  Google Scholar 

  175. Katz-Boon, H. et al. Three-dimensional morphology and crystallography of gold nanorods. Nano Lett. 11, 273–278 (2011).

    Article  ADS  Google Scholar 

  176. Katz-Boon, H. et al. Stability of crystal facets in gold nanorods. Nano Lett. 15, 1635–1641 (2015).

    Article  ADS  Google Scholar 

  177. Erni, R., Heinrich, H. & Kostorz, G. Quantitative characterisation of chemical inhomogeneities in Al–Ag using high-resolution Z-contrast STEM. Ultramicroscopy 94, 125–133 (2003).

    Article  Google Scholar 

  178. Jones, L. Quantitative ADF STEM: acquisition, analysis and interpretation. in 14th European Workshop on Modern Developments and Applications in Microbeam Analysis, IOP Conference Series Materials Science and Engineering Vol. 109 (IOP Publishing, 2016).

  179. Dwyer, C. et al. Sub-0.1 nm-resolution quantitative scanning transmission electron microscopy without adjustable parameters. Appl. Phys. Lett. 100, 4 (2012).

    Article  Google Scholar 

  180. Ishikawa, R., Lupini, A. R., Findlay, S. D. & Pennycook, S. J. Quantitative annular dark field electron microscopy using single electron signals. Microsc. Microanal. 20, 99–110 (2014).

    Article  ADS  Google Scholar 

  181. Sang, X. H. & LeBeau, J. M. Characterizing the response of a scintillator-based detector to single electrons. Ultramicroscopy 161, 3–9 (2016).

    Article  Google Scholar 

  182. LeBeau, J. M. & Stemmer, S. Experimental quantification of annular dark-field images in scanning transmission electron microscopy. Ultramicroscopy 108, 1653–1658 (2008).

    Article  Google Scholar 

  183. LeBeau, J. M., Findlay, S. D., Allen, L. J. & Stemmer, S. Position averaged convergent beam electron diffraction: theory and applications. Ultramicroscopy 110, 118–125 (2010).

    Article  Google Scholar 

  184. Verbeeck, J., Beche, A. & Van den Broek, W. A holographic method to measure the source size broadening in STEM. Ultramicroscopy 120, 35–40 (2012).

    Article  Google Scholar 

  185. Maunders, C., Dwyer, C., Tiemeijer, P. C. & Etheridge, J. Practical methods for the measurement of spatial coherence — a comparative study. Ultramicroscopy 111, 1437–1446 (2011).

    Article  Google Scholar 

  186. Haider, M., Uhlemann, S. & Zach, J. Upper limits for the residual aberrations of a high-resolution aberration-corrected STEM. Ultramicroscopy 81, 163–175 (2000).

    Article  Google Scholar 

  187. Kuramochi, K. et al. Effect of chromatic aberration on atomic-resolved spherical aberration corrected STEM images. Ultramicroscopy 110, 36–42 (2009).

    Article  Google Scholar 

  188. Zheng, C. L. & Etheridge, J. Measurement of chromatic aberration in STEM and SCEM by coherent convergent beam electron diffraction. Ultramicroscopy 125, 49–58 (2013).

    Article  Google Scholar 

  189. Voyles, P. M., Grazul, J. L. & Muller, D. A. Imaging individual atoms inside crystals with ADF-STEM. Ultramicroscopy 96, 251–273 (2003).

    Article  Google Scholar 

  190. Sang, X. H., Oni, A. A. & LeBeau, J. M. Atom column indexing: atomic resolution image analysis through a matrix representation. Microsc. Microanal. 20, 1764–1771 (2014).

    Article  ADS  Google Scholar 

  191. Yankovich, A. B. et al. Picometre-precision analysis of scanning transmission electron microscopy images of platinum nanocatalysts. Nat. Commun. 4155 (2014).

  192. Voyles, P. M., Muller, D. A., Grazul, J. L., Citrin, P. H. & Gossmann, H. J. L. Atomic-scale imaging of individual dopant atoms and clusters in highly n-type bulk Si. Nature 416, 826–829 (2002).

    Article  ADS  Google Scholar 

  193. Ishikawa, R., Lupini, A. R., Findlay, S. D., Taniguchi, T. & Pennycook, S. J. Three-dimensional location of a single dopant with atomic precision by aberration-corrected scanning transmission electron microscopy. Nano Lett. 14, 1903–1908 (2014).

    Article  ADS  Google Scholar 

  194. Mevenkamp, N. et al. Poisson noise removal from high-resolution STEM images based on periodic block matching. Adv. Struct. Chem. Imag. 1, 19 (2015).

    Article  Google Scholar 

  195. Bals, S., Van Aert, S., Van Tendeloo, G. & Avila-Brande, D. Statistical estimation of atomic positions from exit wave reconstruction with a precision in the picometer range. Phys. Rev. Lett. 96, 4 (2006).

    Article  Google Scholar 

  196. De Backer, A., van den Bos, K. H. W., Van den Broek, W., Sijbers, J. & Van Aert, S. StatSTEM: an efficient approach for accurate and precise model-based quantification of atomic resolution electron microscopy images. Ultramicroscopy 171, 104–116 (2016).

    Article  Google Scholar 

  197. So, Y. G. & Kimoto, K. Effect of specimen misalignment on local structure analysis using annular dark-field imaging. J. Electron. Microsc. 61, 207–215 (2012).

    Article  Google Scholar 

  198. Schmid, H., Okunishi, E., Oikawa, T. & Mader, W. Structural and elemental analysis of iron and indium doped zinc oxide by spectroscopic imaging in Cs-corrected STEM. Micron 43, 49–56 (2012).

    Article  Google Scholar 

  199. Fan, L., Zhang, F., Fan, H. & Zhang, C. Brief review of image denoising techniques. Vis. Comput. Indust. Biomed. Art 2, 7 (2019).

    Article  Google Scholar 

  200. Ziatdinov, M. et al. Deep learning of atomically resolved scanning transmission electron microscopy images: chemical identification and tracking local transformations. ACS Nano 11, 12742–12752 (2017).

    Article  Google Scholar 

  201. Ziatdinov, M., Nelson, C., Vasudevan, R. K., Chen, D. Y. & Kalinin, S. V. Building ferroelectric from the bottom up: the machine learning analysis of the atomic-scale ferroelectric distortions. Appl. Phys. Lett. 115, 052902 (2019).

    Article  ADS  Google Scholar 

  202. Lin, R., Zhang, R., Wang, C., Yang, X.-Q. & Xin, H. L. TEMImageNet and AtomSegNet deep learning training library and models for high-precision atom segmentation, localization, denoising, and super-resolution processing of atom-resolution scanning TEM images. Preprint at https://arxiv.org/abs/2012.09093 (2020).

  203. Ziatdinov, M., Dyck, O., Jesse, S. & Kalinin, S. V. Atomic mechanisms for the Si atom dynamics in graphene: chemical transformations at the edge and in the bulk. Adv. Funct. Mater. 29, 1904480 (2019).

    Article  Google Scholar 

  204. Ziatdinov, M. et al. Building and exploring libraries of atomic defects in graphene: scanning transmission electron and scanning tunneling microscopy study. Sci. Adv. 5, eaaw8989 (2019).

    Article  ADS  Google Scholar 

  205. Lee, C. H. et al. Deep learning enabled strain mapping of single-atom defects in two-dimensional transition metal dichalcogenides with sub-picometer precision. Nano Lett. 20, 3369–3377 (2020).

    Article  ADS  Google Scholar 

  206. Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504 (2006).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  207. Roccapriore, K. M., Ziatdinov, M., Cho, S. H., Hachtel, J. A. & Kalinin, S. V. Predictability of localized plasmonic responses in nanoparticle assemblies. Small 17, 2100181 (2021).

    Article  Google Scholar 

  208. Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. Preprint at https://arxiv.org/abs/1312.6114 (2013).

  209. Kingma, D. P. & Welling, M. An introduction to variational autoencoders. Found. Trends Mach. Learn. 12, 307–392 (2019).

    Article  MATH  Google Scholar 

  210. Blei, D. M., Kucukelbir, A. & McAuliffe, J. D. Variational inference: a review for statisticians. J. Am. Stat. Assoc. 112, 859–877 (2017).

    Article  MathSciNet  Google Scholar 

  211. Muller, D. A. & Grazul, J. Optimizing the environment for sub-0.2 nm scanning transmission electron microscopy. J. Electron. Microsc. 50, 219–226 (2001).

    Google Scholar 

  212. Vonharrach, H. S. Instrumental factors in high-resolution FEG STEM. Ultramicroscopy 58, 1–5 (1995).

    Article  Google Scholar 

  213. Couillard, M., Radtke, G. & Botton, G. A. Strain fields around dislocation arrays in a Σ9 silicon bicrystal measured by scanning transmission electron microscopy. Phil. Mag. 93, 1250–1267 (2013).

    Article  ADS  Google Scholar 

  214. Isakozawa, S., Tomonaga, S., Hashimoto, T. & Baba, N. High-precision image-drift-correction method for EM images with a low signal-to-noise ratio. Microscopy 63, 301–312 (2014).

    Article  Google Scholar 

  215. Sang, X. H. & LeBeau, J. M. Revolving scanning transmission electron microscopy: correcting sample drift distortion without prior knowledge. Ultramicroscopy 138, 28–35 (2014).

    Article  Google Scholar 

  216. Dycus, J. H. et al. Accurate nanoscale crystallography in real-space using scanning transmission electron microscopy. Microsc. Microanal. 21, 946–952 (2015).

    Article  ADS  Google Scholar 

  217. Jones, L. et al. Smart Align — a new tool for robust non-rigid registration of scanning microscope data. Adv. Struct. Chem. Imag. 1, 8 (2015).

    Article  Google Scholar 

  218. Ophus, C., Ciston, J. & Nelson, C. T. Correcting nonlinear drift distortion of scanning probe and scanning transmission electron microscopies from image pairs with orthogonal scan directions. Ultramicroscopy 162, 1–9 (2016).

    Article  Google Scholar 

  219. Berkels, B. & Liebscher, C. H. Joint non-rigid image registration and reconstruction for quantitative atomic resolution scanning transmission electron microscopy. Ultramicroscopy 198, 49–57 (2019).

    Article  Google Scholar 

  220. Yankovich, A. B. et al. Non-rigid registration and non-local principal component analysis to improve electron microscopy spectrum images. Nanotechnology 27, 364001 (2016).

    Article  Google Scholar 

  221. Wang, Y. et al. Correcting the linear and nonlinear distortions for atomically resolved STEM spectrum and diffraction imaging. Microscopy 67, i114–i122 (2018).

    Article  Google Scholar 

  222. Barcena-Gonzalez, G. et al. CDrift: an algorithm to correct linear drift from a single high-resolution STEM image. Microsc. Microanal. 26, 913–920 (2020).

    Article  ADS  Google Scholar 

  223. Sanchez, A. M. et al. An approach to the systematic distortion correction in aberration-corrected HAADF images. J. Microsc. 221, 1–7 (2006).

    Article  MathSciNet  Google Scholar 

  224. Braidy, N., Le Bouar, Y., Lazar, S. & Ricolleau, C. Correcting scanning instabilities from images of periodic structures. Ultramicroscopy 118, 67–76 (2012).

    Article  Google Scholar 

  225. Ning, S. C. et al. Scanning distortion correction in STEM images. Ultramicroscopy 184, 274–283 (2018).

    Article  Google Scholar 

  226. Roccapriore, K. M., Ziatdinov, M. & Kalinin, S. V. Identification and correction of temporal and spatial distortions in scanning transmission electron microscopy. Ultramicroscopy 229, 113337 (2021).

    Article  Google Scholar 

  227. Sang, X. H. et al. Dynamic scan control in STEM: spiral scans. Adv. Struct. Chem. Imag. 2, 6 (2016).

  228. Li, X., Dyck, O., Kalinin, S. V. & Jesse, S. Compressed sensing of scanning transmission electron microscopy (STEM) with nonrectangular scans. Microsc. Microanal. 24, 623–633 (2018).

    Article  ADS  Google Scholar 

  229. Stevens, A. et al. A sub-sampled approach to extremely low-dose STEM. Appl. Phys. Lett. 112, 043104 (2018).

    Article  ADS  Google Scholar 

  230. Zobelli, A. et al. Spatial and spectral dynamics in STEM hyperspectral imaging using random scan patterns. Ultramicroscopy 212, 112912 (2020).

    Article  Google Scholar 

  231. Beche, A., Goris, B., Freitag, B. & Verbeeck, J. Development of a fast electromagnetic beam blanker for compressed sensing in scanning transmission electron microscopy. Appl. Phys. Lett. 108, 093103 (2016).

    Article  ADS  Google Scholar 

  232. Velazco, A., Nord, M., Beche, A. & Verbeeck, J. Evaluation of different rectangular scan strategies for STEM imaging. Ultramicroscopy 215, 113021 (2020).

  233. Dai, G. L. et al. Reference nano-dimensional metrology by scanning transmission electron microscopy. Meas. Sci. Technol. 24, 085001 (2013).

    Article  ADS  Google Scholar 

  234. Radaelli, P. G. & Cheong, S. W. Structural phenomena associated with the spin-state transition in LaCoO3. Phys. Rev. B 66, 094408 (2002).

    Article  ADS  Google Scholar 

  235. Zhou, J. S. & Goodenough, J. B. Chemical bonding and electronic structure of RNiO3 (R = rare earth). Phys. Rev. B 69, 153105 (2004).

    Article  ADS  Google Scholar 

  236. Zhou, J. S. & Goodenough, J. B. Universal octahedral-site distortion in orthorhombic perovskite oxides. Phys. Rev. Lett. 94, 065501 (2005).

    Article  ADS  Google Scholar 

  237. Taraci, J. L. et al. Strain mapping in nanowires. Nanotechnology 16, 2365–2371 (2005).

    Article  ADS  Google Scholar 

  238. Hytch, M. J., Putaux, J. L. & Thibault, J. Stress and strain around grain-boundary dislocations measured by high-resolution electron microscopy. Phil. Mag. 86, 4641–4656 (2006).

    Article  ADS  Google Scholar 

  239. Chu, M. W., Szafraniak, I., Hesse, D., Alexe, M. & Gosele, U. Elastic coupling between 90° twin walls and interfacial dislocations in epitaxial ferroelectric perovskites: a quantitative high-resolution transmission electron microscopy study. Phys. Rev. B 72, 174112 (2005).

  240. Zhao, C. W., Xing, Y. M., Zhou, C. E. & Bai, P. C. Experimental examination of displacement and strain fields in an edge dislocation core. Acta Mater. 56, 2570–2575 (2008).

    Article  ADS  Google Scholar 

  241. Pennycook, S. J. & Nellist, P. D. Scanning Transmission Electron Microscopy: Imaging and Analysis (Springer Science & Business Media, 2011).

  242. Sousa, A. A. & Leapman, R. D. Development and application of STEM for the biological sciences. Ultramicroscopy 123, 38–49 (2012).

    Article  Google Scholar 

  243. Tagantsev, A. K., Cross, L. E. & Fousek, J. Domains in Ferroic Crystals and Thin Films (Springer, 2010).

  244. Salje, E. K. H. Ferroelasticity. Contemp. Phys. 41, 79–91 (2000).

    Article  ADS  Google Scholar 

  245. Pan, X. Q., Kaplan, W. D., Ruhle, M. & Newnham, R. E. Quantitative comparison of transmission electron microscopy techniques for the study of localized ordering on a nanoscale. J. Am. Ceram. Soc. 81, 597–605 (1998).

    Article  Google Scholar 

  246. Jia, C. L. et al. Unit-cell scale mapping of ferroelectricity and tetragonality in epitaxial ultrathin ferroelectric films. Nat. Mater. 6, 64–69 (2007).

    Article  ADS  Google Scholar 

  247. Jia, C. L. et al. Atomic-scale study of electric dipoles near charged and uncharged domain walls in ferroelectric films. Nat. Mater. 7, 57–61 (2008).

    Article  ADS  Google Scholar 

  248. Chisholm, M. F., Luo, W. D., Oxley, M. P., Pantelides, S. T. & Lee, H. N. Atomic-scale compensation phenomena at polar interfaces. Phys. Rev. Lett. 105, 197602 (2010).

  249. Jia, C. L. et al. Oxygen octahedron reconstruction in the SrTiO3/LaAlO3 heterointerfaces investigated using aberration-corrected ultrahigh-resolution transmission electron microscopy. Phys. Rev. B 79, 081405 (2009).

  250. Borisevich, A. et al. Mapping octahedral tilts and polarization across a domain wall in BiFeO3 from Z-contrast scanning transmission electron microscopy image atomic column shape analysis. ACS Nano 4, 6071–6079 (2010).

    Article  Google Scholar 

  251. Kim, Y. M. et al. Interplay of octahedral tilts and polar order in BiFeO3 Films. Adv. Mater. 25, 2497–2504 (2013).

    Article  Google Scholar 

  252. He, Q. et al. Towards 3D Mapping of BO6 octahedron rotations at perovskite heterointerfaces, unit cell by unit cell. ACS Nano 9, 8412–8419 (2015).

    Article  Google Scholar 

  253. Kim, Y. M. et al. Direct observation of ferroelectric field effect and vacancy-controlled screening at the BiFeO3/LaxSr1-xMnO3 interface. Nat. Mater. 13, 1019–1025 (2014).

    Article  ADS  Google Scholar 

  254. Borisevich, A. Y. et al. Interface dipole between two metallic oxides caused by localized oxygen vacancies. Phys. Rev. B 86, 140102 (2012).

  255. Borisevich, A. Y. et al. Exploring mesoscopic physics of vacancy-ordered systems through atomic scale observations of topological defects. Phys. Rev. Lett. 109, 065702 (2012).

  256. Li, Q. et al. Quantification of flexoelectricity in PbTiO3/SrTiO3 superlattice polar vortices using machine learning and phase-field modeling. Nat. Commun. 8, 1468 (2017).

  257. Nelson, C. T. et al. Exploring physics of ferroelectric domain walls via Bayesian analysis of atomically resolved STEM data. Nat. Commun. 11, 6361 (2020).

    Article  ADS  Google Scholar 

  258. Rossell, M. D. et al. Atomic structure of highly strained BiFeO3 thin films. Phys. Rev. Lett. 108, 5 (2012).

    Article  Google Scholar 

  259. Campanini, M., Erni, R., Yang, C. H., Ramesh, R. & Rossell, M. D. Periodic giant polarization gradients in doped BiFeO3 thin films. Nano Lett. 18, 717–724 (2018).

    Article  ADS  Google Scholar 

  260. Campanini, M. et al. Atomic-resolution differential phase contrast STEM on ferroelectric materials: a mean-field approach. Phys. Rev. B 101, 184116 (2020).

    Article  ADS  Google Scholar 

  261. Midgley, P. A. & Dunin-Borkowski, R. E. Electron tomography and holography in materials science. Nat. Mater. 8, 271–280 (2009).

    Article  ADS  Google Scholar 

  262. Welborn, M., Tang, W. J., Ryu, J., Petkov, V. & Henkelman, G. A combined density functional and x-ray diffraction study of Pt nanoparticle structure. J. Chem. Phys. 135, 014503 (2011).

    Article  ADS  Google Scholar 

  263. Meredig, B. & Wolverton, C. A hybrid computational-experimental approach for automated crystal structure solution. Nat. Mater. 12, 123–127 (2013).

    Article  ADS  Google Scholar 

  264. Alkauskas, A., McCluskey, M. D. & Van de Walle, C. G. Tutorial: defects in semiconductors — combining experiment and theory. J. Appl. Phys. 119, 181101 (2016).

    Article  ADS  Google Scholar 

  265. Butler, K. T., Davies, D. W., Cartwright, H., Isayev, O. & Walsh, A. Machine learning for molecular and materials science. Nature 559, 547–555 (2018).

    Article  ADS  Google Scholar 

  266. Kolb, B., Lentz, L. C. & Kolpak, A. M. Discovering charge density functionals and structure-property relationships with PROPhet: a general framework for coupling machine learning and first-principles methods. Sci. Rep. 7, 1192 (2017).

    Article  ADS  Google Scholar 

  267. Schuett, O. & VandeVondele, J. Machine learning adaptive basis sets for efficient large scale density functional theory simulation. J. Chem. Theory Comput. 14, 4168–4175 (2018).

    Article  Google Scholar 

  268. Sen, F. G. et al. Sen, F. G. et al. Atomistic Simulations of Grain Boundaries in CdTe in 2015 IEEE 42nd Photovoltaic Specialists Conference (IEEE, 2015).

  269. Schwenker, E. et al. An autonomous microscopy workflow for structure determination from atomic-resolution images. Microsc. Microanal. 24, 510–511 (2018).

    Article  Google Scholar 

  270. Wei, J. et al. Direct imaging of atomistic grain boundary migration. Nat. Mater. 20, 951–955 (2021).

    Article  ADS  Google Scholar 

  271. Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).

    Article  ADS  Google Scholar 

  272. Liu, T.-J., Lin, Y.-C., Lin, W. & Kuo, C. C. J. Visual quality assessment: recent developments, coding applications and future trends. APSIPA Trans. Signal. Inf. Process. 2, e4 (2013).

    Article  Google Scholar 

  273. Lin, J. Y., Liu, T., Wu, E. C. & Kuo, C. J. in Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific 1–5 (IEEE, 2014).

  274. Guo, J. L. et al. Effect of selenium and chlorine co-passivation in polycrystalline CdSeTe devices. Appl. Phys. Lett. 115, 50 (2019).

    Article  Google Scholar 

  275. Krivanek, O. L. et al. Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy. Nature 464, 571–574 (2010).

    Article  ADS  Google Scholar 

  276. Treacy, M. M. J. Z dependence of electron scattering by single atoms into annular dark-field detectors. Microsc. Microanal. 17, 847–858 (2011).

    Article  ADS  Google Scholar 

  277. Yamashita, S. et al. Atomic number dependence of Z contrast in scanning transmission electron microscopy. Sci. Rep. 8, 12325 (2018).

    Article  ADS  Google Scholar 

  278. Li, H. et al. Atomic structure and dynamics of single platinum atom interactions with monolayer MoS2. ACS Nano 11, 3392–3403 (2017).

    Article  Google Scholar 

  279. Wang, S. et al. Preferential Pt nanocluster seeding at grain boundary dislocations in polycrystalline monolayer MoS2. ACS Nano 12, 5626–5636 (2018).

    Article  Google Scholar 

  280. Zan, R., Bangert, U., Ramasse, Q. & Novoselov, K. S. Metal−graphene interaction studied via atomic resolution scanning transmission electron microscopy. Nano Lett. 11, 1087–1092 (2011).

    Article  ADS  Google Scholar 

  281. Wang, S. et al. In situ atomic-scale studies of the formation of epitaxial Pt nanocrystals on monolayer molybdenum disulfide. ACS Nano 11, 9057–9067 (2017).

    Article  Google Scholar 

  282. Li, X. et al. Precursor design for high density single pt atom sites on MoS2: enhanced stability at elevated temperatures and reduced 3D clustering. Chem. Mater. 32, 2541–2551 (2020).

    Article  Google Scholar 

  283. Tai, K.-L. et al. Phase variations and layer epitaxy of 2D PdSe2 grown on 2D monolayers by direct selenization of molecular Pd precursors. ACS Nano 14, 11677–11690 (2020).

    Article  Google Scholar 

  284. Sinha, S. et al. Atomic structure and defect dynamics of monolayer lead iodide nanodisks with epitaxial alignment on graphene. Nat. Commun. 11, 823 (2020).

    Article  ADS  Google Scholar 

  285. Poh, S. M. et al. Molecular beam epitaxy of highly crystalline MoSe2 on hexagonal boron nitride. ACS Nano 12, 7562–7570 (2018).

    Article  Google Scholar 

  286. Gallagher, J. C. et al. Epitaxial growth of iridate pyrochlore Nd2Ir2O7 films. Sci. Rep. 6, 22282 (2016).

    Article  ADS  Google Scholar 

  287. Gerkman, M. A. et al. Direct imaging of individual molecular binding to clean nanopore edges in 2D monolayer MoS2. ACS Nano 14, 153–165 (2020).

    Article  Google Scholar 

  288. Cai, C., Han, S., Wang, Q. & Gu, M. Direct observation of yolk–shell transforming to gold single atoms and clusters with superior oxygen evolution reaction efficiency. ACS Nano 13, 8865–8871 (2019).

    Article  Google Scholar 

  289. Sun, Y. et al. Subunit cell–level measurement of polarization in an individual polar vortex. Sci. Adv. 5, eaav4355 (2019).

    Article  ADS  Google Scholar 

  290. Ohyama, J. et al. Atomic-scale insight into the structural effect of a supported Au catalyst based on a size-distribution analysis using Cs-STEM and morphological image-processing. J. Catal. 335, 24–35 (2016).

    Article  Google Scholar 

  291. Melzer, D. et al. Atomic-scale determination of active facets on the MoVTeNb oxide M1 phase and their intrinsic catalytic activity for ethane oxidative dehydrogenation. Angew. Chem. Int. Ed. 55, 8873–8877 (2016).

    Article  Google Scholar 

  292. Qiao, B. et al. Single-atom catalysis of CO oxidation using Pt1/FeOx. Nat. Chem. 3, 634–641 (2011).

    Article  Google Scholar 

  293. Liu, J. et al. In situ tracing of atom migration in Pt/NiPt hollow spheres during catalysis of CO oxidation. Chem. Commun. 50, 1804–1807 (2014).

    Article  Google Scholar 

  294. Gerkman, M. A., Sinha, S., Warner, J. H. & Han, G. G. D. Direct imaging of photoswitching molecular conformations using individual metal atom markers. ACS Nano 13, 87–96 (2019).

    Article  Google Scholar 

  295. Lee, J. K. et al. Metal atom markers for imaging epitaxial molecular self-assembly on graphene by scanning transmission electron microscopy. ACS Nano 13, 7252–7260 (2019).

    Article  Google Scholar 

  296. Kengmana, E. S., Lee, J. K., Li, X., Warner, J. H. & Han, G. G. D. Self-assembly of Bowlic supramolecules on graphene imaged at the individual molecular level using heavy atom tagging. Small 16, 2002860 (2020).

    Article  Google Scholar 

  297. Henninen, T. R., Bon, M., Wang, F., Passerone, D. & Erni, R. The structure of sub-nm platinum clusters at elevated temperatures. Angew. Chem. Int. Ed. 59, 839–845 (2020).

    Article  Google Scholar 

  298. Furnival, T., Leary, R. K. & Midgley, P. A. Denoising time-resolved microscopy image sequences with singular value thresholding. Ultramicroscopy 178, 112–124 (2017).

    Article  Google Scholar 

  299. Wang, F., Henninen, T. R., Keller, D. & Erni, R. Noise2Atom: unsupervised denoising for scanning transmission electron microscopy images. Appl. Microsc. 50, 23 (2020).

    Article  Google Scholar 

  300. Mirman, R. Point Groups, Space Groups, Crystals, Molecules (World Scientific, 1999).

  301. Glinchuk, M. D. & Stephanovich, V. A. Dynamic properties of relaxor ferroelectrics. J. Appl. Phys. 85, 1722–1726 (1999).

    Article  ADS  Google Scholar 

  302. Vugmeister, B. E. Polarization dynamics and formation of polar nanoregions in relaxor ferroelectrics. Phys. Rev. B 73, 174117 (2006).

  303. Takenaka, H., Grinberg, I., Liu, S. & Rappe, A. M. Slush-like polar structures in single-crystal relaxors. Nature 546, 391–395 (2017).

    Article  ADS  Google Scholar 

  304. Binder, K. & Reger, J. D. Theory of orientational glasses models, concepts, simulations. Adv. Phys. 41, 547–627 (1992).

    Article  ADS  Google Scholar 

  305. Binder, K. & Young, A. P. Spin-glasses — experimental facts, theoretical concepts, and open questions. Rev. Mod. Phys. 58, 801–976 (1986).

    Article  ADS  Google Scholar 

  306. Dagotto, E. Complexity in strongly correlated electronic systems. Science 309, 257–262 (2005).

    Article  ADS  Google Scholar 

  307. Dagotto, E., Hotta, T. & Moreo, A. Colossal magnetoresistant materials: the key role of phase separation. Phys. Rep. Rev. Sec. Phys. Lett. 344, 1–153 (2001).

    Google Scholar 

  308. Blinc, R. et al. Local polarization distribution and Edwards–Anderson order parameter of relaxor ferroelectrics. Phys. Rev. Lett. 83, 424–427 (1999).

    Article  ADS  Google Scholar 

  309. Cross, L. E. Relaxor ferroelectrics. Ferroelectrics 76, 241–267 (1987).

    Article  Google Scholar 

  310. Cliffe, M. J. et al. Structural simplicity as a restraint on the structure of amorphous silicon. Phys. Rev. B 95, 224108 (2017).

  311. Keen, D. A. & Goodwin, A. L. The crystallography of correlated disorder. Nature 521, 303–309 (2015).

    Article  ADS  Google Scholar 

  312. Cheetham, A. K., Bennett, T. D., Coudert, F. X. & Goodwin, A. L. Defects and disorder in metal organic frameworks. Dalton Trans. 45, 4113–4126 (2016).

    Article  Google Scholar 

  313. Dobigeon, N. & Brun, N. Spectral mixture analysis of EELS spectrum-images. Ultramicroscopy 120, 25–34 (2012).

    Article  Google Scholar 

  314. Pennycook, S. J., Varela, M., Lupini, A. R., Oxley, M. P. & Chisholm, M. F. Atomic-resolution spectroscopic imaging: past, present and future. J. Electron. Microsc. 58, 87–97 (2009).

    Article  Google Scholar 

  315. Jiang, Y. et al. Electron ptychography of 2D materials to deep sub-ångstrom resolution. Nature 559, 343–349 (2018).

    Article  Google Scholar 

  316. Belianinov, A. et al. Identification of phases, symmetries and defects through local crystallography. Nat. Commun. 6, 7801 (2015).

  317. Ziatdinov, M. et al. Atomic-scale observation of structural and electronic orders in the layered compound α-RuCl3. Nat. Commun. 7, 13774 (2016).

  318. He, Q., Woo, J., Belianinov, A., Guliants, V. V. & Borisevich, A. Y. Better catalysts through microscopy: mesoscale M1/M2 intergrowth in molybdenum-vanadium based complex oxide catalysts for propane ammoxidation. ACS Nano 9, 3470–3478 (2015).

    Article  Google Scholar 

  319. Wu, Z. et al. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32, 4–24 (2021).

    Article  MathSciNet  Google Scholar 

  320. Kalinin, S. V., Dyck, O., Ghosh, A., Sumpter, B. G. & Ziatdinov, M. Unsupervised machine learning discovery of chemical transformation pathways from atomically-resolved imaging data. Preprint at https://arxiv.org/abs/2010.09196 (2020).

  321. Wicks, S. et al. Collective dynamics in nanostructured polycrystalline ferroelectric thin films using local time-resolved measurements and switching spectroscopy. Acta Mater. 58, 67–75 (2010).

    Article  ADS  Google Scholar 

  322. Vlcek, L., Maksov, A., Pan, M., Vasudevan, R. K. & Kalinin, S. V. Knowledge extraction from atomically resolved images. ACS Nano 11, 10313–10320 (2017).

    Article  Google Scholar 

  323. Landau, L. D. & Lifshitz, E. M. Statistical Physics (Elsevier Science, 2013).

  324. Kitaev, A. Anyons in an exactly solved model and beyond. Ann. Phys. 321, 2–111 (2006).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  325. Vlcek, L., Yang, S., Ziatdinov, M., Kalinin, S. & Vasudevan, R. Statistical physics-based framework and Bayesian inference for model selection and uncertainty quantification. Microsc. Microanal. 25, 130–131 (2019).

    Article  ADS  Google Scholar 

  326. Mani Prudhvi Valleti, S., Vlcek, L., Vasudevan, R. K. & Kalinin, S. V. Inversion of lattice models from the observations of microscopic degrees of freedom: parameter estimation with uncertainty quantification. Preprint at https://arxiv.org/abs/1909.09244 (2019).

  327. Chakraborty, M. et al. Reconstruction of the interatomic forces from dynamic scanning transmission electron microscopy data. J. Appl. Phys. 127, 224301 (2020).

    Article  ADS  Google Scholar 

  328. Nobakht, A. Y. et al. Reconstruction of effective potential from statistical analysis of dynamic trajectories. AIP Adv. 10, 065034 (2020).

  329. Schepman, A. M. H., Vandervoort, J. A. P., Kramer, J. & Mellema, J. E. Getting started with a scanning-transmission electron-microscope coupled to a small computer. Ultramicroscopy 3, 265–269 (1978).

    Article  Google Scholar 

  330. Strahm, M. & Butler, J. H. Fast digital data acquisition and online processing system for an HB5 scanning-transmission electron-microscope. Rev. Sci. Instrum. 52, 840–848 (1981).

    Article  ADS  Google Scholar 

  331. Andree, P. J., Mellema, J. E. & Ruigrok, R. W. H. Discrimination of heavy and light-elements in a specimen by use of scanning-transmission electron-microscopy. Ultramicroscopy 17, 237–241 (1985).

    Article  Google Scholar 

  332. Haider, M., Epstein, A., Jarron, P. & Boulin, C. A versatile, software configurable multichannel stem detector for angle-resolved imaging. Ultramicroscopy 54, 41–59 (1994).

    Article  Google Scholar 

  333. Yang, Y. S. et al. Deciphering chemical order/disorder and material properties at the single-atom level. Nature 542, 75–79 (2017).

    Article  ADS  Google Scholar 

  334. Kalinin, S. V., S., B. G. & Archibald, R. K. Big–deep–smart data in imaging for guiding materials design. Nat. Mater. 14, 973–980 (2015).

    Article  ADS  Google Scholar 

  335. Spurgeon, S. R. et al. Towards data-driven next-generation transmission electron microscopy. Nat. Mater. 20, 274–279 (2021).

  336. Vasilevsky, N. A., Minnier, J., Haendel, M. A. & Champieux, R. E. Reproducible and reusable research: are journal data sharing policies meeting the mark? PeerJ 5, e3208 (2017).

    Article  Google Scholar 

  337. Molloy, J. C. The open knowledge foundation: open data means better science. PLoS Biol. 9, e1001195 (2011).

    Article  Google Scholar 

  338. Woelfle, M., Olliaro, P. & Todd, M. H. Open science is a research accelerator. Nat. Chem. 3, 745–748 (2011).

    Article  Google Scholar 

  339. Hanisch, R. J., Gilmore, I. S. & Plant, A. L. Improving reproducibility in research: the role of measurement science. J. Res. Natl Inst. Stand. Technol. 124, 124024 (2019).

    Article  Google Scholar 

  340. Blaiszik, B. et al. The materials data facility: data services to advance materials science research. JoM 68, 2045–2052 (2016).

    Article  Google Scholar 

  341. Jain, A. et al. Commentary: the Materials Project: a materials genome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).

  342. Iudin, A., Korir, P. K., Salavert-Torres, J., Kleywegt, G. J. & Patwardhan, A. EMPIAR: a public archive for raw electron microscopy image data. Nat. Methods 13, 387–388 (2016).

    Article  Google Scholar 

  343. Berman, H., Henrick, K. & Nakamura, H. Announcing the worldwide Protein Data Bank. Nat. Struct. Biol. 10, 980–980 (2003).

    Article  Google Scholar 

  344. Guo, S. et al. Direct mapping of ion diffusion times on LiCoO2 surfaces with nanometer resolution. J. Electrochem. Soc. 158, A982–A990 (2011).

    Article  Google Scholar 

  345. Nikiforov, M. P. et al. Temperature-dependent phase transitions in zeptoliter volumes of a complex biological membrane. Nanotechnology 22, 055709 (2011).

  346. Kalinin, S. V. et al. Big, deep, and smart data in scanning probe microscopy. ACS Nano 10, 9068–9086 (2016).

    Article  Google Scholar 

  347. Komsa, H. P. et al. Two-dimensional transition metal dichalcogenides under electron irradiation: defect production and doping. Phys. Rev. Lett. 109, 035503 (2012).

  348. Mishra, R., Ishikawa, R., Lupini, A. R. & Pennycook, S. J. Single-atom dynamics in scanning transmission electron microscopy. MRS Bull. 42, 644–652 (2017).

    Article  ADS  Google Scholar 

  349. Toma Susi, J. C. M. & Kotakoski, J. Manipulating low-dimensional materials down to the level of single atoms with electron irradiation. Ultramicroscopy 180, 163–172 (2017).

    Article  Google Scholar 

  350. Dyck, O. et al. Building structures atom by atom via electron beam manipulation. Small 14, 1801771 (2018).

  351. Dyck, O., Jesse, S. & Kalinin, S. V. A self-driving microscope and the atomic forge. MRS Bull. 44, 669–670 (2019).

    Article  ADS  Google Scholar 

  352. Kelley, K. P. et al. Fast scanning probe microscopy via machine learning: non-rectangular scans with compressed sensing and Gaussian process optimization. Small 16, 2002878 (2020).

    Article  MathSciNet  Google Scholar 

  353. Stevens, A. et al. Applying compressive sensing to TEM video: a substantial frame rate increase on any camera. Adv. Struct. Chem. Imag. 1, 10 (2015).

  354. Kovarik, L., Stevens, A., Liyu, A. & Browning, N. D. Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution STEM imaging. Appl. Phys. Lett. 109, 164102 (2016).

  355. Jesse, S. et al. Direct atomic fabrication and dopant positioning in Si using electron beams with active real-time image-based feedback. Nanotechnology 29, 255303 (2018).

  356. Gulrajani, I. & Lopez-Paz, D. In search of lost domain generalization. Preprint at https://arxiv.org/abs/2007.01434 (2020).

  357. Kalinin, S. V. et al. Automated and autonomous experiments in electron and scanning probe microscopy. ACS Nano 15, 12604–12627 (2021).

    Article  Google Scholar 

  358. Wilson, A. G., Hu, Z., Salakhutdinov, R. & Xing, E. P. in Artificial Intelligence and Statistics 370–378 (PMLR, 2021).

  359. Zheng, H. M. et al. Observation of transient structural-transformation dynamics in a Cu2S nanorod. Science 333, 206–209 (2011).

    Article  ADS  Google Scholar 

  360. Klie, R. F., Ito, Y., Stemmer, S. & Browning, N. S. Observation of oxygen vacancy ordering and segregation in perovskite oxides. Ultramicroscopy 86, 289–302 (2001).

    Article  Google Scholar 

  361. Jang, J. H. et al. In situ observation of oxygen vacancy dynamics and ordering in the epitaxial LaCoO3 system. ACS Nano 11, 6942–6949 (2017).

    Article  Google Scholar 

  362. Hoehl, D. et al. Temperature-dependence of electron-beam induced epitaxial crystallization of silicon. Phys. Status Solidi A 122, K35–K37 (1990).

    Article  ADS  Google Scholar 

  363. Jencic, I., Bench, M. W., Robertson, I. M. & Kirk, M. A. Electron-beam-induced crystallization of isolated amorphous regions in Si, Ge, GaP, and GaAs. J. Appl. Phys. 78, 974–982 (1995).

    Article  ADS  Google Scholar 

  364. Remeika, M. & Bezryadin, A. Sub-10 nanometre fabrication: molecular templating, electron-beam sculpting and crystallization of metallic nanowires. Nanotechnology 16, 1172–1176 (2005).

    Article  ADS  Google Scholar 

  365. Bae, I. T., Zhang, Y. W., Weber, W. J., Higuchi, M. & Giannuzzi, L. A. Electron-beam induced recrystallization in amorphous apatite. Appl. Phys. Lett. 90, 021912 (2007).

  366. Zhang, Y. et al. Ion-induced damage accumulation and electron-beam-enhanced recrystallization in SrTiO3. Phys. Rev. B 72, 094112 (2005).

  367. Huang, P. Y. et al. Imaging atomic rearrangements in two-dimensional silica glass: watching Silica’s Dance. Science 342, 224–227 (2013).

    Article  ADS  Google Scholar 

  368. Kurasch, S. et al. Atom-by-atom observation of grain boundary migration in graphene. Nano Lett. 12, 3168–3173 (2012).

    Article  ADS  Google Scholar 

  369. Yang, Z. Q. et al. Direct observation of atomic dynamics and silicon doping at a topological defect in graphene. Angew. Chem. Int. Edit. 53, 8908–8912 (2014).

    Article  Google Scholar 

  370. Krasheninnikov, A. V. & Banhart, F. Engineering of nanostructured carbon materials with electron or ion beams. Nat. Mater. 6, 723–733 (2007).

    Article  ADS  Google Scholar 

  371. Lingerfelt, D. B. et al. Nonadiabatic effects on defect diffusion in silicon-doped nanographenes. Nano Lett. 21, 236–242 (2021).

    Article  ADS  Google Scholar 

  372. Susi, T., Meyer, J. C. & Kotakoski, J. Quantifying transmission electron microscopy irradiation effects using two-dimensional materials. Nat. Rev. Phys. 1, 397–405 (2019).

    Article  Google Scholar 

  373. Krasheninnikov, A. V. & Nordlund, K. Ion and electron irradiation-induced effects in nanostructured materials. J. Appl. Phys. 107, 071301 (2010).

  374. Egerton, R. F. Radiation damage to organic and inorganic specimens in the TEM. Micron 119, 72–87 (2019).

    Article  Google Scholar 

  375. Jesse, S. et al. Atomic-level sculpting of crystalline oxides: toward bulk nanofabrication with single atomic plane precision. Small 11, 5895–5900 (2015).

    Article  Google Scholar 

  376. Mousavi, S. S., Schukat, M. & Howley, E. in Proc. of SAI Intelligent Systems Conf. 426–440 (Springer, 2021).

  377. Schrittwieser, J. et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588, 604–609 (2020).

    Article  ADS  Google Scholar 

  378. Wagner, T. & Raunser, S. The evolution of SPHIRE-crYOLO particle picking and its application in automated cryo-EM processing workflows. Commun. Biol. 3, 61 (2020).

    Article  Google Scholar 

  379. Kalinin, S. V., Dyck, O., Jesse, S. & Ziatdinov, M. Exploring order parameters and dynamic processes in disordered systems via variational autoencoders. Sci. Adv. 7, eabd5084 (2021).

    Article  ADS  Google Scholar 

Download references

Acknowledgements

This work is based upon work supported by the US Department of Energy (DOE), Office of Science, Basic Energy Sciences (BES), Materials Sciences and Engineering Division (S.V.K., M.P.O. and A.R.L.) and was performed and partially supported (M.Z.) at Oak Ridge National Laboratory’s Center for Nanophase Materials Sciences (CNMS), a US Department of Energy, Office of Science User Facility. V.G. acknowledges the support of the European Union Horizon 2020 Research and Innovation Programme under grant agreement no. 766970 Q-SORT (H2020-FETOPEN-1-2016-2017), no. 964591 Smart-electrons (H2020-FETOPEN-1-2016-2017) and no. 101035013 MINEON (H2020-FETOPEN-03-2018-2019-2020 – FET Innovation Launchpad). SuperSTEM (D.K.) is the UK National Facility for Advanced Electron Microscopy funded by the Engineering and Physical Sciences Research Council (EPSRC). P.M.V. acknowledges support from DOE BES (DE-FG02-08ER46547). G.G.D.H. and X.L. acknowledge support from Brandeis NSF MRSEC, Bioinspired Soft Materials, DMR-2011846. Work at the Molecular Foundry (C.O.) was supported by the Office of Science, Office of Basic Energy Sciences, of the US Department of Energy under contract no. DE-AC02-05CH11231. E.S. and M.K.Y.C. acknowledges support from the Center for Nanoscale Materials (DOE SUF) under contract no. DE-AC02-06CH11357. C.O. and M.K.Y.C. acknowledge support from DOE Early Career Research Awards. N.S. acknowledges support from the JSPS KAKENHI (grants 20H05659 and 19H05788). J.E. acknowledges the support of the Australian Research Council Discovery Project (grants DP150104483 and DP160104679). The authors are extremely grateful to K. More (from CNMS) for careful reading and editing of the manuscript.

Author information

Authors and Affiliations

  1. Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Sergei V. Kalinin, Andrew R. Lupini, Mark P. Oxley & Maxim Ziatdinov

  2. NCEM, Molecular Foundry, Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    Colin Ophus

  3. Department of Materials Science and Engineering, University of Wisconsin-Madison, Madison, WI, USA

    Paul M. Voyles

  4. Electron Microscopy Center, EMPA – Swiss Federal Laboratories for Materials Science and Technology, Dubendorf, Switzerland

    Rolf Erni

  5. SuperSTEM laboratory, Sci-Tech Daresbury Campus, Daresbury, UK

    Demie Kepaptsoglou

  6. Department of Physics, University of York, York, UK

    Demie Kepaptsoglou

  7. CNR-NANO, Modena, Italy

    Vincenzo Grillo

  8. Center for Nanoscale Materials, Argonne National Laboratory, Lemont, IL, USA

    Eric Schwenker & Maria K. Y. Chan

  9. Materials Science and Engineering Department, Northwestern University, Evanston, IL, USA

    Eric Schwenker

  10. Monash Centre for Electron Microscopy and Department of Materials Science and Engineering, Monash University, Clayton, Victoria, Australia

    Joanne Etheridge

  11. Department of Chemistry, Brandeis University, Waltham, MA, USA

    Xiang Li & Grace G. D. Han

  12. Computational Sciences and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Maxim Ziatdinov

  13. Institute of Engineering Innovation, School of Engineering, The University of Tokyo, Tokyo, Japan

    Naoya Shibata

  14. Department of Materials Science and Engineering, University of Tennessee, Knoxville, TN, USA

    Stephen J. Pennycook

  15. School of Physical Sciences and CAS Key Laboratory of Vacuum Sciences, University of Chinese Academy of Sciences, Beijing, China

    Stephen J. Pennycook

Authors
  1. Sergei V. Kalinin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Colin Ophus
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul M. Voyles
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rolf Erni
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Demie Kepaptsoglou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Vincenzo Grillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Andrew R. Lupini
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Mark P. Oxley
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Eric Schwenker
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Maria K. Y. Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Joanne Etheridge
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Xiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Grace G. D. Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Maxim Ziatdinov
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Naoya Shibata
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Stephen J. Pennycook
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Introduction (S.J.P. and M.P.O.); Experimentation (J.E., C.O., A.R.L., D.K., R.E. and V.G.); Results (P.M.V., X.L. and G.G.D.H.); Applications (N.S., E.S. and M.K.Y.C.); Reproducibility and data deposition (C.O. and M.Z.); Limitations and optimizations (C.O. and S.V.K.); Outlook (C.O., M.Z. and S.V.K.); Overview of the Primer (S.V.K.).

Corresponding author

Correspondence to Sergei V. Kalinin.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Methods Primers thanks Jamie Warner and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

abTEM: https://doi.org/10.12688/openreseurope.13015.1

AtomAI: https://github.com/pycroscopy/atomai

ESPRIT from Bruker: https://www.bruker.com/en/products-and-solutions/elemental-analyzers/eds-wds-ebsd-SEM-Micro-XRF/software-esprit-family.html

Gatan Microscopy Suite: https://www.gatan.com/products/tem-analysis/gatan-microscopy-suite-software

GitHub: https://github.com/

HyperSpy: https://doi.org/10.5281/zenodo.592838

ImageJ: https://imagej.nih.gov/ij/

LabView: https://www.ni.com/en-us/shop/labview.html

MATLAB: https://www.mathworks.com/products/matlab.html

Octave: https://www.gnu.org/software/octave/index

PyTEMLib: https://github.com/pycroscopy/pyTEMlib

STEMtools toolkit: https://github.com/pycroscopy/stemtool

Velox from Thermo Fisher Scientific: https://www.thermofisher.com/uk/en/home/electron-microscopy/products/software-em-3d-vis/velox-software.html

Supplementary information

Glossary

Aberration

An imperfection in the electron optics of a microscope.

Coherent imaging

Measurements where the local contrast is dominated by the phase alignment of the electron wavefronts: constructive interference (in phase) leads to higher signals and destructive inference (out of phase) leads to lower signals.

Incoherent imaging

When the coherence length of the electron waves is smaller than the resolution element of the measurement, the total signal is given incoherently by the sum of individual electron wavefunction intensities, and the relative phase of these wavefronts does not affect the measured intensity.

Contrast

The spatial variation of intensity.

Z-contrast imaging

A scanning transmission electron microscopy-high-angle annular dark-field imaging method, where the image contrast scales roughly monotonically with the atomic number Z of the atom(s) being imaged, approximately as Z1.7.

Ptychography

A method of generating images from many coherent diffraction patterns formed at different probe positions in the STEM. It is also widely used in X-ray scattering experiments.

Tilt series tomography

By tilting the specimen and recording projected images at different angles, computer algorithms can be used to reconstruct the 3D sample structure.

Transmission modes

Imaging modes in electron microscopy where the electron beam passes through the specimen.

Differential phase contrast

A method that measures the change in the convergent beam diffraction pattern as a function of probe position using either a segmented or a pixelated detector. These changes can be related to the local change in the sample’s potential and corresponding fields.

Phonons

A quantized collective vibration of atoms in a crystalline sample, which can be excited by the electron beam and characterized by scanning transmission electron microscopy-electron energy loss spectroscopy or diffraction measurements.

Plasmons

A quantized collective oscillation of electrons relative to the fixed ions in a sample, which can be excited by the electron beam and characterized by scanning transmission electron microscopy-electron energy loss spectroscopy.

Core-loss edges

Excitation of inner-shell electrons (ionization) by the electron beam, where the energy loss can be probed by scanning transmission electron microscopy-electron energy loss spectroscopy for features referred to as ‘edges’.

Dynamical scattering

A term commonly used in electron microscopy to describe the multiple scattering of the incident electron probe as it propagates through the specimen.

Electron optical elements

Electromagnetic lenses used to focus or otherwise shape the electron beam.

Electron holography

A technique for viewing the phase of the exit surface wavefunction using the interference of a scattered and unscattered electron beam.

Azimuthal phase gradient

(APG). A wavefunction where the phase is linearly proportional to the angle in polar coordinates, and the total phase shift is an integer multiple of 2π for each revolution (see orbital angular momentum).

Orbital angular momentum

(OAM). Orbital angular momenta are quanta given by the number of multiples of 2π in the phase of an electron beam, per angular revolution in polar coordinates (see azimuthal phase gradient).

Electron energy loss near edge structure

The intensity variation of the electron energy loss spectroscopy signal as a function of energy loss near the onset of the core-loss signal.

L23 ratio

The ratio of the L3 to L2 peaks formed by the transition of the 2p3/2 and 2p1/2 electrons to empty states.

Dark count rate

(D). This is the mean value of a scanning transmission electron microscopy image acquired with the beam blanked preferably near the gun by, for example, closing the gun vacuum valve.

Gain

(G). Adjustment to ensure that the measured signal covers the optimal range of the amplifier.

Faraday cup

A conductive cup that can capture charged free particles, with which the electron beam current can be estimated by integrating the recorded signal.

Residual

The difference between the fitted image and the experimental image after atom location.

Latent variables

A variable that is not directly observable, often obtained using variational auto encoders.

Latent spaces

A vector space spanned by the latent variables.

Evidence lower boundary

The lower bound of the probability of observing a particular result for a given model.

Penrose structures

Local structural units that, when displaced and rotated, can fully tile space, but do not have periodic translational symmetry. Such atomic structures can be found in quasicrystals.

Electron beam irradiation

This occurs when an electron beam induces changes in a specimen due to energy transfer, often called beam damage.

Dwell time

The time period of the data collection in each pixel.

Inferential biases

The assumptions and constraints implemented in the structure of the network, loss function or training set that impose specific limitations on the outputs.

Exploration

Uncertainty minimization.

Exploitation

Balancing exploration and pursuing target functionalities.

Out-of-distribution data

When observational conditions change between experiments, precluding a direct comparison of data between experiments.

Distribution shift

In machine learning, this shift occurs when training and test sets do not come from the same distribution.

Knock-on damage thresholds

The energy of the incident electron required to remove an atom from the crystal lattice.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kalinin, S.V., Ophus, C., Voyles, P.M. et al. Machine learning in scanning transmission electron microscopy. Nat Rev Methods Primers 2, 11 (2022). https://doi.org/10.1038/s43586-022-00095-w

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s43586-022-00095-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing