Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

AtomAI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy

Nature Machine Intelligence volume 4, pages 1101–1112 (2022)Cite this article

Subjects

A preprint version of the article is available at arXiv.

Abstract

Over the past several decades, electron and scanning probe microscopes have become critical components of condensed matter physics, materials science and chemistry research. At the same time, the infrastructure for establishing a connection between microscopy observations and materials behaviour over a broader parameter space is lacking. Here we introduce AtomAI, an open-source software package bridging instrument-specific Python libraries, deep learning and simulation tools into a single ecosystem. AtomAI allows direct applications of deep neural networks for atomic and mesoscopic image segmentation converting image and spectroscopy data into class-based local descriptors for downstream tasks such as statistical and graph analysis. For atomically resolved imaging data, the output is types and positions of atomic species, with an option for subsequent refinement. AtomAI further allows the implementation of a broad range of image and spectrum analysis functions, including invariant variational autoencoders for disentangling structural factors of variation and im2spec type of encoder–decoder models for mapping structure–property relationships. Finally, our framework allows seamless connection to the first principles modelling with a Python interface on the inferred atomic positions.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: AtomAI is a flexible and user-friendly package for deep-learning-based image analytics.
Fig. 2: Schematic illustration of a typical AtomAI workflow.
Fig. 3: AtomAI’s Segmentor.
Fig. 4: AtomAI’s ImSpec and VAE models.
Fig. 5
Fig. 6: AtomAI’s additional utilities.

Similar content being viewed by others

Data availability

All data pertinent to this manuscript can be accessed online at https://github.com/pycroscopy/atomai or https://doi.org/10.5281/zenodo.6406276 (ref. 135).

Code availability

The source code and example notebooks are available https://github.com/pycroscopy/atomai or at https://doi.org/10.5281/zenodo.6406276 (ref. 135).

References

  1. Pennycook, S. J. Seeing the atoms more clearly: STEM imaging from the Crewe era to today. Ultramicroscopy 123, 28–37 (2012).

    Article  Google Scholar 

  2. Pennycook, S. J. & Nellist, P. D. Scanning Transmission Electron Microscopy: Imaging and Analysis (Springer, 2011).

  3. Dellby, N., Krivanek, O. L., Nellist, P. D., Batson, P. E. & Lupini, A. R. Progress in aberration-corrected scanning transmission electron microscopy. J. Electron Microsc. 50, 177–185 (2001).

    Google Scholar 

  4. Huang, P. Y. et al. Imaging atomic rearrangements in two-dimensional silica glass: watching silica’s dance. Science 342, 224–227 (2013).

    Article  Google Scholar 

  5. Ophus, C. Four-dimensional scanning transmission electron microscopy (4D-STEM): from scanning nanodiffraction to ptychography and beyond. Microsc. Microanal. 25, 563–582 (2019).

    Article  Google Scholar 

  6. Ardenne, M. V. & Elektronen-Rastermikroskop, Das Praktische Ausführung. Z. Tech. Phys. 19, 407–416 (1938).

    Google Scholar 

  7. Ardenne, M. V. & Elektronen-Rastermikroskop, Das Theoretische Grundlagen. Zeit. Physik 109, 553–572 (1938).

    Article  Google Scholar 

  8. Knoll, M. & Ruska, E. Das Elektronenmikroskop. Zeit. Physik 78, 318–339 (1932).

    Article  Google Scholar 

  9. Ruska, E. The development of the electron-microscope and of electron-microscopy. Rev. Mod. Phys. 59, 627–638 (1987).

    Article  Google Scholar 

  10. Binnig, G., Rohrer, H., Gerber, C. & Weibel, E. 7×7 reconstruction on Si(111) resolved in real space. Phys. Rev. Lett. 50, 120–123 (1983).

    Article  Google Scholar 

  11. Binnig, G., Quate, C. F. & Gerber, C. Atomic force microscope. Phys. Rev. Lett. 56, 930–933 (1986).

    Article  Google Scholar 

  12. Garcia, R. & Perez, R. Dynamic atomic force microscopy methods. Surf. Sci. Rep. 47, 197–301 (2002).

    Article  Google Scholar 

  13. Gruverman, A., Auciello, O., Ramesh, R. & Tokumoto, H. Scanning force microscopy of domain structure in ferroelectric thin films: imaging and control. Nanotechnology 8, A38–A43 (1997).

    Article  Google Scholar 

  14. Martin, Y. & Wickramasinghe, H. K. Magnetic imaging by force microscopy with 1000 Å resolution. Appl. Phys. Lett. 50, 1455–1457 (1987).

    Article  Google Scholar 

  15. Vatel, O. & Tanimoto, M. Kelvin probe force microscopy for potential distribution measurement of semiconductor-devices. J. Appl. Phys. 77, 2358–2362 (1995).

    Article  Google Scholar 

  16. Nonnenmacher, M., Oboyle, M. P. & Wickramasinghe, H. K. Kelvin probe force microscopy. Appl. Phys. Lett. 58, 2921–2923 (1991).

    Article  Google Scholar 

  17. Liu, X. et al. Visualizing broken symmetry and topological defects in a quantum Hall ferromagnet. Science 375, 321–326 (2022).

    Article  Google Scholar 

  18. Rajeswari, J. et al. Filming the formation and fluctuation of skyrmion domains by cryo-Lorentz transmission electron microscopy. Proc. Natl Acad. Sci. USA 112, 14212–14217 (2015).

    Article  Google Scholar 

  19. Kohno, Y., Seki, T., Findlay, S. D., Ikuhara, Y. & Shibata, N. Real-space visualization of intrinsic magnetic fields of an antiferromagnet. Nature 602, 234–239 (2022).

    Article  Google Scholar 

  20. Shibata, N. et al. Electric field imaging of single atoms. Nat. Commun. 8, 7 (2017).

    Article  Google Scholar 

  21. Roushan, P. et al. Topological surface states protected from backscattering by chiral spin texture. Nature 460, 1106–U64 (2009).

    Article  Google Scholar 

  22. Das, S. et al. Observation of room-temperature polar skyrmions. Nature 568, 368–372 (2019).

    Article  Google Scholar 

  23. Kourkoutis, L. F., Plitzko, J. M. & Baumeister, W. Electron microscopy of biological materials at the nanometer scale. Annu. Rev. Mater. Res. 42, 33–58 (2012).

    Article  Google Scholar 

  24. Dufrêne, Y. F. et al. Imaging modes of atomic force microscopy for application in molecular and cell biology. Nat. Nanotechnol. 12, 295–307 (2017).

    Article  Google Scholar 

  25. MacLaren, I., Macgregor, T. A., Allen, C. S. & Kirkland, A. I. Detectors—the ongoing revolution in scanning transmission electron microscopy and why this important to material characterization. APL Mater. 8, 110901 (2020).

    Article  Google Scholar 

  26. Taheri, M. L. et al. Current status and future directions for in situ transmission electron microscopy. Ultramicroscopy 170, 86–95 (2016).

    Article  Google Scholar 

  27. Krivanek, O. L.; Dellby, N.; Spence, A. J.; Camps, R. A.; Brown, L. M. in Electron Microscopy and Analysis 1997 (ed. Rodenburg, J. M.) 35–40 (CRC Press, 1997).

  28. Batson, P. E., Dellby, N. & Krivanek, O. L. Sub-angstrom resolution using aberration corrected electron optics. Nature 418, 617–620 (2002).

    Article  Google Scholar 

  29. Varela, M. et al. Spectroscopic imaging of single atoms within a bulk solid. Phys. Rev. Lett. 92, 095502 (2004).

    Article  Google Scholar 

  30. Zhou, W. et al. Intrinsic structural defects in monolayer molybdenum disulfide. Nano Lett. 13, 2615–2622 (2013).

    Article  Google Scholar 

  31. Browning, N. D. et al. The atomic origins of reduced critical currents at 001 tilt grain boundaries in YBa2Cu3O7-delta thin films. Physica C 294, 183–193 (1998).

    Article  Google Scholar 

  32. Guo, X. & Waser, R. Electrical properties of the grain boundaries of oxygen ion conductors: acceptor-doped zirconia and ceria. Prog. Mater Sci. 51, 151–210 (2006).

    Article  Google Scholar 

  33. McMorran, B. J. et al. Electron vortex beams with high quanta of orbital angular momentum. Science 331, 192–195 (2011).

    Article  Google Scholar 

  34. Grillo, V. et al. Generation of nondiffracting electron bessel beams. Phys. Rev. X 4, 011013 (2014).

  35. Rusz, J. & Bhowmick, S. Boundaries for efficient use of electron vortex beams to measure magnetic properties. Phys. Rev. Lett. 111, 105504 (2013).

  36. Idrobo, J. C. et al. Temperature measurement by a nanoscale electron probe using energy gain and loss spectroscopy. Phys. Rev. Lett. 120, 095901 (2018).

  37. Cho, S. H. et al. Spectrally tunable infrared plasmonic F,Sn:In2O3 nanocrystal cubes. J. Chem. Phys. 152, 014709 (2020).

  38. Senga, R. et al. Position and momentum mapping of vibrations in graphene nanostructures. Nature 573, 247–250 (2019).

    Article  Google Scholar 

  39. Egerton, R. F. Limits to the spatial, energy and momentum resolution of electron energy-loss spectroscopy. Ultramicroscopy 107, 575–586 (2007).

    Article  Google Scholar 

  40. Jiang, Y. et al. Electron ptychography of 2D materials to deep sub-angstrom resolution. Nature 559, 343–349 (2018).

    Article  Google Scholar 

  41. Yang, Y. S. et al. Deciphering chemical order/disorder and material properties at the single-atom level. Nature 542, 75–79 (2017).

    Article  Google Scholar 

  42. Jia, C. L. et al. Unit-cell scale mapping of ferroelectricity and tetragonality in epitaxial ultrathin ferroelectric films. Nat. Mater. 6, 64–69 (2007).

    Article  Google Scholar 

  43. Jia, C. L. et al. Atomic-scale study of electric dipoles near charged and uncharged domain walls in ferroelectric films. Nat. Mater. 7, 57–61 (2008).

    Article  Google Scholar 

  44. Chisholm, M. F., Luo, W. D., Oxley, M. P., Pantelides, S. T. & Lee, H. N. Atomic-scale compensation phenomena at polar interfaces. Phys. Rev. Lett. 105, 197602 (2010).

  45. MacLaren, I. & Ramasse, Q. M. Aberration-corrected scanning transmission electron microscopy for atomic-resolution studies of functional oxides. Int. Mater. Rev. 59, 115–131 (2014).

    Article  Google Scholar 

  46. Jia, C. L. et al. Oxygen octahedron reconstruction in the SrTiO3/LaAlO3 heterointerfaces investigated using aberration-corrected ultrahigh-resolution transmission electron microscopy. Phys. Rev. B. 29, 081405 (2009).

  47. Borisevich, A. et al. Mapping octahedral tilts and polarization across a domain wall in BiFeO3 from Z-contrast scanning transmission electron microscopy image atomic column shape analysis. ACS Nano 4, 6071–6079 (2010).

    Article  Google Scholar 

  48. Kim, Y. M. et al. Interplay of octahedral tilts and polar order in BiFeO3 films. Adv. Mater. 25, 2497–2504 (2013).

    Article  Google Scholar 

  49. He, Q. et al. Towards 3D mapping of BO6 octahedron rotations at perovskite heterointerfaces, unit cell by unit cell. ACS Nano 9, 8412–8419 (2015).

    Article  Google Scholar 

  50. Nord, M. et al. Three-dimensional subnanoscale imaging of unit cell doubling due to octahedral tilting and cation modulation in strained perovskite thin films. Phys. Rev. Mater. 3, 063605 (2019).

    Article  Google Scholar 

  51. Kim, Y. M. et al. Direct observation of ferroelectric field effect and vacancy-controlled screening at the BiFeO3/LaxSr1-xMnO3 interface. Nat. Mater. 13, 1019–1025 (2014).

    Article  Google Scholar 

  52. Borisevich, A. Y. et al. Interface dipole between two metallic oxides caused by localized oxygen vacancies. Phys. Rev. B. 86, 140102 (2012).

  53. Yankovich, A. B. et al. Picometre-precision analysis of scanning transmission electron microscopy images of platinum nanocatalysts. Nat. Commun. 5, 4155 (2014).

  54. Vlcek, L. et al. Learning from imperfections: predicting structure and thermodynamics from atomic imaging of fluctuations. ACS Nano 13, 718–727 (2019).

    Article  Google Scholar 

  55. Vlcek, L., Maksov, A., Pan, M. H., Vasudevan, R. K. & Kahnin, S. V. Knowledge extraction from atomically resolved images. Acs Nano 11, 10313–10320 (2017).

    Article  Google Scholar 

  56. Lin, W. Z. et al. Local crystallography analysis for atomically resolved scanning tunneling microscopy images. Nanotechnology. 24, 415707 (2013).

  57. Gai, Z. et al. Chemically induced Jahn–Teller ordering on manganite surfaces. Nat. Commun. 5, 4528 (2014).

  58. Ovchinnikov, O. S., Jesse, S., Bintacchit, P., Trolier-McKinstry, S. & Kalinin, S. V. Disorder identification in hysteresis data: recognition analysis of the random-bond-random-field Ising model. Phys. Rev. Lett. 103, 157203 (2009).

  59. Kumar, A. et al. Spatially resolved mapping of disorder type and distribution in random systems using artificial neural network recognition. Phys. Rev. B 84, 024203 (2011).

    Article  Google Scholar 

  60. Nikiforov, M. P. et al. Functional recognition imaging using artificial neural networks: applications to rapid cellular identification via broadband electromechanical response. Nanotechnology 20, 405708 (2009).

    Article  Google Scholar 

  61. Wahl, P., Singh, U. R., Tsurkan, V. & Loidl, A. Nanoscale electronic inhomogeneity in FeSe0.4 Te0.6 revealed through unsupervised machine learning. Phys. Rev. B 101, 115112 (2020).

  62. Rashidi, M. & Wolkow, R. A. Autonomous scanning probe microscopy in situ tip conditioning through machine learning. ACS Nano 12, 5185–5189 (2018).

    Article  Google Scholar 

  63. Croshaw, J., Dienel, T., Huff, T. & Wolkow, R. Atomic defect classification of the H–Si (100) surface through multi-mode scanning probe microscopy. Beilstein J. Nanotechnol. 11, 1346–1360 (2020).

    Article  Google Scholar 

  64. Jesse, S. & Kalinin, S. V. Principal component and spatial correlation analysis of spectroscopic-imaging data in scanning probe microscopy. Nanotechnology 20, 085714 (2009).

    Article  Google Scholar 

  65. Bosman, M., Watanabe, M., Alexander, D. T. L. & Keast, V. J. Mapping chemical and bonding information using multivariate analysis of electron energy-loss spectrum images. Ultramicroscopy 106, 1024–1032 (2006).

    Article  Google Scholar 

  66. Allen, L. J. et al. in Electron Microscopy and Multiscale Modeling, Proceedings (eds. Avilov, A. S., Dudarev, S. L. & Marks, L. D.) 32–46 (Am. Inst. Phys., 2008).

  67. Kannan, R. et al. Deep data analysis via physically constrained linear unmixing: universal framework, domain examples, and a community-wide platform. Adv. Struct. Chem. Imag. 4, 6 (2018).

  68. Vasudevan, R. K., Ziatdinov, M., Jesse, S. & Kalinin, S. V. Phases and interfaces from real space atomically resolved data: physics-based deep data image analysis. Nano Lett. 16, 5574–5581 (2016).

    Article  Google Scholar 

  69. Paszke, A. et al. Pytorch: an imperative style, high-performance deep learning library. In eds. Wallach, H. et al. Advances in Neural Information Processing Systems 8026–8037 (NeurIPS, 2019).

  70. Buitinck, L. et al. API design for machine learning software: experiences from the scikit-learn project. Preprint at arXiv https://doi.org/10.48550/arXiv.1309.0238 (2013).

  71. Ziatdinov, M., Nelson, C., Vasudevan, R. K., Chen, D. Y. & Kalinin, S. V. Building ferroelectric from the bottom up: the machine learning analysis of the atomic-scale ferroelectric distortions. Appl. Phys. Lett. 115, 05902 (2019).

  72. Kalinin, S. V., Dyck, O., Jesse, S. & Ziatdinov, M. Exploring order parameters and dynamic processes in disordered systems via variational autoencoders. Sci. Adv. 7, eabd5084 (2021).

    Article  Google Scholar 

  73. Ziatdinov, M., Jesse, S., Sumpter, B. G., Kalinin, S. V. & Dyck, O. Tracking atomic structure evolution during directed electron beam induced Si-atom motion in graphene via deep machine learning. Nanotechnology 32, 035703 (2020).

    Google Scholar 

  74. Ghosh, A., Sumpter, B. G., Dyck, O., Kalinin, S. V. & Ziatdinov, M. Ensemble learning and iterative training (ELIT) machine learning: applications towards uncertainty quantification and automated experiment in atom-resolved microscopy. Npj Comput. Mater. 7, 1-8 (2021).

  75. Ziatdinov, M., Fuchs, U., Owen, J. H., Randall, J. N. & Kalinin, S. V. Robust multi-scale multi-feature deep learning for atomic and defect identification in scanning tunneling microscopy on H-Si (100) 2×1 surface. Preprint at arXiv https://doi.org/10.48550/arXiv.2002.04716 (2020).

  76. Kalinin, S. V., Steffes, J. J., Huey, B. D. & Ziatdinov, M. Disentangling ferroelectric domain wall geometries and pathways in dynamic piezoresponse force microscopy via unsupervised machine learning. Nanotechnology 33, 055707 (2021).

  77. Ignatans, R. et al. Latent mechanisms of polarization switching from in situ electron microscopy observations. Adv. Funct. Mater. 32, 2100271 https://doi.org/10.1002/adfm.202100271 (2022).

  78. Kalinin, S. V. et al. Lab on a beam—big data and artificial intelligence in scanning transmission electron microscopy. MRS Bull. 44, 565–575 (2019).

    Article  Google Scholar 

  79. de la Peña, F. et al. Microanalysis, electron microscopy (big and small) data analysis with the open source software package HyperSpy. Microsc. Microanal. 23, 214–215 (2017).

    Article  Google Scholar 

  80. Somnath, S., Smith, C. R., Laanait, N., Vasudevan, R. K. & Jesse, S. USID and pycroscopy–open source frameworks for storing and analyzing imaging and spectroscopy data. Microsc. Microanal. 25, 220–221 (2019).

    Article  Google Scholar 

  81. Mukherjee, D. STEMTool. GitHub Repository https://github.com/pycroscopy/stemtool (2020).

  82. Duscher, G. pyTEMlib. GitHub Repository https://github.com/pycroscopy/pyTEMlib (2020).

  83. Johnstone, D. pyxem. GitHub Repository (2022). https://pyxem.github.io/pyxem/pyxem

  84. Clausen, A. et al. LiberTEM: software platform for scalable multidimensional data processing in transmission electron microscopy. J. Open Source Softw. 5, 2006 (2020).

    Article  Google Scholar 

  85. Savitzky, B. H. et al. py4DSTEM: open source software for 4D-STEM data analysis. Microsc. Microanal. 25, 124–125 (2019).

    Article  Google Scholar 

  86. Allen, L. J., D׳Alfonso, A. J. & Findlay, S. D. Modelling the inelastic scattering of fast electrons. Ultramicroscopy 151, 11–22 (2015).

    Article  Google Scholar 

  87. Koch, C. T. Determination of Core Structure Periodicity and Point Defect Density along Dislocations. PhD thesis (Arizona State University, 2002).

  88. Madsen, J. & Susi, T. abTEM: ab initio transmission electron microscopy image simulation. Microsc. Microanal. 26, 448–450 (2020).

    Article  Google Scholar 

  89. Lobato, I. & Van Dyck, D. MULTEM: a new multislice program to perform accurate and fast electron diffraction and imaging simulations using graphics processing units with CUDA. Ultramicroscopy 156, 9–17 (2015).

    Article  Google Scholar 

  90. Oelerich, J. O. et al. STEMsalabim: a high-performance computing cluster friendly code for scanning transmission electron microscopy image simulations of thin specimens. Ultramicroscopy 177, 91–96 (2017).

    Article  Google Scholar 

  91. Pryor, A., Ophus, C. & Miao, J. A streaming multi-GPU implementation of image simulation algorithms for scanning transmission electron microscopy. Adv. Struct. Chem. Imag. 3, 15 (2017).

    Article  Google Scholar 

  92. Barthel, J. Dr. Probe: a software for high-resolution STEM image simulation. Ultramicroscopy 193, 1–11 (2018).

    Article  Google Scholar 

  93. Garcia, A. et al. Siesta: recent developments and applications. J. Chem. Phys. 152, 204108 (2020).

  94. al, P. G. E. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. J. Phys. Condens. Matter 21, 395502 (2009).

    Article  Google Scholar 

  95. al, P. G. E. Advanced capabilities for materials modelling with Quantum ESPRESSO. J. Phys. Condens. Matter 29, 465901 (2017).

    Article  Google Scholar 

  96. al, P. G. E. Quantum ESPRESSO toward the exascale. J. Chem. Phys. 152, 154105 (2020).

    Article  Google Scholar 

  97. al, T. D. K. E. CP2K: an electronic structure and molecular dynamics software package—Quickstep: efficient and accurate electronic structure calculations. J. Chem. Phys. 152, 194103 (2020).

    Article  Google Scholar 

  98. Permann, C. J. et al. MOOSE: enabling massively parallel multiphysics simulation. SoftwareX 11, 100430 (2020).

  99. Guido, D. The Finite Element Method for Three-Dimensional Thermomechanical Applications (John Wiley & Sons, 2004).

  100. Thompson, A.P. et al. LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Comput. Phys. Commun. 271, 108171 (2022). https://lammps.sandia.gov/

  101. Soler, J. M. et al. The SIESTA method for ab initio order-N materials simulation. J. Phys. Condens. Matter 14, 2745 (2002).

    Article  Google Scholar 

  102. K. C. Koppenhoefer et al. WARP3D: Dynamic Nonlinear Analysis of Solids Using a Preconditioned Conjugate Gradient Software Architecture (Univ. Illinois at Urbana Champaign, 1994).

  103. Malinen, M. & Råback, P. Elmer finite element solver for multiphysics and multiscale problems. Appl. Mater. Sci. 19, 101–113 (2013).

    Google Scholar 

  104. Kloss, C., Goniva, C., Hager, A., Amberger, S. & Pirker, S. Models, algorithms and validation for opensource DEM and CFD-DEM. Prog. Comput. Fluid Dyn. 12, 140–152 (2012).

    Article  MathSciNet  Google Scholar 

  105. DeLano, W. L., DeLano & Warren, L. Pymol: an open-source molecular graphics tool. CCP4 Newslett. Protein Crystallogr. 40, 82–92 (2002).

    Google Scholar 

  106. Hirel, P. Atomsk: a tool for manipulating and converting atomic data files. Comput. Phys, Comm. 197, 212–219 (2015).

    Article  Google Scholar 

  107. Izumi, K. M. A. F. VESTA 3 for three-dimensional visualization of crystal, volumetric and morphology data. J. Appl. Crystallogr. 44, 1272–1276 (2011).

    Article  Google Scholar 

  108. Ahrens, J., Geveci, B. & Law, C. ParaView: An End-User Tool for Large Data Visualization, Visualization Handbook (Elsevier, 2015).

  109. Martínez, L., Andrade, R., Birgin, E. G. & Martínez, J. M. Packmol: a package for building initial configurations for molecular dynamics simulations. J. Comput. Chem. 30, 2157–2164 (2009).

    Article  Google Scholar 

  110. Hanwell, M. D. et al. Avogadro: an advanced semantic chemical editor, visualization, and analysis platform. J. Cheminformatics 4, 1–17 (2012).

    Article  Google Scholar 

  111. Stukowski, A. Visualization and analysis of atomistic simulation data with OVITO—the Open Visualization Tool. Model. Simul. Mater. Sci. Eng. 18, 015012 (2010).

    Article  Google Scholar 

  112. Utkarsh, A. The Paraview Guide: A Parallel Visulaization Application (Kitware, 2015).

  113. Humphrey, W., Dalke, A. & Schulten, K. VMD—Visual Molecular Dynamics. J. Molec. Graphics 14, 33–38 (1996).

    Article  Google Scholar 

  114. Jain, A. et al. Commentary: The Materials Project: a materials genome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).

  115. Hicks, D. et al. The AFLOW Library of Crystallographic Prototypes: part 2. Comp. Mat. Sci. 161, S1–S1011 (2019).

    Article  Google Scholar 

  116. Saal, J. E., Kirklin, S., Aykol, M., Meredig, B. & Wolverton, C. Materials design and discovery with high-throughput density functional theory: The Open Quantum Materials Database (OQMD). JOM 65, 1501–1509 (2013).

    Article  Google Scholar 

  117. Mehl, M. J. et al. The AFLOW Library of Crystallographic Prototypes: part 1. Comp. Mat. Sci. 136, S1–S828 (2017).

    Article  Google Scholar 

  118. Kirklin, S. et al. The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies. NPJ Comput. Mater. 1, 1-15 (2015).

  119. Scheffler, C. D. A. M. The NOMAD laboratory: from data sharing to artificial intelligence. J. Phys. Mater. 2, 036001 (2019).

    Article  Google Scholar 

  120. Ronneberger, O.; Fischer, P.; Brox, T. in eds. Navab, N. et al. International Conference on Medical Image Computing and Computer-Assisted Intervention 234–241 (Springer, 2015).

  121. Ziatdinov, M., Dyck, O., Jesse, S. & Kalinin, S. V. Atomic mechanisms for the Si atom dynamics in graphene: chemical transformations at the edge and in the bulk. Adv. Funct. Mater. 29, 1904480 (2019).

    Article  Google Scholar 

  122. Farley, S. et al. Improving the segmentation of scanning probe microscope images using convolutional neural networks. Mach. Learn. Sci. Technol. 2, 015015 (2020).

    Article  Google Scholar 

  123. Xie, S. & Tu, Z. Holistically-nested edge detection. In ed. O'Conner, L. Proc. IEEE International Conference on Computer Vision 1395–1403 (IEEE, 2015).

  124. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In ed. O'Conner, L. Proc. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

  125. Badrinarayanan, V., Kendall, A. & Cipolla, R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. 39, 2481–2495 (2017).

    Article  Google Scholar 

  126. Kalinin, S. V., Kelley, K., Vasudevan, R. K. & Ziatdinov, M. Toward decoding the relationship between domain structure and functionality in ferroelectrics via hidden latent variables. ACS Appl. Mater. Interfaces 13, 1693–1703 (2021).

    Article  Google Scholar 

  127. Roccapriore, K. M., Ziatdinov, M., Cho, S. H., Hachtel, J. A. & Kalinin, S. V. Predictability of localized plasmonic responses in nanoparticle assemblies. Small, 17, 2100181 (2021).

  128. Gordon, O. M., Hodgkinson, J. E. A., Farley, S. M., Hunsicker, E. L. & Moriarty, P. J. Automated searching and identification of self-organized nanostructures. Nano Lett. 20, 7688–7693 (2020).

    Article  Google Scholar 

  129. Wilson, A. G. & Izmailov, P. Bayesian deep learning and a probabilistic perspective of generalization. In eds. Larochelle, H. et al. Proc. 34th International Conference on Neural Information Processing Systems, 4697-4708, (Curran Assoc. Inc., 2020).

  130. Kingma, D. P. & Welling, M. An introduction to variational autoencoders. Found. Trends Mach. Learn. 12, 307–392 (2019).

    Article  MATH  Google Scholar 

  131. Lakshminarayanan, B.; Pritzel, A.; Blundell, C., Simple and scalable predictive uncertainty estimation using deep ensembles. In eds. von Luxburg, U. et al. Proc. 31st International Conference on Neural Information Processing Systems 6405–6416 (Curran Assoc. Inc., 2017).

  132. Gordon, O. et al. Scanning tunneling state recognition with multi-class neural network ensembles. Rev. Sci. Instrum. 90, 103704 (2019).

    Article  Google Scholar 

  133. Ziatdinov, M. et al. Predictability as a probe of manifest and latent physics: the case of atomic scale structural, chemical, and polarization behaviors in multiferroic Sm-doped BiFeO3. Appl. Phys. Rev. 8, 011403 (2021).

    Article  Google Scholar 

  134. Vasudevan, R. K. et al. Autonomous experiments in scanning probe microscopy and spectroscopy: choosing where to explore polarization dynamics in ferroelectrics. ACS Nano. 15, 11253-11262 https://doi.org/10.1021/acsnano.0c10239 (2021).

  135. Ziatdinov, M. pycroscopy/atomai: v0.7.4. Zenodo https://doi.org/10.5281/zenodo.6406276 (2022).

  136. Somnath, S., Duscher, G., Ziatdinov, M., Vasudevan, R. & Valleti, M. SIDpy. GitHub Repository https://doi.org/10.5281/zenodo.4679761 (2021).

  137. Ziatdinov, M. et al. Quantifying the dynamics of protein self-organization using deep learning analysis of atomic force microscopy data. Nano Lett. 21, 158–165 (2021).

    Article  Google Scholar 

  138. Kalinin, S. V. et al. Disentangling rotational dynamics and ordering transitions in a system of self-organizing protein nanorods via rotationally invariant latent representations. ACS Nano 15, 6471–6480 (2021).

    Article  Google Scholar 

  139. Ziatdinov, M. et al. Building and exploring libraries of atomic defects in graphene: scanning transmission electron and scanning tunneling microscopy study. Sci. Adv. 5, eaaw8989 (2019).

    Article  Google Scholar 

  140. Ghosh, A., Ziatdinov, M., Dyck, O., Sumpter, B. G. & Kalinin, S. V. Bridging microscopy with molecular dynamics and quantum simulations: an atomAI based pipeline. NPJ Comput. Mater. 8, 1-11 (2022).

Download references

Acknowledgements

This effort was performed and partially supported (M.Z.) at the Oak Ridge National Laboratory’s Center for Nanophase Materials Sciences (CNMS), a US Department of Energy, Office of Science User Facility, and by US Department of Energy, Office of Science, Office of Basic Energy Sciences Data, Artificial Intelligence and Machine Learning at DOE Scientific User Facilities programme under the Digital Twin Project (award number 34532) (A.G.) and MLExchange Project (award number 107514) (C.Y.W. and S.V.K.) The authors gratefully acknowledge multiple discussions with M. Chisholm, A. Lupini, M. Oxley, K. Roccapriore, J. Hachtel, O. Dyck and multiple other colleagues at Oak Ridge National Laboratory whose advice and beta testing have been instrumental throughout the development of AtomAI from 2019 to 2021 and its predecessor AICrystallographer in 2016–2019. The authors also express their deep gratitude to C. Ophus (LBNL), S. Spurgeon (PNNL), F. de la Peña (University of Lille), D. Weber (Juelich) and I. Maclaren (Glasgow University) for critical reading of the manuscript, suggesting several key references and suggesting improvement of key figures.

Author information

Authors and Affiliations

  1. Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Maxim Ziatdinov & Chun Yin (Tommy) Wong

  2. Computational Sciences and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Maxim Ziatdinov & Ayana Ghosh

  3. Bredesen Center for Interdisciplinary Research and Graduate Education, University of Tennessee, Knoxville, TN, USA

    Chun Yin (Tommy) Wong

  4. Department of Materials Science and Engineering, University of Knoxville, Knoxville, TN, USA

    Sergei V. Kalinin

Authors
  1. Maxim Ziatdinov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayana Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chun Yin (Tommy) Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sergei V. Kalinin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.Z. wrote the AtomAI package and led paper writing. A.G. wrote utility functions for connecting theory to AtomAI and contributed to manuscript writing. C.Y.W. tested the ensemble models and contributed to manuscript writing. S.V.K. performed extensive beta testing of AtomAI features, provided extensive suggestions and feedback, and contributed to manuscript writing.

Corresponding author

Correspondence to Maxim Ziatdinov.

Ethics declarations

Competing interests

The authors declare no competing financial or non-financial interests.

Peer review

Peer review information

Nature Machine Intelligence thanks the anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Supplementary information

Supplementary Information

Supplementary Figs. 1–4 and discussion (Supplementary Notes 1–6 and Supplementary Table 1).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ziatdinov, M., Ghosh, A., Wong, C.Y.(. et al. AtomAI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy. Nat Mach Intell 4, 1101–1112 (2022). https://doi.org/10.1038/s42256-022-00555-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00555-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing