Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

From atomically resolved imaging to generative and causal models

Nature Physics volume 18, pages 1152–1160 (2022)Cite this article

Subjects

Abstract

The development of high-resolution imaging methods such as electron and scanning probe microscopy and atomic probe tomography have provided a wealth of information on the atomic structure and functionalities of solids. The availability of this data in turn necessitates the development of approaches to derive quantitative physical information, akin to how the development of scattering techniques provided insights into the atomic-level structure of a vast array of materials in the twentieth century. Here we argue that the endeavour to understand local imaging data requires the adaptation of classical macroscopic definitions. For example, the notion of symmetry can be introduced locally only in a probabilistic sense that must balance our knowledge of the material’s physics and other experimental data points with the imaging data at hand. At the same time, this wealth of local data can enable fundamentally new approaches for the description of solids, based on the construction of statistical and physical models that can ‘generate’ the observed structures. Finally, we note that the availability of observational data opens pathways towards exploring causal mechanisms underpinning solid structure and functionality.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: General representations of solid systems at different length scales.
Fig. 2: Examples of discovering features from structural data.
Fig. 3: Variational autoencoders.
Fig. 4: Connections between observables and measurements.
Fig. 5: Causal and correlative ML.
Fig. 6: Schematic of the causal learning with a VAE.

Similar content being viewed by others

Data availability

The data pertinent to this article can be accessed via the corresponding references as cited in the relevant publications.

Code availability

The codes pertinent to this article can be accessed via the corresponding references as cited in the relevant publications.

References

  1. Kittel, C. Theory of antiferroelectric crystals. Phys. Rev. 82, 729–732 (1951).

    Article  ADS  MATH  Google Scholar 

  2. Tagantsev, A. K., Cross, L. E. & Fousek, J. Domains in Ferroic Crystals and Thin Films (Springer, 2010).

  3. Binder, K. & Young, A. P. Spin-glasses—experimental facts, theoretical concepts, and open questions. Rev. Mod. Phys. 58, 801–976 (1986).

    Article  ADS  Google Scholar 

  4. Binder, K. & Reger, J. D. Theory of orientational glasses models, concepts, simulations. Adv. Phys. 41, 547–627 (1992).

    Article  ADS  Google Scholar 

  5. Herbrych, J. et al. Block-spiral magnetism: an exotic type of frustrated order. Proc. Natl Acad. Sci. USA 117, 16226–16233 (2020).

    Article  ADS  Google Scholar 

  6. Heidrich-Meisner, F., Sergienko, I. A., Feiguin, A. E. & Dagotto, E. R. Universal emergence of the one-third plateau in the magnetization process of frustrated quantum spin chains. Phys. Rev. B 75, 064413 (2007).

  7. Zhang, H. et al. Anomalous magnetoresistance by breaking ice rule in Bi2Ir2O7/Dy2Ti2O7 heterostructure. Preprint at https://arxiv.org/abs/2011.09048 (2020).

  8. Lin, L.-F., Zhang, Y., Moreo, A., Dagotto, E. & Dong, S. Frustrated dipole order induces noncollinear proper ferrielectricity in two dimensions. Phys. Rev. Lett. 123, 067601 (2019).

  9. Mohanta, N., Christianson, A. D., Okamoto, S. & Dagotto, E. Signatures of a liquid-crystal transition in spin-wave excitations of skyrmions. Commun. Phys. 3, 229 (2020).

  10. Patel, N. D., Mukherjee, A., Kaushal, N., Moreo, A. & Dagotto, E. Non-Fermi liquid behavior and continuously tunable resistivity exponents in the Anderson–Hubbard model at finite temperature Phys. Rev. Lett. 119, 086601 (2017).

  11. Zhang, S.-S., Kaushal, N., Dagotto, E. & Batista, C. D. Spin–orbit interaction driven dimerization in one dimensional frustrated magnets. Phys. Rev. B 96, 214408 (2017).

  12. Vugmeister, B. E. & Rabitz, H. Coexistence of the critical slowing down and glassy freezing in relaxor ferroelectrics. Phys. Rev. B 61, 14448–14453 (2000).

    Article  ADS  Google Scholar 

  13. Grinberg, I., Suchomel, M. R., Davies, P. K. & Rappe, A. M. Predicting morphotropic phase boundary locations and transition temperatures in Pb- and Bi-based perovskite solid solutions from crystal chemical data and first-principles calculations. J. Appl. Phys. 98, 094111 (2005).

    Article  ADS  Google Scholar 

  14. Keen, D. A. & Goodwin, A. L. The crystallography of correlated disorder. Nature 521, 303–309 (2015).

    Article  ADS  Google Scholar 

  15. Cheetham, A. K., Bennett, T. D., Coudert, F. X. & Goodwin, A. L. Defects and disorder in metal organic frameworks. Dalton Trans. 45, 4113–4126 (2016).

    Article  Google Scholar 

  16. Overy, A. R. et al. Design of crystal-like aperiodic solids with selective disorder-phonon coupling. Nat. Commun. 7, 10445 (2016).

  17. Gerber, C. & Lang, H. P. How the doors to the nanoworld were opened. Nat. Nanotechnol. 1, 3–5 (2006).

    Article  ADS  Google Scholar 

  18. Binnig, G., Quate, C. F. & Gerber, C. Atomic force microscope. Phys. Rev. Lett. 56, 930–933 (1986).

    Article  ADS  Google Scholar 

  19. Binnig, G., Rohrer, H., Gerber, C. & Weibel, E. 7×7 reconstruction on Si(111) resolved in real space. Phys. Rev. Lett. 50, 120–123 (1983).

    Article  ADS  Google Scholar 

  20. Pennycook, S. J. & Nellist, P. D. Scanning Transmission Electron Microscopy: Imaging and Analysis (Springer, 2011).

  21. Krivanek, O. L. et al. Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy. Nature 464, 571–574 (2010).

    Article  ADS  Google Scholar 

  22. Jiang, Y. et al. Electron ptychography of 2D materials to deep sub-angstrom resolution. Nature 559, 343–34 (2018).

    Article  ADS  Google Scholar 

  23. Hytch, M. J. & Potez, L. Geometric phase analysis of high-resolution electron microscopy images of antiphase domains: example Cu3Au. Phil. Mag. A 76, 1119–1138 (1997).

    Article  ADS  Google Scholar 

  24. Hytch, M. J., Putaux, J. L. & Thibault, J. Stress and strain around grain-boundary dislocations measured by high-resolution electron microscopy. Phil. Mag. 86, 4641–4656 (2006).

    Article  ADS  Google Scholar 

  25. Vasudevan, R. K., Ziatdinov, M., Jesse, S. & Kalinin, S. V. Phases and Interfaces from real space atomically resolved data: physics-based deep data image analysis. Nano Lett. 16, 5574–5581 (2016).

    Article  ADS  Google Scholar 

  26. Vasudevan, R. K. et al. Big data in reciprocal space: sliding fast Fourier transforms for determining periodicity. Appl. Phys. Lett. 106, 091601 (2015).

  27. Rawat, W. & Wang, Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 29.9, 2352–2449 (2017).

    Article  MathSciNet  MATH  Google Scholar 

  28. Bates, R. H. T. & Mnyama, D. The status of practical Fourier phase retrieval. Adv. Electron. Electron Phys. 67, 1–64 (1986).

    Article  ADS  Google Scholar 

  29. Pennycook, S. J. Structure determination through Z-contrast microscopy. Adv. Imaging Electron Phys. 123, 173–206 (2002).

    Article  Google Scholar 

  30. Sayre, D. X-ray crystallography: the past and present of the phase problem. Struct. Chem. 13, 81–96 (2002).

    Article  Google Scholar 

  31. Patterson, A. L. A Fourier series method for the determination of the components of interatomic distances in crystals. Phys. Rev. 46, 372–376 (1934).

  32. Lin, W. Z. et al. Local crystallography analysis for atomically resolved scanning tunneling microscopy images. Nanotechnology 24, 415707 (2013).

    Article  Google Scholar 

  33. He, Q., Woo, J., Belianinov, A., Guliants, V. V. & Borisevich, A. Y. Better catalysts through microscopy: mesoscale M1/M2 intergrowth in molybdenum-vanadium based complex oxide catalysts for propane ammoxidation. ACS Nano 9, 3470–3478 (2015).

    Article  Google Scholar 

  34. Belianinov, A. et al. Identification of phases, symmetries and defects through local crystallography. Nat. Commun. 6, 7801 (2015).

  35. Goodwin, D. A. K. A. L. The crystallography of correlated disorder. Nature 521, 303–309 (2015).

    Article  ADS  Google Scholar 

  36. Kruschke, J. Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan 2nd edn (Academic Press, 2014).

  37. Nelson, C. T. et al. Exploring physics of ferroelectric domain walls via Bayesian analysis of atomically resolved STEM data. Nat. Commun. 11, 12 (2020).

    Article  Google Scholar 

  38. Thulasiraman, K. & Swamy, M. N. S. Graphs: Theory and Algorithms (Wiley, 2011).

  39. Louizos, C. et al. Causal effect inference with deep latent-variable models. In Proc. 31st International Conference on Neural Information Processing Systems 6449–6459 (Curran Associates, 2017).

  40. Khemakhem, I., Kingma, D., Monti, R. & Hyvarinen, A. Variational autoencoders and nonlinear ICA: a unifying framework. In International Conference on Artificial Intelligence and Statistics 2207–2217 (PMLR, 2020).

  41. Schölkopf, B. et al. Toward causal representation learning. Proc. IEEE 109, 612–634 (2021).

    Article  Google Scholar 

  42. Ziatdinov, M. et al. Building and exploring libraries of atomic defects in graphene: scanning transmission electron and scanning tunneling microscopy study. Sci. Adv. 5, eaaw898 (2019).

    Article  Google Scholar 

  43. Salakhutdinov, R. Learning deep generative models. Annu. Rev. Stat. Appl. 2, 361–385 (2015).

    Article  Google Scholar 

  44. Alqahtani, H., Kavakli-Thorne, M. & Kumar, G. Applications of generative adversarial networks (GANs): an updated review. Arch. Comput. Methods Eng. 28.2, 525–552 (2021).

    Article  MathSciNet  Google Scholar 

  45. Borisevich, A. Y. et al. Exploring mesoscopic physics of vacancy-ordered systems through atomic scale observations of topological defects. Phys. Rev. Lett. 109, 065702 (2012).

    Article  ADS  Google Scholar 

  46. Li, Q. et al. Quantification of flexoelectricity in PbTiO3/SrTiO3 superlattice polar vortices using machine learning and phase-field modeling. Nat. Commun. 8, 1468 (2017).

    Article  ADS  Google Scholar 

  47. Li, W., Bazant, M. Z. & Zhu, J. A physics-guided neural network framework for elastic plates: comparison of governing equations-based and energy-based approaches. Comput. Methods Appl. Mech. Eng. 383, 113933 (2021).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  48. Surya Effendy, J. S. A. M. Z. B. Analysis, design, and generalization of electrochemical impedance spectroscopy (EIS) inversion algorithms. J. Electrochem. Soc. 167, 106508 (2020).

    Article  Google Scholar 

  49. Zhao, Hongbo, B., R. D. & Bazant, MartinZ. Image inversion and uncertainty quantification for constitutive laws of pattern formation. J. Comput. Phys. 436, 110279 (2021).

    Article  MathSciNet  MATH  Google Scholar 

  50. Cottrill, AntonL. et al. Simultaneous inversion of optical and infra-red image data to determine thermo-mechanical properties of thermally conductive solid materials. Int. J. Heat. Mass Transf. 163, 120445 (2020).

    Article  Google Scholar 

  51. Newman, J. & Battaglia, V. in The Newman Lectures on Mathematics 1st edn (Pan Stanford Publishing, 2018).

  52. Vlcek, L., Yang, S., Ziatdinov, M., Kalinin, S. & Vasudevan, R. Statistical physics-based framework and Bayesian inference for model selection and uncertainty quantification. Microsc. Microanal. 25, 130–131 (2019).

    Article  ADS  Google Scholar 

  53. Valleti, M., Vlcek, L., Ziatdinov, M., Vasudevan, R. K. & Kalinin, S. V. Reconstruction and uncertainty quantification of lattice Hamiltonian model parameters from observations of microscopic degrees of freedom. J. Appl. Phys. 128, 214103 (2020).

    Article  ADS  Google Scholar 

  54. Vlcek, L., Maksov, A., Pan, M., Vasudevan, R. K. & Kalinin, S. V. Knowledge extraction from atomically resolved images. ACS Nano 11, 10313–10320 (2017).

    Article  Google Scholar 

  55. Sivaraman, G. et al. Machine-learned interatomic potentials by active learning: amorphous and liquid hafnium dioxide. npj Comput. Mater. 6, 104 (2020).

  56. Behler, J. Perspective: machine learning potentials for atomistic simulations. J. Chem. Phys. 145, 170901 (2016).

    Article  ADS  Google Scholar 

  57. Rosenbrock, C. W. et al. Machine-learned interatomic potentials for alloys and alloy phase diagrams. npj Comput. Mater. 7, 24 (2021).

  58. Shahriari, B., Swersky, K., Wang, Z., Adams, R. P. & De Freitas, N. Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104, 148–175 (2015).

    Article  Google Scholar 

  59. Mueller, T., Hernandez, A. & Wang, C. Machine learning for interatomic potential models. J. Chem. Phys. 152, 050902 (2020).

  60. Wellawatte, G. P., Seshadri, A. & White, A. D. Model agnostic generation of counterfactual explanations for molecules. Chem. Sci. 13, 3697–3705 (2022).

    Article  Google Scholar 

  61. Champion, K., Lusch, B., Kutz, J. N. & Brunton, S. L. Data-driven discovery of coordinates and governing equations. Proc. Natl Acad. Sci. USA 116, 22445–22451 (2019).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  62. Rudy, S.H., Brunton, S. L., Proctor, J. L. & Kutz, J. N. Data-driven discovery of partial differential equations. Sci. Adv. 3, e1602614 (2017).

    Article  ADS  Google Scholar 

  63. Liu, Y., Kutz, J. N. & Brunton, S. L. Hierarchical deep learning of multiscale differential equation time-steppers. Philos. Trans. R. Soc. A 380, 20210200 (2022).

    Article  ADS  MathSciNet  Google Scholar 

  64. Dyck, O., Kim, S., Kalinin, S. V. & Jesse, S. Placing single atoms in graphene with a scanning transmission electron microscope. Appl. Phys. Lett. 111, 113104 (2017).

  65. Dyck, O. et al. Building structures atom by atom via electron beam manipulation. Small 14, 1801771 (2018).

  66. Ziatdinov, M. et al. Causal analysis of competing atomistic mechanisms in ferroelectric materials from high-resolution scanning transmission electron microscopy data. npj Comput. Mater. 6, 127 (2020).

    Article  ADS  Google Scholar 

  67. Bareinboim, E. & Pearl, J. Causal inference and the data-fusion problem. Proc. Natl Acad. Sci. USA 113, 7345–7352 (2016).

    Article  ADS  Google Scholar 

  68. Pearl, J. The seven tools of causal inference, with reflections on machine learning. Commun. ACM 62, 54–60 (2019).

    Article  Google Scholar 

  69. Galles, D. & Pearl, J. Axioms of causal relevance. Artif. Intell. 97, 9–43 (1997).

    Article  MathSciNet  MATH  Google Scholar 

  70. Pearl, J. On the interpretation of do(x). J. Causal Inference 7, 6 (2019).

    Article  MathSciNet  Google Scholar 

  71. Pearl, J. A linear ‘microscope’ for interventions and counterfactuals. J. Causal Inference 5, 15 (2017).

    Article  MathSciNet  Google Scholar 

  72. Janzing, D. et al. Information-geometric approach to inferring causal directions. Artif. Intell. 182, 1–31 (2012).

    Article  MathSciNet  MATH  Google Scholar 

  73. Peters, J., Mooij, J. M., Janzing, D. & Scholkopf, B. Causal discovery with continuous additive noise models. J. Mach. Learn. Res. 15, 2009–2053 (2014).

    MathSciNet  MATH  Google Scholar 

  74. Janzing, D., Chaves, R. & Scholkopf, B. Algorithmic independence of initial condition and dynamical law in thermodynamics and causal inference. New J. Phys. 18, 13 (2016).

    Article  Google Scholar 

  75. Mooij, J. M., Peters, J., Janzing, D., Zscheischler, J. & Scholkopf, B. Distinguishing cause from effect using observational data: methods and benchmarks. J. Mach. Learn. Res. 17, 102 (2016).

    MathSciNet  MATH  Google Scholar 

  76. Lopez-Paz, D., Muandet, K. & Recht, B. The randomized causation coefficient. J. Mach. Learn. Res. 16, 2901–2907 (2015).

    MathSciNet  MATH  Google Scholar 

  77. Hernandez-Lobato, D., Morales-Mombiela, P., Lopez-Paz, D. & Suarez, A. Non-linear causal inference using Gaussianity measures. J. Mach. Learn. Res. 17, 39 (2016).

    MathSciNet  MATH  Google Scholar 

  78. Bottou, L. et al. Counterfactual reasoning and learning systems: the example of computational advertising. J. Mach. Learn. Res. 14, 3207–3260 (2013).

    MathSciNet  MATH  Google Scholar 

  79. Pearl, J. Causality: Models, Reasoning and Inference (Cambridge Univ. Press, 2009).

  80. Forré, P. & Mooij, J. M. Causal calculus in the presence of cycles, latent confounders and selection bias. In Proc. 35th Uncertainty in Artificial Intelligence Conference Vol. 115 (eds Ryan, P. A. & Vibhav, G.) 71–80 (PMLR, 2020).

Download references

Acknowledgements

The core idea and sections on imaging to models work was supported by the US Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering. The sections on theoretical descriptors was supported by US DOE, Office of Science, Office of Basic Energy Sciences Data, Artificial Intelligence and Machine Learning at DOE Scientific User Facilities. The causal inference and autoencoders section was supported by the Center for Nanophase Materials Sciences (CNMS), which is a US Department of Energy, Office of Science User Facility at Oak Ridge National Laboratory.

Author information

Authors and Affiliations

  1. Department of Materials Science and Engineering, University of Tennessee, Knoxville, TN, USA

    Sergei V. Kalinin

  2. Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Ayana Ghosh, Rama Vasudevan & Maxim Ziatdinov

  3. Computational Sciences and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Ayana Ghosh & Maxim Ziatdinov

Authors
  1. Sergei V. Kalinin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayana Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rama Vasudevan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maxim Ziatdinov
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.V.K. proposed the concept and wrote the initial draft of the opinion. A.G. contributed towards writing the section on the description of solids, structural descriptors, static and dynamic systems and making illustrative figures. R.V. contributed towards writing and editing the manuscript, especially the sections on the generative adversarial models and the parts on symmetry and disorder. M.Z. contributed parts on the variational autoencoders and causal learning.

Corresponding authors

Correspondence to Sergei V. Kalinin, Rama Vasudevan or Maxim Ziatdinov.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Physics thanks Philip Moriarty and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kalinin, S.V., Ghosh, A., Vasudevan, R. et al. From atomically resolved imaging to generative and causal models. Nat. Phys. 18, 1152–1160 (2022). https://doi.org/10.1038/s41567-022-01666-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41567-022-01666-0

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing