The development of high-resolution imaging methods such as electron and scanning probe microscopy and atomic probe tomography have provided a wealth of information on the atomic structure and functionalities of solids. The availability of this data in turn necessitates the development of approaches to derive quantitative physical information, akin to how the development of scattering techniques provided insights into the atomic-level structure of a vast array of materials in the twentieth century. Here we argue that the endeavour to understand local imaging data requires the adaptation of classical macroscopic definitions. For example, the notion of symmetry can be introduced locally only in a probabilistic sense that must balance our knowledge of the material’s physics and other experimental data points with the imaging data at hand. At the same time, this wealth of local data can enable fundamentally new approaches for the description of solids, based on the construction of statistical and physical models that can ‘generate’ the observed structures. Finally, we note that the availability of observational data opens pathways towards exploring causal mechanisms underpinning solid structure and functionality.
This is a preview of subscription content, access via your institution
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
The data pertinent to this article can be accessed via the corresponding references as cited in the relevant publications.
The codes pertinent to this article can be accessed via the corresponding references as cited in the relevant publications.
Kittel, C. Theory of antiferroelectric crystals. Phys. Rev. 82, 729–732 (1951).
Tagantsev, A. K., Cross, L. E. & Fousek, J. Domains in Ferroic Crystals and Thin Films (Springer, 2010).
Binder, K. & Young, A. P. Spin-glasses—experimental facts, theoretical concepts, and open questions. Rev. Mod. Phys. 58, 801–976 (1986).
Binder, K. & Reger, J. D. Theory of orientational glasses models, concepts, simulations. Adv. Phys. 41, 547–627 (1992).
Herbrych, J. et al. Block-spiral magnetism: an exotic type of frustrated order. Proc. Natl Acad. Sci. USA 117, 16226–16233 (2020).
Heidrich-Meisner, F., Sergienko, I. A., Feiguin, A. E. & Dagotto, E. R. Universal emergence of the one-third plateau in the magnetization process of frustrated quantum spin chains. Phys. Rev. B 75, 064413 (2007).
Zhang, H. et al. Anomalous magnetoresistance by breaking ice rule in Bi2Ir2O7/Dy2Ti2O7 heterostructure. Preprint at https://arxiv.org/abs/2011.09048 (2020).
Lin, L.-F., Zhang, Y., Moreo, A., Dagotto, E. & Dong, S. Frustrated dipole order induces noncollinear proper ferrielectricity in two dimensions. Phys. Rev. Lett. 123, 067601 (2019).
Mohanta, N., Christianson, A. D., Okamoto, S. & Dagotto, E. Signatures of a liquid-crystal transition in spin-wave excitations of skyrmions. Commun. Phys. 3, 229 (2020).
Patel, N. D., Mukherjee, A., Kaushal, N., Moreo, A. & Dagotto, E. Non-Fermi liquid behavior and continuously tunable resistivity exponents in the Anderson–Hubbard model at finite temperature Phys. Rev. Lett. 119, 086601 (2017).
Zhang, S.-S., Kaushal, N., Dagotto, E. & Batista, C. D. Spin–orbit interaction driven dimerization in one dimensional frustrated magnets. Phys. Rev. B 96, 214408 (2017).
Vugmeister, B. E. & Rabitz, H. Coexistence of the critical slowing down and glassy freezing in relaxor ferroelectrics. Phys. Rev. B 61, 14448–14453 (2000).
Grinberg, I., Suchomel, M. R., Davies, P. K. & Rappe, A. M. Predicting morphotropic phase boundary locations and transition temperatures in Pb- and Bi-based perovskite solid solutions from crystal chemical data and first-principles calculations. J. Appl. Phys. 98, 094111 (2005).
Keen, D. A. & Goodwin, A. L. The crystallography of correlated disorder. Nature 521, 303–309 (2015).
Cheetham, A. K., Bennett, T. D., Coudert, F. X. & Goodwin, A. L. Defects and disorder in metal organic frameworks. Dalton Trans. 45, 4113–4126 (2016).
Overy, A. R. et al. Design of crystal-like aperiodic solids with selective disorder-phonon coupling. Nat. Commun. 7, 10445 (2016).
Gerber, C. & Lang, H. P. How the doors to the nanoworld were opened. Nat. Nanotechnol. 1, 3–5 (2006).
Binnig, G., Quate, C. F. & Gerber, C. Atomic force microscope. Phys. Rev. Lett. 56, 930–933 (1986).
Binnig, G., Rohrer, H., Gerber, C. & Weibel, E. 7×7 reconstruction on Si(111) resolved in real space. Phys. Rev. Lett. 50, 120–123 (1983).
Pennycook, S. J. & Nellist, P. D. Scanning Transmission Electron Microscopy: Imaging and Analysis (Springer, 2011).
Krivanek, O. L. et al. Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy. Nature 464, 571–574 (2010).
Jiang, Y. et al. Electron ptychography of 2D materials to deep sub-angstrom resolution. Nature 559, 343–34 (2018).
Hytch, M. J. & Potez, L. Geometric phase analysis of high-resolution electron microscopy images of antiphase domains: example Cu3Au. Phil. Mag. A 76, 1119–1138 (1997).
Hytch, M. J., Putaux, J. L. & Thibault, J. Stress and strain around grain-boundary dislocations measured by high-resolution electron microscopy. Phil. Mag. 86, 4641–4656 (2006).
Vasudevan, R. K., Ziatdinov, M., Jesse, S. & Kalinin, S. V. Phases and Interfaces from real space atomically resolved data: physics-based deep data image analysis. Nano Lett. 16, 5574–5581 (2016).
Vasudevan, R. K. et al. Big data in reciprocal space: sliding fast Fourier transforms for determining periodicity. Appl. Phys. Lett. 106, 091601 (2015).
Rawat, W. & Wang, Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 29.9, 2352–2449 (2017).
Bates, R. H. T. & Mnyama, D. The status of practical Fourier phase retrieval. Adv. Electron. Electron Phys. 67, 1–64 (1986).
Pennycook, S. J. Structure determination through Z-contrast microscopy. Adv. Imaging Electron Phys. 123, 173–206 (2002).
Sayre, D. X-ray crystallography: the past and present of the phase problem. Struct. Chem. 13, 81–96 (2002).
Patterson, A. L. A Fourier series method for the determination of the components of interatomic distances in crystals. Phys. Rev. 46, 372–376 (1934).
Lin, W. Z. et al. Local crystallography analysis for atomically resolved scanning tunneling microscopy images. Nanotechnology 24, 415707 (2013).
He, Q., Woo, J., Belianinov, A., Guliants, V. V. & Borisevich, A. Y. Better catalysts through microscopy: mesoscale M1/M2 intergrowth in molybdenum-vanadium based complex oxide catalysts for propane ammoxidation. ACS Nano 9, 3470–3478 (2015).
Belianinov, A. et al. Identification of phases, symmetries and defects through local crystallography. Nat. Commun. 6, 7801 (2015).
Goodwin, D. A. K. A. L. The crystallography of correlated disorder. Nature 521, 303–309 (2015).
Kruschke, J. Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan 2nd edn (Academic Press, 2014).
Nelson, C. T. et al. Exploring physics of ferroelectric domain walls via Bayesian analysis of atomically resolved STEM data. Nat. Commun. 11, 12 (2020).
Thulasiraman, K. & Swamy, M. N. S. Graphs: Theory and Algorithms (Wiley, 2011).
Louizos, C. et al. Causal effect inference with deep latent-variable models. In Proc. 31st International Conference on Neural Information Processing Systems 6449–6459 (Curran Associates, 2017).
Khemakhem, I., Kingma, D., Monti, R. & Hyvarinen, A. Variational autoencoders and nonlinear ICA: a unifying framework. In International Conference on Artificial Intelligence and Statistics 2207–2217 (PMLR, 2020).
Schölkopf, B. et al. Toward causal representation learning. Proc. IEEE 109, 612–634 (2021).
Ziatdinov, M. et al. Building and exploring libraries of atomic defects in graphene: scanning transmission electron and scanning tunneling microscopy study. Sci. Adv. 5, eaaw898 (2019).
Salakhutdinov, R. Learning deep generative models. Annu. Rev. Stat. Appl. 2, 361–385 (2015).
Alqahtani, H., Kavakli-Thorne, M. & Kumar, G. Applications of generative adversarial networks (GANs): an updated review. Arch. Comput. Methods Eng. 28.2, 525–552 (2021).
Borisevich, A. Y. et al. Exploring mesoscopic physics of vacancy-ordered systems through atomic scale observations of topological defects. Phys. Rev. Lett. 109, 065702 (2012).
Li, Q. et al. Quantification of flexoelectricity in PbTiO3/SrTiO3 superlattice polar vortices using machine learning and phase-field modeling. Nat. Commun. 8, 1468 (2017).
Li, W., Bazant, M. Z. & Zhu, J. A physics-guided neural network framework for elastic plates: comparison of governing equations-based and energy-based approaches. Comput. Methods Appl. Mech. Eng. 383, 113933 (2021).
Surya Effendy, J. S. A. M. Z. B. Analysis, design, and generalization of electrochemical impedance spectroscopy (EIS) inversion algorithms. J. Electrochem. Soc. 167, 106508 (2020).
Zhao, Hongbo, B., R. D. & Bazant, MartinZ. Image inversion and uncertainty quantification for constitutive laws of pattern formation. J. Comput. Phys. 436, 110279 (2021).
Cottrill, AntonL. et al. Simultaneous inversion of optical and infra-red image data to determine thermo-mechanical properties of thermally conductive solid materials. Int. J. Heat. Mass Transf. 163, 120445 (2020).
Newman, J. & Battaglia, V. in The Newman Lectures on Mathematics 1st edn (Pan Stanford Publishing, 2018).
Vlcek, L., Yang, S., Ziatdinov, M., Kalinin, S. & Vasudevan, R. Statistical physics-based framework and Bayesian inference for model selection and uncertainty quantification. Microsc. Microanal. 25, 130–131 (2019).
Valleti, M., Vlcek, L., Ziatdinov, M., Vasudevan, R. K. & Kalinin, S. V. Reconstruction and uncertainty quantification of lattice Hamiltonian model parameters from observations of microscopic degrees of freedom. J. Appl. Phys. 128, 214103 (2020).
Vlcek, L., Maksov, A., Pan, M., Vasudevan, R. K. & Kalinin, S. V. Knowledge extraction from atomically resolved images. ACS Nano 11, 10313–10320 (2017).
Sivaraman, G. et al. Machine-learned interatomic potentials by active learning: amorphous and liquid hafnium dioxide. npj Comput. Mater. 6, 104 (2020).
Behler, J. Perspective: machine learning potentials for atomistic simulations. J. Chem. Phys. 145, 170901 (2016).
Rosenbrock, C. W. et al. Machine-learned interatomic potentials for alloys and alloy phase diagrams. npj Comput. Mater. 7, 24 (2021).
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P. & De Freitas, N. Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104, 148–175 (2015).
Mueller, T., Hernandez, A. & Wang, C. Machine learning for interatomic potential models. J. Chem. Phys. 152, 050902 (2020).
Wellawatte, G. P., Seshadri, A. & White, A. D. Model agnostic generation of counterfactual explanations for molecules. Chem. Sci. 13, 3697–3705 (2022).
Champion, K., Lusch, B., Kutz, J. N. & Brunton, S. L. Data-driven discovery of coordinates and governing equations. Proc. Natl Acad. Sci. USA 116, 22445–22451 (2019).
Rudy, S.H., Brunton, S. L., Proctor, J. L. & Kutz, J. N. Data-driven discovery of partial differential equations. Sci. Adv. 3, e1602614 (2017).
Liu, Y., Kutz, J. N. & Brunton, S. L. Hierarchical deep learning of multiscale differential equation time-steppers. Philos. Trans. R. Soc. A 380, 20210200 (2022).
Dyck, O., Kim, S., Kalinin, S. V. & Jesse, S. Placing single atoms in graphene with a scanning transmission electron microscope. Appl. Phys. Lett. 111, 113104 (2017).
Dyck, O. et al. Building structures atom by atom via electron beam manipulation. Small 14, 1801771 (2018).
Ziatdinov, M. et al. Causal analysis of competing atomistic mechanisms in ferroelectric materials from high-resolution scanning transmission electron microscopy data. npj Comput. Mater. 6, 127 (2020).
Bareinboim, E. & Pearl, J. Causal inference and the data-fusion problem. Proc. Natl Acad. Sci. USA 113, 7345–7352 (2016).
Pearl, J. The seven tools of causal inference, with reflections on machine learning. Commun. ACM 62, 54–60 (2019).
Galles, D. & Pearl, J. Axioms of causal relevance. Artif. Intell. 97, 9–43 (1997).
Pearl, J. On the interpretation of do(x). J. Causal Inference 7, 6 (2019).
Pearl, J. A linear ‘microscope’ for interventions and counterfactuals. J. Causal Inference 5, 15 (2017).
Janzing, D. et al. Information-geometric approach to inferring causal directions. Artif. Intell. 182, 1–31 (2012).
Peters, J., Mooij, J. M., Janzing, D. & Scholkopf, B. Causal discovery with continuous additive noise models. J. Mach. Learn. Res. 15, 2009–2053 (2014).
Janzing, D., Chaves, R. & Scholkopf, B. Algorithmic independence of initial condition and dynamical law in thermodynamics and causal inference. New J. Phys. 18, 13 (2016).
Mooij, J. M., Peters, J., Janzing, D., Zscheischler, J. & Scholkopf, B. Distinguishing cause from effect using observational data: methods and benchmarks. J. Mach. Learn. Res. 17, 102 (2016).
Lopez-Paz, D., Muandet, K. & Recht, B. The randomized causation coefficient. J. Mach. Learn. Res. 16, 2901–2907 (2015).
Hernandez-Lobato, D., Morales-Mombiela, P., Lopez-Paz, D. & Suarez, A. Non-linear causal inference using Gaussianity measures. J. Mach. Learn. Res. 17, 39 (2016).
Bottou, L. et al. Counterfactual reasoning and learning systems: the example of computational advertising. J. Mach. Learn. Res. 14, 3207–3260 (2013).
Pearl, J. Causality: Models, Reasoning and Inference (Cambridge Univ. Press, 2009).
Forré, P. & Mooij, J. M. Causal calculus in the presence of cycles, latent confounders and selection bias. In Proc. 35th Uncertainty in Artificial Intelligence Conference Vol. 115 (eds Ryan, P. A. & Vibhav, G.) 71–80 (PMLR, 2020).
The core idea and sections on imaging to models work was supported by the US Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering. The sections on theoretical descriptors was supported by US DOE, Office of Science, Office of Basic Energy Sciences Data, Artificial Intelligence and Machine Learning at DOE Scientific User Facilities. The causal inference and autoencoders section was supported by the Center for Nanophase Materials Sciences (CNMS), which is a US Department of Energy, Office of Science User Facility at Oak Ridge National Laboratory.
The authors declare no competing interests.
Peer review information
Nature Physics thanks Philip Moriarty and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Kalinin, S.V., Ghosh, A., Vasudevan, R. et al. From atomically resolved imaging to generative and causal models. Nat. Phys. 18, 1152–1160 (2022). https://doi.org/10.1038/s41567-022-01666-0