Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Automating crystal-structure phase mapping by combining deep learning with constraint reasoning

Nature Machine Intelligence volume 3, pages 812–822 (2021)Cite this article

Subjects

A preprint version of the article is available at arXiv.

Abstract

Crystal-structure phase mapping is a core, long-standing challenge in materials science that requires identifying crystal phases, or mixtures thereof, in X-ray diffraction measurements of synthesized materials. Phase mapping algorithms have been developed that excel at solving systems with up to several unique phase mixtures, where each phase has a readily distinguishable diffraction pattern. However, phase mapping is often beyond materials scientists’ capabilities and also poses challenges to state-of-the-art algorithms due to complexities such as the existence of dozens of phase mixtures, alloy-dependent variation in the diffraction patterns and multiple compositional degrees of freedom, creating a major bottleneck in high-throughput materials discovery. Here we show how to automate crystal-structure phase mapping. We formulate phase mapping as an unsupervised pattern demixing problem and describe how to solve it using deep reasoning networks (DRNets). DRNets combine deep learning with constraint reasoning for incorporating prior scientific knowledge and consequently require only a modest amount of (unlabelled) data. DRNets compensate for the limited data by exploiting and magnifying the rich prior knowledge about the thermodynamic rules governing the mixtures of crystals. DRNets are designed with an interpretable latent space for encoding prior-knowledge domain constraints and seamlessly integrate constraint reasoning into neural network optimization. DRNets surpass previous approaches on crystal-structure phase mapping, unravelling the Bi–Cu–V oxide phase diagram and aiding the discovery of solar fuels materials.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Crystal-structure phase mapping and Multi-MNIST-Sudoku.
Fig. 2: DRNets framework and the semantics of the latent space for different tasks.
Fig. 3: DRNets for crystal-structure phase mapping.
Fig. 4: Comparison of the activation maps produced by DRNets with other unsupervised approaches for the Bi–Cu–V oxide and Al–Li–Fe oxide systems.
Fig. 5: DRNets solution for the Bi–Cu–V oxide system.
Fig. 6: Quantitative results of DRNets for crystal-structure phase mapping.

Similar content being viewed by others

Data availability

Data are available from ‘UDiscoverIt: Materials’ (https://www.cs.cornell.edu/gomes/udiscoverit/?tag=materials) and also from GitHub (https://github.com/gomes-lab/DRNets-Nature-Machine-Intelligence). Source data are provided with this paper.

Code availability

Code is available from ‘UDiscoverIt: Materials’ (https://www.cs.cornell.edu/gomes/udiscoverit/?tag=materials) and also from GitHub (https://github.com/gomes-lab/DRNets-Nature-Machine-Intelligence).

References

  1. Stajic, J., Stone, R., Chin, G. & Wible, B. Rise of the machines. Science 349, 248–249 (2015).

    Article  Google Scholar 

  2. Szegedy, C. et al. Going deeper with convolutions. In Proc. 2015 IEEE Conference on Computer Vision and Pattern Recognition 1–9 (IEEE, 2015).

  3. Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. DeepFace: closing the gap to human-level performance in face verification. In Proc. 2014 IEEE Conference on Computer Vision and Pattern Recognition 1701–1708 (IEEE, 2014).

  4. Graves, A., Mohamed, A. R. & Hinton, G. Speech recognition with deep recurrent neural networks. In Proc. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing 6645–6649 (IEEE, 2013).

  5. Gil, Y., Greaves, M., Hendler, J. & Hirsh, H. Amplify scientific discovery with artificial intelligence. Science 346, 171–172 (2014).

    Article  Google Scholar 

  6. Ball, P. Learning from the big picture. Nat. Mater. 17, 1062 (2018).

    Article  Google Scholar 

  7. Tabor, D. P. et al. Accelerating the discovery of materials for clean energy in the era of smart automation. Nat. Rev. Mater. 3, 5–20 (2018).

    Article  Google Scholar 

  8. Sanchez-Lengeling, B. & Aspuru-Guzik, A. Inverse molecular design using machine learning: generative models for matter engineering. Science 361, 360–365 (2018).

    Article  Google Scholar 

  9. Sun, S. et al. Accelerated development of perovskite-inspired materials via high-throughput synthesis and machine-learning diagnosis. Joule 3, 1437–1451 (2019).

    Article  Google Scholar 

  10. Kusne, A. G. et al. On-the-fly closed-loop materials discovery via Bayesian active learning. Nat. Commun. 11, 5966 (2020).

    Article  Google Scholar 

  11. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  12. Chen, D. et al. Deep reasoning networks for unsupervised pattern de-mixingwith constraint reasoning. In Proc. 37th International Conference on Machine Learning (ICML-2020) Vol. 119, 1500–1509 (PMLR, 2020).

  13. Yato, T. & Seta, T. Complexity and completeness of finding another solution and its application to puzzles. IEICE Trans. Fundamentals Electron. Commun. Comput. Sci. 86, 1052–1060 (2003).

    Google Scholar 

  14. Rossi, F., Van Beek, P. & Walsh, T. Handbook of Constraint Programming (Elsevier, 2006).

  15. Gravel, S. & Elser, V. Divide and concur: a general approach to constraint satisfaction. Phys. Rev. E 78, 036706 (2008).

    Article  Google Scholar 

  16. Elser, V., Rankenburg, I. & Thibault, P. Searching with iterated maps. Proc. Natl Acad. Sci. USA 104, 418–423 (2007).

    Article  MathSciNet  Google Scholar 

  17. LeBras, R. et al. Constraint reasoning and kernel clustering for pattern decomposition with scaling. In International Conference on Principles and Practice of Constraint Programming 508–522 (Springer, 2011).

  18. d’Avila Garcez, A. & Lamb, L. C. Neurosymbolic AI: the 3rd wave. Preprint at https://arxiv.org/pdf/2012.05876.pdf (2020).

  19. LeCun, Y. et al. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

    Article  Google Scholar 

  20. Cohen, G., Afshar, S., Tapson, J. & van Schaik, A. EMNIST: Extending MNIST to handwritten letters. In Proc. 2017 International Joint Conference on Neural Networks (IJCNN) 2921–2926 (2017).

  21. Sabour, S., Frosst, N. & Hinton, G. E. Dynamic routing between capsules. In Advances in Neural Information Processing Systems 3856–3866 (NIPS, 2017).

  22. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

  23. Ludwig, A. Discovery of new materials using combinatorial synthesis and high-throughput characterization of thin-film materials libraries combined with computational methods. npj Comput. Mater. 5, 70 (2019).

    Article  Google Scholar 

  24. Green, M. L. et al. Fulfilling the promise of the materials genome initiative with high-throughput experimental methodologies. Appl. Phys. Rev. 4, 011105 (2017).

    Article  Google Scholar 

  25. Kusne, A. G., Keller, D., Anderson, A., Zaban, A. & Takeuchi, I. High-throughput determination of structural phase diagram and constituent phases using grendel. Nanotechnology 26, 444002 (2015).

    Article  Google Scholar 

  26. Stanev, V. et al. Unsupervised phase mapping of X-ray diffraction data by nonnegative matrix factorization integrated with custom clustering. npj Comput. Mater. 4, 43 (2018).

    Article  Google Scholar 

  27. Gomes, C. P. et al. Crystal: a multi-agent AI system for automated mapping of materials’ crystal structures.MRS Commun. 9, 600–608 (2019).

    Article  Google Scholar 

  28. Park, W. B. et al. Classification of crystal structure using a convolutional neural network. IUCrJ 4, 486–494 (2017).

    Article  Google Scholar 

  29. Oviedo, F. et al. Fast and interpretable classification of small X-ray diffraction datasets using data augmentation and deep neural networks. npj Comput. Mater. 5, 60 (2019).

    Article  Google Scholar 

  30. Lee, J. W., Park, W. B., Lee, J. H., Singh, S. P. & Sohn, K. S. A deep-learning technique for phase identification in multiphase inorganic compounds using synthetic XRD powder patterns. Nat. Commun. 11, 86 (2020).

    Article  Google Scholar 

  31. Bunn, J. K. et al. Generalized machine learning technique for automatic phase attribution in time variant high-throughput experimental studies. J. Mater. Res. 30, 879–889 (2015).

    Article  Google Scholar 

  32. Rossouw, D. et al. Multicomponent signal unmixing from nanoheterostructures: overcoming the traditional challenges of nanoscale X-ray analysis via machine learning. Nano Lett. 15, 2716–2720 (2015).

    Article  Google Scholar 

  33. Le Bras, R. et al. Challenges in materials discovery—synthetic generator and real datasets. In Proc. Twenty-Eighth AAAI Conference on Artificial Intelligence Vol. 28, 438–443 (AAAI, 2014).

  34. Chen, D., Xue, Y., Chen, S., Fink, D. & Gomes, C. Deep multi-species embedding. In Proc. Twenty-Sixth International Joint Conference on Artificial Intelligence 3639–3646 (IJCAI, 2017); https://doi.org/10.24963/ijcai.2017/509

  35. Chen, D., Xue, Y. & Gomes, C. P. End-to-end learning for the deep multivariate probit model. In Proc. International Conference on Machine Learning 932–941 (2018).

  36. Sullivan, B. L. et al. The eBird enterprise: an integrated approach to development and application of citizen science. Biol. Conserv. 169, 31–40 (2014).

    Article  Google Scholar 

  37. Sutherland, W. J. et al. A horizon scan of emerging issues for global conservation in 2019. Trends Ecol. Evol. 34, 83–94 (2019).

    Article  Google Scholar 

  38. Kong, S., Guevarra, D., Gomes, C. P. & Gregoire, J. M. Materials representation and transfer learning for multi-property prediction. Appl. Phys. Rev. 8, 021409 (2021).

    Article  Google Scholar 

  39. Mirza, M. & Osindero, S. Conditional generative adversarial nets. Preprint at https://arxiv.org/pdf/1411.1784.pdf (2014).

  40. Hu, Z., Ma, X., Liu, Z., Hovy, E. & Xing, E. Harnessing deep neural networks with logic rules. In Proc. 54th Annual Meeting of the Association for Computational Linguistics (Association for Computational Linguistics, Berlin) https://aclanthology.org/P16-1228 (2016).

  41. Xu, J., Zhang, Z., Friedman, T., Liang, Y. & Broeck, G. V. D. A semantic loss function for deep learning with symbolic knowledge. In Proc. International Conference on Machine Learning. 5502–5511 (2018).

  42. You, J., Ying, R., Ren, X., Hamilton, W. L. & Leskovec, J. GraphRNN: generating realistic graphs with deep auto-regressive models. In Proc. International Conference on Machine Learning. 5708–5717 (2018).

  43. Kingma, D. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/pdf/1412.6980.pdf (2014).

Download references

Acknowledgements

The development of DRNets was supported by the US National Science Foundation, under the Expeditions in Computing award CCF-1522054 (C.P.G., B.S., J.M.G., D.C., S.A., Y.B. and W.Z.) and award CNS-1059284 (C.P.G.). DRNets for phase mapping and corresponding experimental work were also supported by the US AFOSR Multidisciplinary University Research Initiative (MURI) under award FA9550-18-1-0136 (R.B.v.D., C.P.G., B.S., J.M.G., D.C., Y.B. and S.A.), a compute cluster under the Defense University Research Instrumentation Program (DURIP), award W911NF-17-1-0187 (C.P.G.) and an award from the Toyota Research Institute (J.M.G., C.P.G., D.C., Y.B. and S.A.). Solar fuels experiments were supported by the US Department of Energy (DOE) under award DESC0004993 (J.M.G., D.A. and L.Z.) and solar photochemistry analysis in the context of the DRNets solution was supported by the US DOE under award DE-SC0020383 (J.M.G. and D.A.). We also thank J. Bai for assistance with running the IAFD baseline, A. Shinde for photoelectrochemistry experiments and R. Berstein for assistance with figure generation.

Author information

Authors and Affiliations

  1. Department of Computer Science, Cornell University, Ithaca, NY, USA

    Di Chen, Yiwei Bai, Sebastian Ament, Wenting Zhao, Bart Selman & Carla P. Gomes

  2. Division of Engineering and Applied Science and Liquid Sunlight Alliance, California Institute of Technology, Pasadena, CA, USA

    Dan Guevarra, Lan Zhou & John M. Gregoire

  3. Department of Materials Science and Engineering, Cornell University, Ithaca, NY, USA

    R. Bruce van Dover

Authors
  1. Di Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiwei Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sebastian Ament
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenting Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dan Guevarra
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bart Selman
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. R. Bruce van Dover
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. John M. Gregoire
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Carla P. Gomes
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.P.G. conceived and managed the overall study. J.M.G. and C.P.G. conceived and managed the crystal-structure phase mapping project. D.C. and C.P.G. conceived the MNIST-Sudoku project. D.C. and C.P.G. conceptualized the DRNets. D.C. developed and implemented DRNets, in particular DRNets for MNIST-Sudoku and crystal-structure phase mapping. Y.B. performed the large-scale experiments, assisted on implementing DRNets for MNIST-Sudoku, and carried out baseline comparisons for MNIST-Sudoku. S.A. performed background subtraction for the Bi–Cu–V–O system. W.Z. implemented baselines for crystal-structure phase mapping and assisted on generating MNIST-Sudoku data. L.Z. and D.G. generated phase mapping datasets and interpreted and validated solutions. C.P.G., D.C. and J.M.G. were the main authors of the manuscript, with contributions from B.S. and R.B.v.D. and comments from all authors.

Corresponding authors

Correspondence to John M. Gregoire or Carla P. Gomes.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Machine Intelligence thanks Artur Garcez, Olga Kononova and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 The transformation flow for Deep Reasoning Networks and examples of continuous relaxations.

a, The process of formulating a problem into the DRNets framework. (i) We start by formulating tasks as data-driven constrained optimization problems, with discrete and continuous variables. For example, the data-driven constrained optimization task in MNIST-Sudoku is to demix images of two overlapping Sudokus such that the demixed Sudokus satisfy the Sudoku constraints and their reconstruction loss is minimized. Furthermore, in this formulation for MNIST-Sudoku, we assume that a the generative decoder (cGAN) reconstructs the demixed Sudoku digit images using a two-part latent space that encodes the digit probabilities and shapes and that the two-part latent space is produced by two convolutional neural networks (ResNet) and is subject to the Sudoku constraints. (ii) This data-driven constrained optimization problem is converted into a data-driven unconstrained optimization problem using Lagrangean relaxation, which essentially moves the constraints to the objective function, with associated penalty weights. (iii) We use entropy-based continuous relaxations to encode and replace discrete (non-differentiable) constraints with continuous functions, such as sparsity, cardinality, the all-different constraint, and logical constraints. The objective function combines two components: the reconstruction loss of the generative decoder (which for Sudoku corresponds to the reconstruction of the demixed overlapping digit images), and the reasoning loss (which for Sudoku corresponds to the penalty weighted entropy-based continuous function that capture the Sudoku rules). The result of these transformations is the DRNets data-driven unconstrained optimization formulation. (iv) DRNets optimize the overall objective function using constraint-aware stochastic gradient descent (SGD). Note that we refer to these problems as data-driven problems since although we assign semantics to the structured latent-space (probabilities and shapes), their full meaning is ultimately determined by the data. b, Examples of the continuous relaxation of discrete constraints. ei,j, Pi, Qi, PM, and H, represent indicator variables denoting if a given input image contains a given digit, the discrete distribution over digits 1 to 4, the discrete distribution over digits 5 to 8, the discrete distribution over values 1 to M, and the entropy function, respectively. See notation and further details in Supplementary Methods.

Extended Data Fig. 2 The performance of DRNets on Multi-MNIST-Sudoku tasks with different dataset scales.

Learning over multiple instances substantially (especially, for the 9x9 cases) improves the performance of DRNets. Nevertheless, DRNets can reach 99% Sudoku accuracy with only 100 Multi-MNIST-Sudoku instances, a considerable smaller amount of data compared to standard deep learning approaches.

Source data

Extended Data Fig. 3 DRNets for Multi-MNIST-Sudoku.

DRNets perform end-to-end deep reasoning by using a convolutional neural network to encode an interpretable structured latent space that is used by a fixed generative decoder, a conditional generative adversarial network (cGAN), to reconstruct the input mixed digits. The interpretable structured latent space also allows the encoding of reasoning constraints, which enforce that the latent space adheres to prior knowledge about Sudoku rules. Prior knowledge also includes digit prototypes, which are used to pre-train and build the fixed decoder’s generative module. An overall objective combines responses from the fixed generative decoder and the reasoning module and is optimized using constraint-aware stochastic gradient descent and backpropagation.

Extended Data Fig. 4 Comparison of the performance of different methods for Multi-MNIST-Sudoku.

We show the “solving time” for unsupervised DRNets and its ablation variants and “test time + training time” for supervised baselines. The test time for CapsuleNet/ResNet + local search includes the local search time. Note that we used two different local search algorithms for 4x4 cases and 9x9 cases. “local_search1” performs an enumeration for the top-2 likely digits in all 16 cells to try to satisfy Sudoku rules. For 9x9 cases, it is impossible to enumerate the top-2 likely digits for 81 cells (281). Therefore, “local_search2” conducts a depth-first search for digits in each cell from most likely to less likely until it finds a valid Sudoku combination, which is faster than “local_search1”. For 4x4 cases, we also applied exhaustive search for all methods, where we enumerate all possible 4x4 Sudokus and return the one with the highest likelihood given our predictions. Note that such strategy is not feasible for 9x9 Sudokus, given there are around 6.67 × 1021 9x9 Sudokus. The ablation study of removing the reasoning modules (DRNets w/o Reasoning) shows that not only does the Sudoku accuracy degrades, the digit accuracy also degrades, especially for 9x9 Sudokus. The ablation study of replacing the cGAN with a (weaker) standard learnable decoder, without prior knowledge about single digits (DRNets w/o cGAN) shows that both the Sudoku and digit accuracy degrades dramatically.

Extended Data Fig. 5 Comparison of the phase patterns discovered by different methods vs. the ground truth phases for the Al-Li-Fe oxide system.

For each phase we plot the pattern of the recongized phase and the ICDD stick patterns. While the phases discovered by DRNets closely match the ground truth phases, some of the IAFD and NMF-k’s phases do not match well the ground truth phases (for example, phase 3 (IAFD) and phase 6 (NMF-k)) as also reflected in the phase fidelity loss (0.00002 (DRNets); 11.920 (IAFD); and 46.156 (NMF-k)); see also Fig. 6a).

Source data

Extended Data Fig. 6 Characterization of Bi-Cu-V oxide library for photoelectrocatalysis of the oxygen evolution reaction, a critical reaction for solar fuels technology.

After XRD and XRF measurements, a grid of compositions was characterized with chronoamperometry (CA) with 4 different light emitting diode (LED) illumination sources from which photocurrent (J) is calculated, as well as cyclic voltammetry with 3.2 eV illumination (CV) from the photoelectrochemical power generation (P) is calculated. The resulting 5 performance metrics are plotted with respect to composition, and select pairs of points from 3 different phase fields in Extended Data Fig. 5 are indicated with labels C, D and F. The common false color scale from 0 to a maximum value is used for each metric, with maximum values of 1.8 mW cm−2 for P and 13.3, 14.1, 0.5, 0.045 mA cm−2 for J with 3.2, 2.8, 2.4 and 2.1 eV illumination, respectively. The anodic sweep of the CV is shown for 3 select samples labeled by their phase region. All 3 of these regions contain BiVO4, a well-known metal oxide photoanode, with much higher Cu concentration than typical Cu-free BiVO4 photoelectrocatalysts. All 3 noted phase regions contain BiVO4 and Cu3(VO4)2 with D and F additionally containing Cu2BiVO6 and Cu2V2O7, respectively. The different compositions and phase combinations lead to different performances, in particular the 3 phase region F exhibits higher photocurrent at low applied bias (see inset) and higher photocurrent with 2.1 eV illumination, which are 2 critical properties for BiVO4 photoanodes that have been historically difficult to optimize. Despite common belief that phase mixtures are deleterious to photoactivity, these results demonstrate alloying and optimal phase mixtures as promising directions for photoanode discovery and optimization.

Source data

Extended Data Fig. 7 Different components of DRNets for the different tasks.

N/A.

Supplementary information

Source data

Source Data Fig. 1

XRD and phase mapping results data for Fig. 1a–f.

Source Data Fig. 4

The phase mapping data and the loss heatmap data of Fig. 4.

Source Data Fig. 5

DRNets phase mapping solution data for Fig. 5.

Source Data Fig. 6

Statistical source data of the phase activation accuracy curve.

Source Data Extended Data Fig. 2

Source data for the down-scale performance curve.

Source Data Extended Data Fig. 5

Al–Li–Fe oxide solution phase pattern data for the different methods and ground truth, for Extended Data Fig. 5.

Source Data Extended Data Fig. 6

Photoelectrocatalysis characterization data for the Bi–Cu–V oxide library, for Extended Data Fig. 6.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, D., Bai, Y., Ament, S. et al. Automating crystal-structure phase mapping by combining deep learning with constraint reasoning. Nat Mach Intell 3, 812–822 (2021). https://doi.org/10.1038/s42256-021-00384-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00384-1

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics