Introduction

The recent times have witnessed unprecedented growth in materials engineering, enabled by major advances in manufacturing technologies like 3D printing, as well as modern methods of materials modeling and design. These advancements are creating avenues for material design to reach new heights in performance and life-related properties that would have been previously unachievable. Robust microstructure-informed hierarchical mechanistic modeling for structure-property-response relations is at the core of these developments. The models enable material designers to utilize the full potential of these materials by predicting important deformation and extreme behavior characteristics like fracture and fatigue response.

Image-based micromechanical models rely on the creation of 3D virtual microstructures, mostly through synthetic image reconstruction from 3D experimental data on sample microstructural volumes. Such experimental data may be extracted from electron backscatter diffraction (EBSD) and scanning electron microscopy (SEM) images of serially sectioned samples1,2, or from computed tomography (CT)3. Deterministic models of the microstructure, developed from limited experimental data4, are not necessarily representative of the statistical distributions, essential for effective micromechanical modeling. A more compelling approach is to create virtual microstructures that have statistical equivalence to experimentally observed distributions and correlations of morphological and crystallographic descriptors. These statistically equivalent virtual microstructures (SEVM)5,6,7,8 must capture the statistics of characteristic variables, such as grain or inclusion shape and size, crystallographic orientations, misorientations and their correlations9,10. The statistically equivalent RVE (SERVE) has been introduced in5,6,7,11,12, as the smallest SEVM with optimally convergent statistical functions of microstructural descriptors to the experimental measurements of the microstructure for efficient micromechanical simulations. The SERVE is further classified as microstructure-based SERVE (M-SERVE) for convergent statistics of morphological and crystallographic descriptor functions and as property-based SERVE (P-SERVE) for convergent target material response functions or properties in5,7,8.

A variety of models and tools have been developed in the Integrated Computational Materials Engineering field for generating image-based virtual material microstructures. These include the Dream3D software9,10,13, Neper14, MicroStructPy15, Kanapy16, MicroLib17, among others. Using Voronoi tesselation18, Laguerre-Voronoi tessellation19, sphere packing20,21,22,23, Markov random field algorithms24, cellular automata25, dilation-erosion26 or spatial correlations functions27,28,29, these tools generate SEVMs by selectively matching data from experimental microstructures. While these tools have been typically effective for microstructures with a unimodal distribution of heterogeneities, e.g., grains and particles, they tend to struggle when regenerating microstructures with multi-modal distributions of morphological and crystallographic features. For example, it is difficult to generate polycrystalline microstructures containing dispersed precipitates with correlated precipitate-grain boundary configurations8, or localized grain clusters having distinct characteristics from the rest of the microstructure, with acceptable fidelity30.

In practice, however, many of the emerging new materials exhibit significant microstructural complexities, e.g., in the form of multimodal grain size distributions as a consequence of their processing routes, such as deformation followed by heat treatments like annealing31, severe plastic deformation32, sintering33 and cold spray forming34, schematically depicted in Fig. 1a. Figure 1b, c show EBSD maps of a cold spray-formed (CSF) microstructure of the aluminum alloy AA7050 with a bimodal grain size distribution. Cold spray forming is a solid-state coating deposition technique, in which deposition layers are formed by the collision and subsequent bonding of particle-substrate and particle-particle interactions. Particle impact results in deformation-induced recrystallization in the CSF layers34, which leads to a bimodal polycrystalline microstructure composed of recrystallized ultra-fine grains (UFGs) and deformed coarse grains (CGs) within the prior particles. This is shown in Fig. 1b, c for two orthogonal planes, It shows deformed CGs in prior particles that are interjected into substrates containing UFGs. The bimodal grain size has been associated with higher yield strength and ductility35. While a grain size threshold can be used to separate CGs and UFGs, the kernel-averaged misorientation (KAM)36,37 is a more effective means of this separation for bimodal microstructures. Figure 1d shows zones and grain boundaries of UFGs (in red) and CGs (in black) using grain size and KAM thresholds. Image-based microstructural modeling of these materials with multimodal distributions is a highly challenging enterprise.

Fig. 1: Schematic of the cold-spray forming process and associated material microstructures.
figure 1

a Schematic diagram of the cold spray forming process, (b) low-resolution inverse pole figure (IPF) of the Z-orientation map in the XY plane showing standard triangle colors for the orientation maps, (c) low-resolution IPF of the Z-orientation map in the YZ plane, (d) high-resolution band contrast map of the boxed region in (c) with grain boundaries of ultra-fine grains (UFGs) and coarse grains (CGs) colored in black and red (grain boundaries delineated by an assigned misorientation angle), (e) histogram of number fraction and (f) histogram of area fraction of the equivalent grain diameter indicating multimodal grain size distribution.

Full size image

Figure 1e, f show histograms of the number and area fractions of the grain size, generated from the 2D EBSD maps of the polycrystalline microstructure in Fig. 1b. The size of the α − th grain in the ensemble with an area sα is represented in the histogram by an equivalent grain diameter \({d}^{\alpha }=2\sqrt{{s}^{\alpha }/\pi }\). Each bin in the histogram spans a grain size of 0.5 μm. The number and area fractions in the histogram for the i − th bin containing ni grains are respectively computed as:

$${n}_{f}^{i}=\frac{{n}^{i}}{{N}_{G}}\quad {{{\rm{and}}}}\quad {s}_{f}^{i}=\frac{\mathop{\sum }\nolimits_{\alpha = 1}^{{n}^{i}}{s}^{\alpha }}{S}$$
(1)

where NG is the total number of grains in the EBSD map with a total area S. The area fraction distribution of the grain size in Fig. 1f indicates a multi-modal distribution that deviates considerably from the log-normal distribution typically used in synthetic microstructure generators like Dream.3D13.

Computational studies involving bimodal microstructures are rather limited in the literature30,38,39. In most of these studies, the microstructure has been restricted to 2D, and the coarse grains are represented using simplified geometric constructs. A 2D bimodal microstructure, consisting of smaller equiaxed and large grain, has been generated in30 using the Neper software14 and a bimodal Laguerre-Voronoi tessellation method18. Larger grains (CGs) belonging to bimodal microstructures in38 have been represented as circles in a continuum matrix representing the smaller UFGs that are modeled using continuum plasticity. In39, the CGs have been represented by 2D ellipses that are modeled using continuum plasticity, whereas the UFGs have been represented using 2D Voronoi tessellation and modeled by crystal plasticity. For many emerging novel materials, the geometry of the CG domain is non-trivial, and simplified geometric models for multi-modal regions are likely to ignore some of the important features that are of consequence to the overall behavior and life.

Machine learning-based virtual microstructure generators are showing considerable promise in recent years, e.g., in40,41,42,43,44,45. Generative Adversarial Networks (GANs)46,47, which incorporate a deep learning architecture consisting of two neural networks competing against each other in a zero-sum game framework, are emerging as popular generative tools in this operation. GANs are a powerful class of neural networks for unsupervised learning that generate new, synthetic data resembling known data distribution. They have been used to generate topologically complex binary and three-phase microstructures in40,41,43,44,45,48. In43, the authors have compared the results of a microstructural model of a three-phase solid oxide fuel cell electrode, generated by GANs with those generated by the Dream3D software. Their studies have concluded that GANs are able to create more realistic three-phase microstructures in terms of visual resemblance, statistics, and topological properties. While their potential is well recognized, the application and effectiveness of GANs for creating multi-domain materials, such as polycrystals, remain to be determined. A GAN-based computational tool SliceGAN has been recently developed in41 for generating 3D microstructures from 2D data in binary, three-phase, or polycrystalline microstructures. While SliceGAN is able to generate high-fidelity binary and three-phase microstructures, the quality diminishes drastically for polycrystalline microstructures with poor grain boundary identification.

The present paper develops a methodology for robust 3D SEVMs and micromechanical models of complex multi-modal microstructures that are difficult to construct by either the conventional synthetic microstructure builders like Dream3D13 or the machine learning-based methods e.g., SliceGAN41. While the synthetic builders have traditionally captured microstructural information like grain size distributions, volume fractions, aspect ratio distributions, misorientation angle distributions, these quantities alone are not sufficient for characterizing the inherent complexities of multi-modal microstructures. GANs, on the other hand, provide a rapid sampling of high-dimensional and intractable density functions without specifying an apriori distribution function that can miss some important characteristics of the microstructure. For example, they do not accurately represent the morphological shapes and sizes of CGs and UFGs for the EBSD scans of Fig. 1b, c. This approach harnesses the advantages of Dream3D and GANs-based methods and couples them to overcome their individual shortcomings. Furthermore, the paper also introduces an effective multiscale method for utilizing the resulting SEVMs in micromechanical analysis for microstructure-property relations.

The present study is focused on models for a CSF Aluminum alloy AA7050 with a bimodal grain size distribution in the form of CGs and UFGs. While computational studies on CSF have typically focused on the particle deposition process49,50,51,52, very little has been done on modeling deformation leading to fatigue life in CSF materials with complex microstructures. Such studies are essential for comprehending the effect of processing on the material behavior53. A primary reason for this paucity is the difficulties encountered in generating 3D SEVMs and SERVEs for analyzing the complex bimodal microstructures, such as in Fig. 1. The large size discrepancy between the CGs and UFGs necessitates a multiscale modeling approach for computational efficiency. In this approach, the CG domains are explicitly represented as polycrystalline ensembles, whereas the UFGs are represented as a homogenized domain whose response is obtained by upscaling the high-resolution microstructural UFG behavior. Correspondingly, it is important to separate these domains in the construction of microstructural SEVMs.

This paper is organized as follows. The Results section shows the results of a methodology for generating 3D SEVMs of bimodal CSF microstructure. The method combines the SliceGAN code for generating the UFG and CG domains with the Dream3D code for packing the CG domain with grains matching the experimental morphological and crystallographic data. A Finite Element (FE) simulation strategy is subsequently developed for the mechanical response of the overall SEVM from the ensemble average of a range of statistical volume elements (SVEs) created from the parent SEVM. The Discussion section summarizes the importance of the methods developed. The Methods section discusses the specifics of the material and processing conditions, steps for coupling SliceGAN and Dream3D in the creation of the SEVMs, as well as the fundamentals of GANs along with modifications in the SliceGan code.

Results

A robust methodology is proposed in this paper to reconstruct 3D SEVMs of the CSF AA7050 alloy for modeling topologically complex 3D bimodal microstructures. This is followed by its implementation for simulating the mechanical response. The SEVMs are generated using a strategic combination of a GAN-based SliceGAN code41 and a conventional synthetic microstructure generator software Dream3D13. The material and processing condition specifics are discussed in the Methods section. The SEVM reconstruction methodology requires at least two orthogonal EBSD maps for input information on the bimodal domain morphology and an additional high-resolution map that provides high-quality statistical information on grain morphology and crystallography. The following sections detail the reconstruction and simulation process.

Characteristics of experimentally acquired EBSD images of the microstructure

Microstructures are analyzed using an Oxford Instruments EBSD system, attached to a JEOL 7000 field emission SEM. The EBSD images are treated using the MTEX toolbox in a Matlab environment54 and the acquired low-resolution EBSD maps in the XY and YZ planes are shown in Fig. 1b, c respectively. Each of these EBSD maps is scanned with a step size of 0.5 μm and has a size of 0.15 mm × 0.15 mm. Figure 1d corresponds to a 50 μm × 50 μm high-resolution EBSD map, scanned with a step size of 0.166 μm. The low-resolution EBSD maps are assumed to provide a statistically significant sample size of prior particles for the reconstruction of prior particle morphology, while the high-resolution map provides the necessary morphological and crystallographic information on the UFGs and CGs.

The impact during the cold spray (CS) deposition process produces a geometry change from roughly spherical particles to lune-like or imbricated prior particles in55. During impact, deformation-induced recrystallization takes place and results in microstructures with a bimodal grain size distribution. Figure 1b, c depict the characteristics of a CSF microstructure in the XY and YZ planes, impacted in the Z direction. The black zones in the EBSD maps correspond to a large number of recrystallized, UFGs. As seen in the XY and YZ plane EBSD maps respectively in Fig. 1b, c, the UFG domain generally envelops the prior particles in a complex 3D network. In the XY plane, the initial spherical geometry of the particles is preserved, while the YZ plane map shows evidence of significant deformation with impact, resulting in a deviation from the spherical shape. The CGs within the prior particles do not recrystallize but deform with the parent particle as a whole.

The deformed CGs can be differentiated from the recrystallized UFGs in the EBSD maps by using measures like the Kernel Average Misorientation (KAM) and the Grain Average KAM (GAKAM)36,37, in addition to the grain size distribution. The KAM measures, which are based on the lattice misorientation in the grain interior induced by deformation, are found to be ideal for the CS-formed microstructures. KAMi represents the average misorientation angle θij between a point i and its neighbor j, (j = 1 ⋯ n) within the same grain g. The GAKAMg is a property for a grain g that is calculated as the average KAMi over i = N points within the grain, expressed as:

$$KA{M}_{i}=\frac{1}{n}\mathop{\sum }\limits_{j=1}^{n}{\theta }_{ij},\quad {{{\rm{and}}}}\quad GAKA{M}_{g}=\frac{1}{N}\mathop{\sum }\limits_{i=1}^{N}\frac{1}{n}\mathop{\sum }\limits_{j=1}^{n}{\theta }_{ij}$$
(2)

The KAM and GAKAM vary with the geometrically necessary dislocations56 and are effective in differentiating the recrystallized UFGs from the CGs, which have a higher KAM value. Figure 2a, b respectively show the GAKAM map and the GAKAM histogram of the EBSD map in Fig. 1b. There is a direct correlation between the low values of GAKAM colored in blue in Fig. 2a and the zones around the prior particles in Fig. 1b. The GAKAM distribution exhibits a clear threshold between the two types of grain populations, where the peak around 0 degrees corresponds to the UFGs and the remaining parts of the histogram correspond to CGs.

Fig. 2: Grain-averaged kernel-averaged misorientation characteristics.
figure 2

a GAKAM map of the XY plane, and (b) GAKAM distribution (see Eq. (2)) weighted by surface.

Full size image

Generating 3D SEVM of complex bimodal AA7050 microstructures

The generation of 3D SEVM from orthogonal 2D EBSD maps of complex bimodal microstructures in Fig. 1b, c is based on the statistics of physically relevant microstructural features. The large size disparity between the CGs and UFGs necessitates a multiscale modeling approach, in which, the CG domains are explicitly represented as polycrystalline ensembles, whereas the UFGs are represented as homogenized domains whose response is obtained by upscaling the high-resolution microstructural behavior. Correspondingly, it is important to separate these domains in the construction of microstructural SEVMs as well as their simulations.

Challenges with conventional synthetic builders

The creation of 3D SEVMs from 2D EBSD maps of these bimodal microstructures, using commonly used synthetic generation software like Dream3D13 and SliceGAN41, are faced with various difficulties. The first corresponds to the inappropriate representation of the domains occupied by the UFGs due to the resolution differential between CG and UFG domains. Often this results in a fixed width UFG region between CG grains as seen in Fig. 3a. A second difficulty arises with the representation of anisotropy in the CG morphology in the prior particles shown in the EBSD scans of Fig. 1b, c.

Fig. 3: 3D SEVM created using Dream3D.
figure 3

a Dream3D-generated SEVM of cold spray formed AA7050 alloy, showing prior particles with packed CGs, and the UFG domain with thick black particle boundary regions, and (b) comparison of histograms showing the frequency of the aspect ratio of the prior particles from two orthogonal sections of the EBSD scans and the Dream3D image. The aspect ratio in this figure is measured as the ratio of the maximum to minimum principal axis length of an equivalent ellipse for the prior particles containing coarse grains (CGs).

Full size image

Figure 3 shows a SEVM of the CSF microstructure in Fig. 1, generated using the Dream3D software13. Major steps in the Dream3D generation include (i) separating the UFG and CG domains, (ii) identifying prior particles containing the CG domain in the EBSD map using a 2D watershed algorithm, (iii) packing prior particles in the SEVM domain from consideration of morphological characteristics, (iv) evaluating morphological and crystallographic characteristics of the CGs, and (v) packing CGs within the prior particles delineated in step (iii). The resulting SEVM in Fig. 3 has a number of discrepancies. First, the shape of prior particles in the CG domain is much more regular in comparison with those seen in the real microstructure of Fig. 1. Second, the representation of the UFG domain has a nearly constant width of approximately 2.625 μm, which is not characteristic of the real UFG domain. Furthermore, the morphological anisotropy of the Dream3D-generated grains, delineated by the histogram of aspect ratios in two orthogonal planes, is quite different from that for the EBSD maps. The aspect ratio in the paper is defined in terms of the ratio of the maximum to minimum principal axis length of an equivalent ellipse (in 2D) and ellipsoid (in 3D).

Next, the quality of GAN-generated CSF SEVMs using the SliceGAN software41 is investigated. The original SliceGAN code in41 is modified to receive multiple inputs in the form of Euler angles, quaternions, grain IDs, and grain sizes. Quaternions are used as inputs to the microstructure generation process for the microstructure shown in Fig. 10, whereas Euler Angles are used for post-processing. In addition to being non-singular, quaternions also showed a better image quality in trial simulations. The direct output from SliceGAN, using Quaternions, does not generate a well-defined microstructure and grains are difficult to distinguish. The post-processing steps necessary to obtain a distinct transition between grains and generate Fig. 4 are specified in the Methods: Section IV. Figure 4 compares the crystallographic and morphological characteristics of virtual microstructures generated by the SliceGAN software with data from EBSD maps. While the virtual UFG and CG domains generated by SliceGAN are slightly more representative of the real microstructure, the grains are generally equiaxed and larger than those observed in the EBSD maps. Furthermore, the generated texture is dissimilar to that seen in the EBSD maps. The misorientation angle shows two peaks, but the fraction of the peaks is incorrect. The crystallographic data can be improved through a sampling of the orientation from the experimental data and the misfit can be reduced using different grain size thresholds, and variations in the watershed filter. Moreover, the morphological data can be corrected through the improvement of the artificial network (AN) used in the GAN model by utilizing the difference shown in Fig. 10f and the statistical data from Fig. 4 as metrics in a hyper-parameter optimization algorithm.

Fig. 4: 3D SEVM created using SliceGAN.
figure 4

a Inverse pole figure (IPF) Z of the virtual microstructure created using SliceGAN, (b) comparison of pole figures of the EBSD data and synthetic microstructures, and comparison of (c) misorientation angle, (d) grain size, and (e) aspect ratio distribution of the virtual and EBSD microstructures.

Full size image

Integration of GANs in SliceGAN and the synthetic builder Dream3D for creating SEVMs

From the above study, it is evident that the Dream3D software is capable of packing grains conforming to the experimental statistics in EBSD maps, while GANs are capable of producing good binary or three-phase microstructures. Consequently, their complementary advantages may be strategically integrated using the GANs-based approaches in SliceGAN41 and the synthetic microstructure builder Dream3D13 to create the complex bimodal microstructures of the CSF alloy AA7050. The overall generation methodology is detailed in the Methods: Section II and Fig. 8.

Figure 5a shows a SEVM containing 24924 coarse grains in the prior particles, that is reconstructed by the integrated SliceGan-Dream3D toolkit. The figure illustrates details of the prior particle morphology with the included grain crystallography at different locations in the SEVM. The crystallographic orientations are assigned to the grains of the SEVM by sampling from the experimental EBSD data to yield statistical equivalence to the microstructural texture. The grain-to-grain misorientation angle distribution is augmented by introducing a crystallographic orientation swapping algorithm. Figure 5b, c compare the SEVM texture (pole figures), misorientation angle distribution, grain size distribution, and grain aspect ratio distribution with those from the EBSD maps. The texture, misorientation, and equivalent diameter generally show very good agreement. The aspect ratio distribution in Fig. 5e, however, shows a difference with the SEVM distribution peaking at a higher aspect ratio.

Fig. 5: 3D SEVM created using the integrated SliceGAN-Dream3D platform.
figure 5

a Integrated SliceGAN-Dream3D generated SEVM of the CSF AA7050 alloy containing CG and UFG domains from data in the orthogonal EBSD maps (Two prior particles with the included CG morphology and crystallography are shown in details); Comparison of the (b) texture (pole figures), (c) misorientation angle distribution, (d) grain size distribution, and (e) aspect ratio distribution, of the SEVM and EBSD data.

Full size image

FE simulation strategy of the SEVMs for mechanical response

The large size of the SEVM containing a high number of particles with included CGs makes direct FE analysis using crystal plasticity FEM, computationally prohibitive. To overcome this limitation, the SEVM can be divided into smaller volume elements for direct FE analysis to be performed over each volume element. Subsequently, the overall mechanical response of each volume element can be averaged to obtain the overall SEVM response. For the viability of such analysis in terms of overall accuracy and reduced computational time, it is important to determine the number of sub-volume elements that should be considered.

Two convergence tests are performed to assess this number. In the first set of tests, the microstructure-based statistical volume elements (M-SVE) are based on the convergence of morphological and crystallographic characteristics. Figure 6a–c plot the mean grain size, grain aspect ratio, and misorientation angle as a function of the number of M-SVEs sampled. During sampling of the SVEs from the SEVM, their boundaries can intersect a large number of grains. A special technique that averages SVE statistics with and without intersected grains is implemented for these plots. The red dashed line corresponds to the average value of the microstructural descriptor function for the entire SEVM in Fig. 5a. Convergence with respect to the red line, within a tolerance, estimates the minimum number of M-SVEs required for micromechanical analysis to make property assessment. The equivalent diameter reaches convergence for approximately 10 M-SVEs with a maximum error of 5.8% in Fig. 6a. The aspect ratio converges with a maximum error of 4.5% for approximately 30 SVEs in Fig. 6b. In Fig. 6c, the misorientation angle reaches convergence with a maximum error of 1.9% in approximately 20 SVEs. The oscillatory behavior of the misorientation angle convergence is a consequence of the bimodal distribution seen in Fig. 5c.

Fig. 6: Convergence plots of microstructural descriptors and properties for estimating microstructure and property-based statistical volume elements (M-SVE and P-SVE) taken from the SEVM created using the integrated SliceGAN-Dream3D platform.
figure 6

M-SVE convergence with respect to (a) misorientation angle, (b) equivalent diameter, and (c) aspect ratio, and (d) P-SVE convergence with respect to yield stress. The red dashed lines show the average value of the given microstructural characteristic/property for all grains in the SEVM, and the blue points correspond to the averaged property value over N SVEs.

Full size image

The second test corresponds to the convergence of property-based statistical volume elements (P-SVE) with respect to a chosen property or response function. The property is determined from micro-mechanical analysis of the SVEs. Each SVE shown in Fig. 7a, b, contains both CGs and UFGs in their domain that are modeled by two different constitutive models. The CG domain is modeled using a crystal plasticity (CP) constitutive model developed in8, while the UFG domain is modeled using an upscaled constitutive model (UCM) accounting for anisotropy, length-scale dependent flow stresses, tension-compression asymmetry, strain-rate dependency, and cyclic hardening under reversed loading conditions discussed in57. Parameters in the UCM, used for modeling the continuum UFG domain, are calibrated from CP finite element (CPFE) simulations of UFG SERVEs. In Fig. 6d the convergence in the SVE yield stress is shown through a plot as a function of the number of P-SVEs sampled. The plot has an oscillatory behavior for a lower number of samples but reaches convergence for approximately 40 samples. The M-SVE and P-SVE convergence analysis shows that around 40 SVE samples are necessary to capture the properties of the SEVM presented in Fig. 6d. This number of samples also ensures that the complexity in the geometry of the microstructure is statistically well-represented.

Fig. 7: Results of simulations of two different P-SVEs containing 301 and 262 CGs respectively subjected to tension loading in three orthogonal axis X, Y, and Z.
figure 7

a, b P-SVEs v-1 and v-2 respectively show the crystallographic orientations of the CGs (Left), Cauchy stress component σxx for the loading in the X direction (Center), and Cauchy stress component σzz for loading in the Z-direction (Right), (c) True stress-strain plot for the two P-SVEs under different loading directions. Simulations are performed with an applied strain rate of \(\dot{\varepsilon }=0.001{s}^{-1}\).

Full size image

Figure 7a, b show two different P-SVEs v-1 and v-2 with different morphological and crystallographic (Euler angles) characteristics that are simulated under uniaxial tension loading along three orthogonal axis X, Y, and Z respectively. The simulations are conducted for a constant strain rate of \(\dot{\varepsilon }=0.001{s}^{-1}\) with constrained rigid body modes. The SVE v-1 consists of 301 CGs in three clusters, whereas the SVE v-2 consists of 264 CGs in one dominant cluster. Figure 7a, b show the contour plots of the Cauchy stress components σxx for loading in the X direction and σzz for loading in the Z-direction respectively corresponding to a true strain of ~ 1.1 × 10−2. Significant stress concentration is observed at the interfaces of particles and CG-UFG domains. The true stress-strain plot for the two SVEs under different loading directions is depicted in Fig. 7c. The difference in local stress concentrations is reflected in the volume-averaged stress-strain response with different yield stresses and hardening slopes.

Discussions

As discussed in the introduction, material properties, especially extreme properties like fatigue life and ductility, are very sensitive to the local morphological and crystallographic manifestations of the material microstructure. Experimental studies of fatigue life in CSF Al alloys58 have observed that cracks initiate in the prior particle substructure, corresponding to the CG grain boundaries. Furthermore, the cracks that initiate at the CG boundaries have a stronger influence on failure than notches in the specimen. Studies in53 have observed that cracks nucleate near the surface from particle-particle bonding defects, and subsequently evolve in the CG domain along the particle boundaries. It is evident that the prior particle boundaries and the CG microstructure are major determinants of crack evolution in CSF alloys. This underscores the importance of developing high-fidelity SEVMs capturing geometrical anisotropies of the CG-UFG boundary and the CG microstructure, along with their implementation in a simulation tool for determining local mechanical response. Evolving local variables like stresses and plastic strain that affect the local mechanical behavior and crack evolution leading to fatigue or ductile failure59 are strongly affected by these morphological and crystallographic features that are not easily captured by conventional synthetic microstructure builders. The integrated SliceGAN-Dream3D toolkit, developed in this work, is a robust enabler in this regard with the ability to capture the local characteristics that lead to localized extreme behavior. This methodology uses the best attributes of the GAN-based microstructure generator SliceGAN and the grain-packing algorithm using the Dream3D code. SliceGAN learns the main characteristics of the binary microstructure and can generate different microstructures, resulting in a rich microstructure database that can be generated to study the heterogeneity of CSF microstructures. Dream3D, on the other hand, can readily fit the grain distributions and crystallographic texture in the microstructure. It is important to emphasize that three EBSD maps are necessary for the creation of the binary SEVM in 3D. However, the current model can be modified to incorporate 3D EBSD data as input using one discriminator with a 3D convolutional neural network instead of three separate discriminators for each plane.

While in this work, the GAN-based microstructure creation has been combined with the statistical aspects of synthetic microstructure builders, in principle it is possible to execute the entire process within a GAN framework. However, this would require a large amount of data from multiple EBSD maps for training the GAN models, which can be a significant bottleneck with respect to current data acquisition techniques. In reality, the amount of EBSD data can be rather limited for a given material under a specific processing condition. To overcome these issues, aspects of the GAN model can be enhanced to obtain the morphology and crystallography of complex polycrystalline microstructures. First, it is possible to modify images before passing them to the discriminator using a dataset augmentation technique. This step helps the model to modify and interpret every image differently. Another important aspect of GANs is the computation of the loss function. It has been shown that the gradient penalty term in Eq. (5) has limitations with respect to convergence, and regularization terms can be implemented to improve convergence and limit over-fitting60. Finally, the GANs can be modified to use EBSD data for different material pedigrees under different processing conditions, with supervised learning using the processing conditions as additional inputs in the GANs model. Beyond the GANs, diffusion models and variational auto-encoders (VAE) are both strong candidates for the microstructure generation process of polycrystalline microstructures61,62. However, a few potential shortcomings have thus far kept them from being applied to this class of problems. For example, the diffusion models do not have a good sampling speed, while the VAEs have typically demonstrated less quality than either the GANs or diffusion models. However, there is a good potential for these models to be coupled with the GANs for a more robust framework, which can be developed in the near future.

Methods

Material and manufacturing process

The chemical composition of AA7050 alloy, comprising the prior particle and substrate material, is given in Table 1. The cold spray deposition process is performed using a Gen II system from VRC Metal Systems with deposition parameters given in Table 2. Subsequently, the coating is removed from the substrate and machined to eliminate wrought material for the subsequent mechanical tests. Samples are then prepared for EBSD characterization by mechanical polishing followed by fine polishing.

Table 1 Chemical composition of the CSF AA7050 alloy (weight percent).
Full size table
Table 2 Information on the cold spray forming process parameters.
Full size table

Steps for generating SEVMs using a combination of SliceGAN and Dream3D

The generation of 3D SEVMs from orthogonal 2D EBSD maps in Fig. 1b, c accounts for physically relevant microstructural feature statistics, including grain size and grain aspect ratio distributions, texture, grain misorientation angle distributions, as well as the area fraction of the CG domain. The generation process entails three major steps as shown in Fig. 8.

Fig. 8: Sequence of steps for constructing a SEVM of a complex bimodal microstructure.
figure 8

CNN and DNN stand for Convolutional and Deconvolutional Neural Networks, respectively.

Full size image

Step 1: Separating the UFG and CG domains: The first step in constructing SEVMs of CSF microstructures entails the separation of the two categories of grains, viz. the recrystallized UFGs and the prior particle CGs. The two orthogonal EBSD maps in Fig. 1b, c are considered for providing the input microstructural data source. The UFGs and CGs have different dislocation densities, and hence the KAM data can be used to distinguish between the two groups of grains63,64. Figure 9a–d show the results of operations in step 1 to generate a binary microstructure from a given EBSD map.

Fig. 9: Constructing the 3D binary microstructure of the CSF Al alloy.
figure 9

Result of operations in step 1 (Separating the UFG and CG domains) towards creating SEVMS containing UFGs (blue) and CGs (red) using: (a) GAKAM threshold, (b) both GAKAM and grains size threshold, and a dilation/erosion operation to avoid the UFGs inside the CGs for the (c) XY plane and (d) YZ plane; 3D binary microstructure obtained in step 2 (Generating binary 3D SEVMs) directly from (e) SliceGAN and (f) after using the improved mask with a volume fraction threshold; (g) Comparing area fractions (\(\frac{{A}_{UFG}}{{A}_{Total}}\)) of the EBSD data and orthogonal slices of the 3D mask by just using SliceGAN, and from SliceGAN with improved masking.

Full size image

An initial separation of the UFG and CG is obtained by choosing a GAKAM threshold of 0.056°. Fig. 9a shows the UFG (in blue) and CG (in red) domains created by using only the GAKAM threshold. As recrystallization leads to much smaller grains, a grain size threshold criterion, in addition to the GAKAM threshold, is enacted to refine the result in Fig. 9a. Figure 9b shows the resulting UFG and CG domains created using both the GAKAM and a cluster size criteria obtained via a Gaussian blur filter to remove small, isolated clusters of CGs identified by GAKAM. Figure 9c, d show the final UFG and CG domains of the two orientation maps shown in Fig. 1b, c respectively after an additional erode/dilate operation is performed. The two domains have different geometrical features in the XY plane in Fig. 9c and the YZ plane in Fig. 9d. In the XY plane, the CGs preserve the initial circular geometry of the particles, while in the YZ plane, the CGs have elongated and imbricated geometries resulting from the deformation of particles during the impact.

Step 2: Generating binary 3D SEVMs from 2D data using a generative adversarial network (GAN): The second step in creating the SEVM is the generation of a 3D binary microstructure from the orthogonal 2D binary microstructures shown in Fig. 9c, d. For this operation, the GAN code SliceGAN41 is used to create 3D data from orthogonal 2D images. To accomplish this, SliceGAN combines a 3D generator and 3 separate 2D discriminators, one for each orthogonal direction. After generating a 3D microstructure, a slicing step is activated in the 3 mutually orthogonal directions to compare the statistics from the 3D volume to those from the 2D training images, as shown in Fig. 8. SliceGAN has two major advantages, viz. (i) the training data is reduced to a single 2D representative image for isotropic microstructures, and (ii) the method can be extended to anisotropic microstructures if three different image scans are provided as input. This is important for applications in which limited data is available to create virtual instantiations of the microstructure.

From visual identification, it is assumed that the YZ and XZ planes of the EBSD maps have similar statistics and can be represented by a single data set. Consequently, only two EBSD-based images are used to generate the 3D binary microstructure, viz. (i) the XY plane in Fig. 9c and (ii) the YZ plane in Fig. 9d with similar assumed statistics for the XZ plane. Figure 9e shows the 3D binary microstructure generated by SliceGAN, where the CG domain is colored in red and the UFG domain is colored in blue. While the results from SliceGAN represent the two phases well, the grain morphology is altered due to the truncation of grain data beyond the UFG-CG domain boundary, when applied directly as a mask in the Dream3D software. Consequently, an improvement of the mask is developed to better retain the morphological statistics of the grains after packing with better conformity to the area fractions of the EBSD data. First, the entire cubic volume is packed with grains, having equivalent statistical distributions of morphological features, using Dream3D. To accomplish this, the volume fraction vfrac of each grain intersected by the 3D mask from SilceGAN is defined using a relation:

$${v}_{frac}=\frac{{v}_{G}\bigcap {v}_{mask}}{{v}_{G}},$$
(3)

where vG is the volume of the grain and vmask the volume of the mask. If vfrac falls below a given threshold, i.e., vfrac ≤ vfth, the entire grain is omitted from the mask. Otherwise, the grain is retained. Figure 9f shows the improved mask using a volume fraction threshold of vfth = 0.1. Using these steps, the important characteristics of the 3D microstructure obtained by SliceGAN are preserved. Furthermore, the area fractions \(\frac{{A}_{UFG}}{{A}_{Total}}\) are closer to the EBSD, as shown in the statistics of Fig. 9.

The 3D binary microstructure generated using SliceGAN is visually similar to the geometry of the 2D binary microstructure generated in step 1, as shown in Fig. 9c, d. Figure 9g shows the area fraction of the CGs (red domain). For the area fraction distributions, 2D slices are sampled from the synthetic 3D microstructure in each orthogonal direction, and compared with data from corresponding EBSD windows of the same dimensions as the 3D slices. It is evident that the masking technique improves the area fraction of the domains in the three orthogonal directions. In the supplementary document, Supplementary Fig. 1 shows details from particles inside the SEVM shown in Fig. 5a, and Supplementary Fig. 2 depicts slices from orthogonal sections of the improved binary microstructure in Fig. 9f at different locations. The images confirm that the principal characteristics of the experimental data are reproduced, and the topological complexity of both the UFG and CG domains is well captured.

Step 3: Generating the 3D SEVM by grain packing using Dream3D: The third step entails coarse grain packing in the prior particles such that the morphological and crystallographic characteristics of the EBSD data are conserved. Dream3D13 is employed to match the EBSD morphological distributions, e.g., grain size and aspect ratio distributions of CGs in the prior particles. Figure 5a shows the reconstructed microstructure with 24924 coarse grains. Crystallographic orientations are assigned to the grains of the virtual microstructure by sampling from the experimental EBSD data, thus conforming to the microstructural texture as shown in Fig. 5b. The misorientation angle distribution is also improved by using a crystallographic orientation swapping algorithm, in which the orientations of two or more grains are swapped and the difference in the misorientation angle distributions between the newly instantiated SEVM and EBSD maps are compared using Kolmogorov-Smirnov (KS) test to analyze the goodness-of-fit with respect to the experimental data65. The KS test quantifies the maximum difference in the cumulative distribution functions of the virtual and experimental data sets. If the KS value of the new SEVM misorientation angle distribution is lower than the KS value before the swapping, the new orientations are accepted. This process is repeated up to a certain tolerance where the KS value converges.

It is evident that the final SEVM characteristics have good agreement with the morphological and crystallographic data in the EBSD maps. The pole figures of the SEVM, shown in Fig. 5b, are close to the experimental pole figures. The misorientation angle distribution, a crucial determinant of fatigue crack evolution, is also improved with the swapping algorithm as shown in Fig. 5c. Step 3 takes around 20 min to pack the grains and perform the grain swapping algorithm to fit the misorientation angle distribution on 24 CPUs Intel Xenon Gold Cascade Lake.

SliceGAN: a generative adversarial network (GAN) model

The GAN used in this work, along with its architecture and parameters, is based on the model in41. The binary microstructure in Fig. 9c, d is used as input for characterizing the UFG and CG domains in the SEVM, while the microstructure shown in Fig. 4a is used to delineate the orientation distribution in the form of Euler angles. The training run for the binary microstructure in Fig. 9e is performed on 1 GPU Nvidia A100, running for a period of 2 h. Once the model has been trained, new microstructures with different stochastic characteristics can be generated in less than one minute. Examples of different microstructures created by this process are shown in Supplementary Fig. 3.

The GAN implements two tasks, viz. a generator for learning to generate plausible data, and a discriminator for learning to distinguish the generator’s fake data from real data. As illustrated in Fig. 8, the discriminator is composed of five convolution layers, while the generator is composed of five transposed convolution layers and one softmax layer. The original SiliceGAN code is modified to handle more than 1 input in the form of txt or hd5f files. The hd5f files are used to bridge SliceGAN and Dream3D. The generated 3D microstructure is sectioned along three orthogonal planes and compared to the corresponding experimental data. Hence, every plane has its own loss function. The Wasserstein loss function66 is used in this work. In the Wasserstein GANs (WGANs) with gradient penalty, the Generator \({{{\mathcal{G}}}}\) and Discriminator \({{{\mathcal{D}}}}\) play a zero-sum game. The WGAN objective function is defined as:

$$\mathop{\min }\limits_{\begin{array}{c}{{{\mathcal{G}}}}\end{array}}\mathop{\max }\limits_{\begin{array}{c}{{{\mathcal{D}}}}\end{array}}\left\{{{\mathbb{E}}}_{\begin{array}{c}{{{{\boldsymbol{f}}}}}_{{{{\boldsymbol{s}}}}} \sim {{\mathbb{P}}}_{f}\end{array}}[D({{{{\boldsymbol{f}}}}}_{{{{\boldsymbol{s}}}}})]-{{\mathbb{E}}}_{\begin{array}{c}{{{\boldsymbol{r}}}} \sim {{\mathbb{P}}}_{r}\end{array}}[D({{{\boldsymbol{r}}}})]\right\}$$
(4)

where \({{\mathbb{P}}}_{r}\) and \({{\mathbb{P}}}_{f}\) are the real and fake data distributions respectively, and D() is the discriminator function computed on the real data subset r and the fake slice of the generated 3D microstructure fs. \({\mathbb{E}}[D({{{\boldsymbol{X}}}})]\) is the expectation of D(X) given a random variable X. In SliceGAN41, an additional gradient penalty term proposed in67, is considered for the calculation of the discriminator loss function as:

$${L}_{D}=\mathop{\min }\limits_{\begin{array}{c}{{{\mathcal{G}}}}\end{array}}\mathop{\max }\limits_{\begin{array}{c}{{{\mathcal{D}}}}\end{array}}\{{{\mathbb{E}}}_{\begin{array}{c}{{{{\boldsymbol{f}}}}}_{{{{\boldsymbol{s}}}}} \sim {{\mathbb{P}}}_{f}\end{array}}[D({{{{\boldsymbol{f}}}}}_{{{{\boldsymbol{s}}}}})]-{{\mathbb{E}}}_{\begin{array}{c}{{{\boldsymbol{r}}}} \sim {{\mathbb{P}}}_{r}\end{array}}[D({{{\boldsymbol{r}}}})]+\lambda {{\mathbb{E}}}_{\begin{array}{c}{{{\boldsymbol{k}}}} \sim {{\mathbb{P}}}_{{{{\boldsymbol{k}}}}}\end{array}}[{(\parallel {\nabla }_{{{{\boldsymbol{k}}}}}D({{{\boldsymbol{k}}}}){\parallel }_{2}-1)}^{2}]\}$$
(5)

where λ is the penalty coefficient, k = ε fs + (1 − ε)r with ε = [0, 1] being randomly sampled, and ∇k is the Laplacian with gradients represented in the coordinates k. Additionally, the generator loss function is defined in a more simplified way as:

$${L}_{G}=-\mathop{\min }\limits_{\begin{array}{c}{{{\mathcal{G}}}}\end{array}}\mathop{\max }\limits_{\begin{array}{c}{{{\mathcal{D}}}}\end{array}}{{\mathbb{E}}}_{\begin{array}{c}x \sim {{\mathbb{P}}}_{g}\end{array}}[D({{{{\boldsymbol{f}}}}}_{{{{\boldsymbol{s}}}}})]$$
(6)

Post-processing steps for generating 3D SEVMs using quaternions as inputs to the SliceGAN software

Quaternions are used as inputs to the SliceGAN code to avoid the singularities from Euler angles. The conversion from quaternions to Euler angles is performed using MTEX54. MTEX operations are performed in quaternions, and hence the utilization of quaternions is implicit. Euler angles are however chosen for visualization and post-processing purposes. The MTEX software accounts for crystal symmetry and specimen symmetry. In the quaternion representation of orientation space, an orientation is represented with a unit quaternion as:

$$\bar{{{{\bf{q}}}}}=({q}_{0},{{{\bf{q}}}})=\left\{{q}_{0},{q}_{1},{q}_{2},{q}_{3}\right\}\quad {{{\rm{with}}}}\,{{{\rm{the}}}}\,{{{\rm{constraint}}}}\quad {({q}_{0})}^{2}+{({q}_{1})}^{2}+{({q}_{2})}^{2}+{({q}_{3})}^{2}=1$$
(7)

Here q0 and q = (q1, q2, q3) represent the scalar and vector parts of the quaternion respectively. Euler angles are a more classical parameterization of the grain orientations. The final orientation is defined by three rotations O(n, ω) around the axis n by the angle ω. The Bunge convention defines three rotations O(z, φ1), O(x, ϕ) and O(z, φ2), where φ1, ϕ and φ2 are the Euler angles. The conversion of quaternions to Euler angles is performed using the following equation:

$$\left\{\begin{array}{c}{\varphi }_{1}\\ \phi \\ {\varphi }_{2}\\ \end{array}\right\}=\left\{\begin{array}{c}atan2({q}_{3},{q}_{0})-atan2({q}_{1},{q}_{2})+\pi /2\\ 2\,atan2(\sqrt{{q}_{1}^{2}+{q}_{2}^{2}},\sqrt{{q}_{0}^{2}+{q}_{3}^{2}})\\ atan2({q}_{3},{q}_{0})+atan2({q}_{1},{q}_{2})+3\pi /2\\ \end{array}\right\},$$
(8)

where atan2 is the four-quadrant inverse tangent. An additional modulo operation of is performed for the Euler angles φ1 and φ2 with a divisor of 2π.

The orientations generated by SliceGAN don’t allow to identify grains. Figure 10a, showing the Euler angle component ϕ1 from the SliceGAN results, depicts a considerable amount of noise. Two steps are required to get a distinguishable microstructure: a minimum grain size threshold shown in Fig. 10b, and a watershed filter shown in Fig. 10c. The watershed filter uses the gradients of the Euler angles, which are computed through Sobel operators68. Figure 10d shows the sum of the gradients of each Euler angle component \(SGE=\mathop{\sum }\nolimits_{i = 1}^{3}G{\phi }_{i}\), where G is the magnitude of the gradient computed as \(G=\sqrt{{G}_{x}+{G}_{y}+{G}_{z}}\) with Gx = Sx*ϕi, Gy = Sy*ϕi, and Gz = Sz*ϕi being the derivatives in the three orthogonal directions. Sx, Sy, and Sz are the kernels weighting functions in the Sobel operator, ϕi are the components of the Euler angles, and * is a convolutional operator. SGE values close to 1 correspond to grain boundaries. The gradient field is processed using a contrast filter and a Gaussian blur filter, as shown in Fig. 10e. The difference between the denoised SliceGAN output and the average Euler angle within each watershed basin is shown in Fig. 10f.

Fig. 10: Post-processing steps of the Euler angles obtained using the SliceGAN code.
figure 10

(a) Euler angle component ϕ1 directly obtained from SliceGAN, (b) voxel size and gradient denoising, (c) average Euler angle from step (b) within each watershed basin, (d) sum of gradients of each Euler angle component, (e) gradients from step (d) after contrast and Gaussian blur, and (f) difference between denoised Euler angles from step (b) and watershed basin average Euler angles from step (c).

Full size image

Data visualization

Data visualization was performed using Paraview69, MTEX54, Matplotlib70 and Inkscape.