Multi-plane denoising diffusion-based dimensionality expansion for 2D-to-3D reconstruction of microstructures with harmonized sampling

Acquiring reliable microstructure datasets is a pivotal step toward the systematic design of materials with the aid of integrated computational materials engineering (ICME) approaches. However, obtaining three-dimensional (3D) microstructure datasets is often challenging due to high experimental costs or technical limitations, while acquiring two-dimensional (2D) micrographs is comparatively easier. To deal with this issue, this study proposes a novel framework for 2D-to-3D reconstruction of microstructures called Micro3Diff using diffusion-based generative models (DGMs). Specifically, this approach solely requires pre-trained DGMs for the generation of 2D samples, and dimensionality expansion (2D-to-3D) takes place only during the generation process (i.e., reverse diffusion process). The proposed framework incorporates a new concept referred to as multi-plane denoising diffusion, which transforms noisy samples (i.e., latent variables) from different planes into the data structure while maintaining spatial connectivity in 3D space. Furthermore, a harmonized sampling process is developed to address possible deviations from the reverse Markov chain of DGMs during the dimensionality expansion. Combined, we demonstrate the feasibility of Micro3Diff in reconstructing 3D samples with connected slices that maintain morphologically equivalence to the original 2D images. To validate the performance of Micro3Diff, various types of microstructures (synthetic and experimentally observed) are reconstructed, and the quality of the generated samples is assessed both qualitatively and quantitatively. The successful reconstruction outcomes inspire the potential utilization of Micro3Diff in upcoming ICME applications while achieving a breakthrough in comprehending and manipulating the latent space of DGMs.


Introduction
The properties and physical behavior of a material are profoundly influenced by its microstructure, which encompasses the topology, distribution, and physical characteristics of the constituent phases.By leveraging computational mechanics [1,2] and integrated computational materials engineering (ICME) approaches [3,4], the relationship between the microstructure and properties can be systemically investigated for exploring the design space of materials [5][6][7][8].For precise modeling and understanding of complex material behavior in terms of microstructure-property linkage, conducting three-dimensional (3D) analysis and characterization of microstructures are imperative.Furthermore, the accuracy of computational analysis for investigating material properties relies on the quality of accessible 3D microstructural datasets.In order to acquire precise 3D information about microstructures, micro-computed tomography (micro-CT) has demonstrated its effectiveness as a tool allowing for the visualization of a material's internal structure and potential defects in a 3D domain.[9][10][11][12].Moreover, the use of micro-CT-based characterization of microstructures has been regarded as a reliable approach to understand material behavior with computational analysis, such as finite element analysis (FEA), particularly at the representative volume element (RVE) scale [9,[12][13][14][15][16].
However, the difficulty lies in acquiring an extensive database of microstructures with experimental analysis, primarily due to time and cost constraints.Furthermore, obtaining a 3D microstructural dataset is more challenging, as it requires serial sectioning views, in contrast to the relatively straightforward capturing independent two-dimensional (2D) micrographs.To address this challenge, numerous microstructure characterization and reconstruction (MCR) approaches employing a diverse range of microstructural descriptors have been proposed [13,[17][18][19][20][21][22][23][24].In general, these descriptor-based MCR methods obtain morphologically equivalent samples by iterative optimization of the discrepancy between the target and the current descriptors.Various kinds of descriptors, including simple volume fraction as well as highdimensional correlation functions such as n-point correlations [17,25] or lineal path functions [26], can be employed to quantify the morphology of the microstructure.Once specific descriptors are chosen, the optimization problem can be addressed using stochastic reconstruction techniques like the Yeong-Torquato algorithm [24].Moreover, it is possible to utilize differentiable descriptors to reconstruct equivalent 3D microstructural samples through the application of a gradient-based optimizer [17,27].In particular, Seibert et al. [20] proposed a 2D-to-3D reconstruction method to obtain realistic microstructure samples using differentiable microstructural descriptors and optimization algorithms provided by the openaccess MCRpy package [19].
Although MCR methods with microstructural descriptors have been demonstrated to be effective for microstructure reconstruction, a major concern of this approach is that it requires specific descriptors to be optimized.Thus, for achieving favorable reconstruction outcomes, the selection of appropriate descriptors is crucial considering the specific morphological attributes of the targeted microstructure.Meanwhile, there has been extensive research on generative models, such as variational autoencoders (VAEs) [28][29][30][31] and generative adversarial networks (GANs) [32][33][34], for reconstructing microstructures by learning the underlying distribution of data.For instance, Kim et al. [28] proposed a VAEbased framework for reconstruction of 2D microstructure samples from a deep-learned continuous microstructure space.They also demonstrated that inverse design is possible by establishing connections between the features in the latent space of VAE and the mechanical properties of materials.Fokina et al. [49] utilized the style-based GAN architecture for reconstruction of 2D microstructural samples that are close to the original samples in terms of area density and Euler characteristics distributions.In addition, 2D-to-3D microstructure reconstruction with generative models has been gaining significant attention recently [34][35][36].One of the remarkable works is the novel GAN architecture called SliceGAN, proposed by Kench and Cooper [34], which synthesizes 3D microstructure datasets from a single representative 2D image.They demonstrated their GAN-based model can effectively reconstruct 3D microstructures of various types of materials, including polycrystalline metals, ceramics, and battery electrodes On the other hand, VAEs suffer from a notable drawback, as the generated samples tend to be distorted and blurred [37,38], while GANs are susceptible to the issue of mode collapse and unstable training due to the adversarial loss function [39,40].In light of these concerns, the diffusion-based generative models (DGMs) [41][42][43] are currently gaining significant attention as a promising state-of-the-art generative model.In particular, there are two popular formulations of DGMs which are called the denoising diffusion probabilistic models (DDPMs) and the score-based generative models (SGMs).According to the formulation of DDPMs [43], a DGM can be perceived as a model of reverse Markovian chain, intended for progressive denoising a data structure, while SGMs [42] view DGMs as models for estimating gradients of data distribution with denoising score matching [44].The DGMs can also be generalized to the problem of solving reverse stochastic differential equations (SDEs) in order to transform the noise distribution into the data distribution [41,45].Due to the transformation of generation problem into the progressive reverse diffusion (i.e., denoising) process, DGMs have shown superior performance compared to GANs in generating highquality images [46], and they are not prone to mode collapse or unstable training which are commonly observed in GANs.
However, the possibility of using DGMs for 2D-to-3D reconstruction of microstructures has not been explored, according to the best knowledge of the authors.While some studies have suggested the use of DGMs for reconstructing 2D microstructure samples [5,47,48], the problem of dimensional expansion (2D-to-3D) has not been explored extensively due to the lack of knowledge in the latent space of DGMs.In order to bridge this gap, a novel dimensionality-expansion framework for 2D-to-3D reconstruction of microstructures based on denoising diffusion, called 'Micro3Diff', is proposed in this study.The proposed method does not require any 3D datasets, and it only requires a DGM trained for the generation of 2D images (2D-DGM).The feasibility of this approach lies in the fact that Micro3Diff targets only the manipulation of the reverse diffusion process for dimensionality expansion.In addition, this greatly facilitates the application of this framework as the commonly used 2D-DGM can be easily incorporated.The structure of this article with the key contributions of this work are summarized as follows.
(1) In the 'Preliminaries' section, the formulations of DGMs are introduced along with the SDEbased generalization to provide an understanding of the basic theory of DGMs.The concept of transforming noise distribution into data distribution through progressive denoising is then employed to support the assumptions required for Micro3Diff, which is introduced in the next section.
(2) In the following section, the detailed procedure for 2D-to-3D reconstruction using Micro3Diff is introduced, based on the novel multi-plane denoising diffusion-based approach.Furthermore, to mitigate potential discrepancies arising from multi-plane denoising, the technique of harmonized sampling is introduced to enhance the quality of generated samples.By combining these approaches, this section demonstrates how it becomes feasible to simultaneously transform the noise distribution in multiple planes into the data distribution while preserving connectivity and continuity in 3D space.
(3) To validate the proposed methodology, various types of microstructures including spherical inclusions, polycrystalline grains, battery electrodes, and carbonates, sourced from both synthetic and experimental data, are considered to be reconstructed using the proposed Micro3Diff.The generated results are subsequently validated by comparing spatial correlation functions to assess the similarity between the generated 3D samples and the original data.

Preliminaries
In this section, the formulations of DGMs are introduced, encompassing the two predominant approaches (SGMs and DDPMs), for the development of DGM-based 2D-to-3D reconstruction of microstructures (section 3).

Generalization of DGMs
Over the past few years, there have been several formulations and variations of DGMs [41][42][43][49][50][51][52][53][54] proposed, yet they all share two important common features: 1. the pre-defined progressive noising process for transforming data distribution into a prior noise distribution, and 2. the incorporation of a model that learns to denoise a sample to transform the noise distribution towards a desired conditional/unconditional data distribution.The formal is also called the forward diffusion process and the latter one is called the reverse diffusion process [45,54].Among the different formulations, the one that employs the SDEs can be regarded as a generalized form of DGMs.A forward diffusion process for perturbing (i.e., noising) data  can be defined in a form of Ito ̂ SDE [55] with a continuous time variable  ∈ [0, ] as d = (, ) + ()d where  is a standard Wiener process, which can also be thought of as Gaussian noise or Brownian motion [41,56], (•, ): ℝ  → ℝ  and (): ℝ → ℝ represent drift and diffusion coefficients of SDE at time step , respectively.To perturb data with the SDE, the drift coefficient can be defined to nullify the data while the diffusion coefficient is adopted to progressively add noise to the data.In other words, the diffusion process can be modeled with different choices of (•, ) and () while d represents the stochastic component.Considering this aspect, the two popular formulations of DGMs, which are SGMs [42] and DDPMs [43], can also be interpreted as discretized variations of the SDE with different drift and diffusion coefficients.

Forward diffusion process
As stated above, the diffusion processes in the various formulations of DGMs (e.g., SGMs and DDPMs) can be generalized using the concept of SDE.For instance, SGMs employ the concept of the score function (i.e., the gradient of the log probability density) and the denoising score matching with Langevin dynamics [44] for transforming random noise to data with high probability density.According to the perturbation process with a specified noise distribution proposed by Song et al. [42], the SDE for SGMs can be written as where  (   ) converges to   (i.e., the noise level at time  ) as  → ∞.Considering a discrete sequence of samples (i.e. ,  0 ,  1 , … ,   ), the update rule for obtaining the noised sample ( +1 ) from the sample at the previous time step (  ) can be defined as follows with the noise component   ~(, ).
On the other hand, DDPMs utilize the concept of Markov chain to define the forward/reverse diffusion process, along with the variational lower bound (VLB) for training the models to reconstruct an original data structure from the prior noise distribution.Similar yet different, the SDE for the forward diffusion process of DDPMs can be expressed as where  is the noise scheduling parameter and  ( ) converges to   as  → ∞ .
According to this definition and the Stirling's approximation, the following update rule can be derived.
The schematic of the forward diffusion process of DGMs using the introduced formulations is shown in Figure 1.It is again worth noting that the primary objective of defining the forward diffusion process in DGMs is to transform the data distribution into the prior noise distribution, even though the drift and diffusion coefficients may vary.

Reverse diffusion process
Based on the forward diffusion SDE (Eq.( 1)), the SDE for reverse diffusion of data can be derived as where  ̂ is a standard Wiener process when the time is reversed.In an intuitive sense, the SDE in reversed time (Eq.( 6)) implies that it is able to reconstruct data by starting with the noise distribution.It is worth noting that the only unknown in Eq. ( 6) is the value of score function, which is denoted as ∇  log   ().In other words, if we can obtain the value of ∇  log   (), it is able to generate as many samples as desired.In this regard, the following loss function of SGMs for denoising score matching [42] can be optimized: where () is a positive weighting function at time  and   (  , ) is a model trained to estimate the value of the score function within the discrete time interval.In addition, this can be considered equivalent to solving the reverse-time SDE.
On the other hand, the loss function of DDPMs originates from the theory of variational inference [43,57].This is the reason why a DDPM is also called a hierarchical Markovian VAE [45].Based on the Eq. ( 5), a process of forward diffusion can be defined in the form of conditional Gaussian distribution as follows.

𝑝(𝐱 𝑡 |𝐱 𝑡
Based on this forward process, a neural network model for the reverse process can be defined as where   and   are the predicted mean function and the covariance, respectively.To estimate the original data distribution ( 0 ) with the forward/reverse process through the time steps, the Kullback-Leibler divergence (   ) between the joint distributions   ( 0 ,  1 , … ,   ) and ( 0 ,  1 , … ,   ) need to be minimized.This is achieved by minimizing the VLE of the negative log-likelihood as: In particular,  −1 can be rewritten as the following expectation of ℓ 2 loss as where   (  , ) is the model trained to estimate the mean function and   (  ,  0 ) is the mean function of noised sample at  according to the pre-defined forward diffusion process (Eq.( 5)).In particular, Ho et al. [43] reparametrized Eq. ( 14) with the model   (  , ) for predicting the noise component at  as follows.

Denoising diffusion-based 2D-to-3D reconstruction with harmonized sampling 3.1 Equation of sampling
In the previous section, the equivalence of the different formulations of DGMs is introduced.Among the different formulations, the formulation of DDPMs is adopted in this study to build DGMs for the reconstruction of microstructures.Thus, Eq. ( 15) is utilized as the objective function for training the model   (  , ).After training the model, it is able to obtain the mean function of   in Eq. (9) with the prediction of  using the following equation: where It is worth noting that   is assumed to be constant, which can be simply computed using the pre-defined noise schedule (i.e.,   =   ), as learning only the mean function leads to better sample quality according to Ho et al. [43].In addition, the variance can be also learned by incorporating a parameterized diagonal Σ  into the VLE (Eq.( 10)) [58].

Reverse diffusion process for dimensionality expansion (2D-to-3D)
One of the most distinct characteristics of DGMs, compared to other generative models such as VAEs [14,15] and GANs [16,17], is the progressive and gradual diffusion process.As discussed in the section 2, DGMs incorporate the forward/reverse diffusion process to estimate the data distribution ( 0 ) and generate samples from the known prior distribution (  ).For instance, if a DGM for the generation of 2D images (i.e., 2D-DGM) is prepared, it can generate novel 2D images start from Gaussian noise through the diffusion times steps and the pre-defined noise schedule [46,51].In addition, this process is equivalent to the gradual transformation of ( 0 ) to (  ) in 2D pixel space.Drawing from this perspective, this study introduces a novel method for 2D-to-3D dimensionality expansion through the multiplane denoising diffusion process (Figure 2 and Figure 3).As shown in Figure 2, we can define three orthogonal planes (yz, xz, xy-planes) and use a trained 2D-DGM for generation of a sample at each plane.According to the original formulation of DDPMs, the generation process of each sample on each plane is independent of the others, resulting in the creation of three distinct samples that follow the data distribution ( 0 ).However, if each generation process (i.e., reverse diffusion process) at each plane proceeds concurrently and is conditioned on the others, we could potentially achieve samples with connectivity and continuity ensured at the junctions of the planes.Thus, the problem now becomes developing a new model for estimating the conditional distributions at each time step during the reverse diffusion process as: ( −1 ( * ) |  (1) ,   (2) ,   (3) ) (20) where  −1 ( * ) denotes the denoised sample at a certain plane, and   (1) ,   (2) and   (3) represent the samples before denoising at the three orthogonal planes.However, to train the model in Eq. (20), it is evident that a 3D dataset with samples at three orthogonal planes is required.In addition, the model requires additional channels or modules to impose conditional signals during the generation process, which necessitates modifying the model architecture, such as incorporating classifiers or using classifier-free guidance [46,59].Thus, this cannot be classified as 2D-to-3D reconstruction because both the dataset and the training process need to extend beyond the confines of 2D space To avoid the preparation of a 3D dataset and maintain the architecture of 2D-DGM, the proposed method for 2D-to-3D dimensionality expansion in this study solely focuses on the reverse diffusion process.Consequently, the training process for the 2D-DGM remains unchanged, following the procedures used to train generative models for conventional image generation tasks with 2D datasets.Therefore, throughout this paper, the only neural network model we deal with is   ( −1 |  ) trained on 2D microstructure images without any additional conditions incorporated.In this regard, the key idea of the proposed multi-plane denoising diffusion is to generate 3D voxels using 2D-DGM by changing the target denoising plane periodically as shown in Figure 3.The procedure of the multi-plane denoising diffusion is as follows: 1. Initialize voxels with dimensions of  ×  ×  using Gaussian noise (Figure 3a).It is worth noting that the size of  should match the size of input/output size of 2D-DGM (i.e.,  × ).
2. For the first reverse diffusion step, denoise the  samples with dimensions of  ×  (which are from the initialized voxels) using the trained 2D-DGM (i.e.,   ( −1 |  )) along a particular plane.
3. Before the second reverse diffusion step, change the target denoising plane and obtain  −1 * , which represents the samples arranged in a different plane (Figure 3c).Then, denoise the samples by taking  −1

Harmonized sampling for dimensionality expansion
To address the discrepancies caused by the proposed multi-plane denoising diffusion, this study introduces the method of harmonized sampling (or resampling [60]) for 2D-to-3D reconstruction of microstructures.Since   * is manipulated data from   , there is disharmony introduced to   adopted during the denoising process as depicted in Figure 5, which involves a cycle of renoising and denoising the sample at time .In other words, we give more chances to the model to harmonize the conditional information   * before proceeding to the next denoising step.The renoising process is the same as the original forward process (Eq.( 8)) which can be written as follows.

𝑝(𝐱 𝑡
In addition, the concept of renoising (or harmonizing) was initially introduced for 'inpainting' with masked inputs to obtain the most probable inpainting result conditioned on the unmasked region by Lugmayr et al. [60].Their work demonstrated that incorporating several harmonizing steps can result in more harmonized images, compared to sampling without harmonizing steps Taking inspiration from this idea, this study utilizes harmonizing steps to mitigate the disharmony introduced during the 2D-to-3D reconstruction with the multi-plane denoising diffusion (section 3.2).To be more specific, the harmonizing steps are applied  ℎ times at each time step along with   to enable sampling conditional to the samples at different planes.This approach allows us to obtain more realistic 3D data, ensuring that all the samples viewed from the three orthogonal planes follow the original data distribution and maintain connectivity.
The entire process of multi-plane denoising diffusion with harmonizing steps for 2D-to-3D reconstruction of microstructures is described in Algorithm 1. Algorithm 1: Algorithm for multi-plane denoising diffusion for 2D-to-3D reconstruction of microstructure with harmonized sampling (the upper-left superscript for denoting the order of samples (Figure 4) is neglected for simplicity) Rearrange batch-wise input for multi-plane denoising diffusion:  −1 * =   ( −1 ,  − 1) 13: end for 14: return  0

Implementation
The implementation of the 2D-DGMs for sampling 2D microstructural images and the incorporation of the multi-plane denoising diffusion (section 3.2) with harmonizing steps (section 3.3) were carried out using the Pytorch library [61].In order to build models for reverse diffusion process (i.e.,   ( −1 |  )), this study adopted the Imagen model, proposed by Saharia et al. [51] which proved to be highly effective in generating photorealistic images, to generate 64 × 64 sized 2D samples.The details of the model architecture with the hyperparameters are described in Appendix A.1.
The training of these models was conducted using Nvidia RTX A6000 graphics processing units (GPUs) coupled with the Adam optimizer, employing a learning rate of 3 × 10 −4 .For each exemplary case, which will be further elaborated in the subsequent sections, the batch size was set to 32 per GPU, and the training process consisted of 50,000 training steps.The diffusion time steps for all models were set to be  = 1000 with the linear noise schedule [43], where   s are evenly spaced values over the interval  1 = 10 −4 and   = 10 −2 .

Evaluation metrics for validation
To quantify the quality of the generated samples using the proposed methodology in this study, the following criterion metrics are considered: two-point correlation function ( 2 ) and lineal path function ( ) in 2D/3D space.The two-point correlation function, which characterizes the statistical distribution of the material phase, can be written as follows: where (•) is a binary function that becomes 1 if a material phase of interest is present at a given location and 0 otherwise.In a similar manner, the lineal path function to evaluate the connectivity between clusters of material phase can be computed as follows: = { 1 if a line connecting  1 and  2 is on material phase of interest 0 otherwise (23) Then, the error rate(%) between the correlation functions of training dataset and generated data can be computed using the following discrepancy equation [47,62]: where   represents the area between of two correlation functions and   represents the area under the correlation function of the training dataset.Depending on the sources of training data (e.g., 2D images sampled from 3D volume, or 2D images themselves), the correlation functions are evaluated in either 3D space or 2D space for each case study, which is further explained in the following sections

Case I: Synthetic microstructural samples of spherical inclusions
To validate the proposed multi-plane denoising diffusion in this study, 2D-to-3D reconstruction of microstructures with spherical inclusions embedded in a matrix was conducted.As shown in Figure 6, the 2D microstructure images used for training the 2D-DGM were sampled from a synthesized 3D data.The 3D sample of spherical inclusions was generated using Geodict® software by Math2Market GmbH.The diameters of the inclusions were set to follow a normal distribution with a mean value of 15 voxels and a standard deviation of 4 voxels in a volume of 64×64×300 voxels.Then, the 3D data was sliced along the z-axis, generating a total of 300 images with dimensions of 64 × 64.To facilitate the training of the 2D-DGM, the dataset was augmented by manipulating each image eight times (i.e., eight folds).This augmentation involves applying both horizontal and vertical flips, as well as rotations of 90°, 180°, and 270° to the original images.After training the 2D-DGM, the multi-plane denoising with the harmonizing steps was conducted to create a 3D volume with spherical inclusions.To investigate the effect of number of harmonizing steps  ℎ , 2D-to-3D reconstruction was conducted with varying  ℎ as illustrated in Figure 7.As depicted in the figure, the quality of the generated samples improves with an increase in the number of harmonizing steps.The results also show that the spherical shapes of the inclusions were well reproduced with  ℎ = 10.This implies that there is a minimal value of  ℎ to guarantee the quality of the generated sample.For the sake of simplicity in the validation process,  ℎ was set to be constant value of 10 for the example cases demonstrated in the following sections.Furthermore, the 3D visualization and sectional views in Figure 7 demonstrate that the inclusions (i.e., white pixels) along the three orthogonal planes are well connected, preserving continuity in 3D space.
To quantitatively evaluate the quality of the samples, the error rate (Eq.( 24)) between the correlation functions of the original data (Figure 6) and generated data was evaluated, as shown in Figure 8a.For validating purpose, a total of 25 samples were generated for each case study using Micro3Diff to calculate the averaged  2 and  (more examples of generated 3D samples can be found in Appendix A.2).In the case of spherical inclusions (case I), the error rates for  2 and   are 1.88% and 4.56% (Figure 9), respectively, indicating that the generated samples are in good agreement with the original data.In other words, the original and generated samples exhibit similar spatial correlations, indicating that they have statistically equivalent morphologies.Additionally, it is worth noting that since the source of training data (i.e., 2D images) for case I was sampled from the synthesized 3D volume, the 3D correlation functions were computed to evaluate the sample quality.
The results show that the multi-plane denoising diffusion with   ( −1 * |  * ), aided by the harmonized sampling, effectively guides the distribution of images at different planes simultaneously to ( 0 ), which is the distribution for images of spherical inclusions.It can also be said that the generation of 3D data with conditional distribution of images at different planes (Eq.( 20)) is addressed by the dimensionality expansion problem using the gradual multi-plane diffusion process and harmonized sampling.In the following sections, additional example cases to evaluate the performance of Micro3Diff are explored, including the cases with the microstructure of polycrystalline grains with grain boundaries (case II) and the realworld experimental micrographs (case III and IV)   For the cases where 2D samples were used for training the 2D-DGM (Case II and Case IV), the error rate is evaluated in 2D space.

Case II: Synthetic microstructures of polycrystalline grains
For further assessing the effectiveness of proposed Micro3Diff, this section focuses on the microstructural images of polycrystalline grains as the case study.To acquire 2D images of polycrystalline grains for the 2D-to-3D reconstruction task, a set of 300 synthetic images were generated using Laguerre tessellation [22] as shown in Figure 9.In each generated 2D image, the white pixels represent the grain boundaries.Three values of grain sizes were considered (6, 9 and 12 pixels) to be distributed, each occupying an equal fraction within a 64 × 64 image.The synthesized images were then augmented eightfold, similar to case I for training the 2D-DGM.
It is worth noting that the synthetic 2D samples of polycrystalline grains (Figure 10) considered in this case are genuinely independent samples, meaning that they were not sampled from a 3D data as in case I (section 4).Consequently, while the 2D-DGM lacks the ability to learn the 3D relationship for both cases I and II, the authors hypothesized that case II poses a more challenging problem since it lacks an intermediate image that can connect different slices.Thus, the Markov chain of   ( −1 * |  * ) may have a higher likelihood of deviating from the trajectories of the Markov chain of   ( −1 |  ). Figure 10 shows a generated 3D volume with polycrystalline grains using Micro3Diff, along with its sectional views at each orthogonal plane.To show the connectivity and the grain structures clearly, the 3D volumetric view includes the normal view, where only the grain boundaries (i.e., white voxels) are visualized, and its inverted view.At first glance, the grain boundaries appear to be well connected in 3D space, which is the important feature of polycrystalline materials.However, the 2D sectional views show that there are some disconnected and curved grain boundaries in the generated 3D volume.This error is likely due to the lack of intermediate images in the training dataset for connecting different slices in the 3D volume, which makes the model to generate sample with lower probabilities in ( 0 ).In addition, the occurrence of disconnected grain boundaries has been commonly reported even in the recent studies with the descriptor-based MCR method [17,19] and GAN-based reconstruction [34].Meanwhile, the error rates for  2 and  are 6.43% and 5.88%, respectively, indicating the good agreement between the original and generated data (Figure 8b and Figure 9).It is worth noting that, since the source of the training data for case II is 2D images (Figure 10), the 2D correlation functions were computed to evaluate the generated sample quality at the three orthogonal planes.Then, the mean values of the computed correlation functions were used to evaluate the error rate.To enhance the performance of 2Dto-3D reconstruction for microstructures of polycrystal materials such as alloys, the authors suggest considering preprocessing and augmentation of dataset to address the lack of intermediate images, as well as optimizing the hyperparameters (e.g.,  ℎ and ) Figure 11.2D-to-3D reconstruction results of microstructure with polycrystalline grains using Micro3Diff.

Case III: Experimental micro-CT scan images of battery electrodes
The next example case is the real micro-CT scan images of nickel manganese cobalt (NMC) cathode.The micro-CT data of NMC cathode was obtained from open-access collected data in reference [63], and a volume of 64×64×300 voxels was created for sampling 300 images with dimensions of 64 × 64 (Figure 12).Subsequently, the sampled images were augmented using eightfold augmentation as the previous cases.Figure 13 illustrates the generated 3D volume of NMC cathode and its 2D sectional views (for simplicity, only the active material phase and pore were considered).Similar to case I and II, the results show that the connectivity of the white voxels (i.e., active material phase) is preserved.Both the 3D volume and the 2D sectional views exhibit visual similarity to the original data in Figure 12.The error rates of 4.56% for  2 and 1.26% for  (Figure 8c and Figure 9) also demonstrate that the generated samples have close spatial correlations compared to the original micro-CT data.This is particularly encouraging as it demonstrates the effectiveness of the proposed Micro3Diff in 2D-to-3D reconstruction, utilizing both synthetic (case I and II) and experimentally observed data.Since obtaining a reliable dataset of microstructure is crucial for characterizing the material behavior, there has been significant attention given to the reconstruction of battery electrode microstructures in recent years [64][65][66].Although the performance of the proposed methodology must be validated with more diverse experimental data, the results highlight the potential application of Micro3Diff in the systematic exploration and design of materials for battery applications aided by ICME methods [3,4].

Case IV: Experimental micro-CT scan image of carbonates
Lastly, a more challenging scenario is considered where the availability of 3D microstructure data is severely restricted, and only obtainable data is a single representative 2D micrograph image.Suppose a single 2D sample of carbonates [67] is available, and we aim to reconstruct a 3D volume from this 2D sample, which should have an equivalent statistical distribution of material phases in 3D space.To train 2D-DGM first, multiple cropped images can be sampled from the representative image as shown in Figure 14.In particular, a total of 250 images were sampled and augmented using eightfold augmentation, as in the previous cases.
Figure 15 shows the generated 3D volume of carbonates with its sectional views.As can be seen, the generated 3D volume exhibits visually similar sectional views at the three orthogonal planes, demonstrating the capability of Micro3Diff to create a 3D volume from a single experimental representative image.Meanwhile, the maximum error rates for  2 and  are 9.21% and 6.31%, respectively (Figure 8d and Figure 9).The higher error rates compared to the previous cases are likely due to the lack of diversity in the data used to train the 2D-DGM, which could lead to incorrect estimation of the data distribution during the multiplane denoising process.In other words, the absence of intermediate data to ensure connectivity in 3D space could be a reason for the higher error rates (similar to case II).However, the results are remarkably encouraging, as the sectional views display morphological similarities with the training data, and the volumetric view illustrates Micro3Diff's ability to discover potential combinations of 2D images from the estimated ( 0 ) (derived from a single micrograph) to construct a 3D volume.Moreover, the proposed Micro3Diff enables the generation of multiple and diverse volumes with comparable visuals and spatial correlations to the original data, as shown in Appendix A.2.This capability could significantly assist in the quantitative assessment of material behavior in 3D space [13,[68][69][70], taking into account the microstructures and inherent randomness.

Conclusions
The results in this study demonstrate the capability of Micro3Diff for reconstructing 3D microstructure samples from 2D micrographs using the proposed multi-plane denoising diffusion and the harmonized sampling.Based on the multi-plane denoising, a 2D-DGM can be used to effectively guide the noised samples across multiple planes towards the data distribution while maintaining connectivity in 3D space.The disharmony caused by potential deviations from the intended trajectories in the trained 2D noise-to-data denoising diffusion process can be addressed by the proposed harmonized sampling method.Since this is the first study that proposes DGM-based 2D-to-3D microstructure reconstruction, the authors believe that the proposed Micro3Diff has the potential to chart novel avenues in the realm of datadriven material science.Meanwhile, a broader range of microstructure types, including multiphase and anisotropic microstructures, should be considered to improve the applicability of Micro3Diff in the future.In addition, the proposed Micro3Diff also introduces a novel approach of manipulating the latent variables (i.e., noised samples) in DGMs for dimensionality expansion.The authors suggest that this concept could substantially expand the range of applications for DGMs in various fields, including computational materials engineering and 3D generative modeling.

Figure 1 .
Figure 1.Schematic of the forward diffusion process (i.e., noising process) with the formulations based on SDE, SGM and DDPM.

Figure 2 .
Figure 2. Schematic of multi-plane denoising diffusion in the three orthogonal planes (i.e., yz, xz, xy-planes) for transforming the noise distribution into the data distribution while ensuring connectivity between the planes.The colored lines within the time interval represent the trajectories of reverse diffusion ((  ) → (  )).

Figure 3 .
Figure 3. Multi-plane denoising with the discrete times steps of the reverse diffusion: a) Initial voxels with Gaussian noise, b) Slice view of the voxels according to the reversed time steps, c) Corresponding planes and samples for denoising at each time step with the trained model   ( − |  ), and d) reconstructed 3D microstructure sample.

Figure 4 .
Figure 4. Comparison between a) conventional 2D denoising diffusion and b) multi-plane denoising diffusion in terms of the conditional probability and Markov chain based on the formulation of DDPM.The upper-left superscript denotes the th sample along a particular plane.

Figure 5 .
Figure 5. Multi-plane denoising diffusion with harmonizing steps to address the disharmony introduced by sampling in different planes.

Figure 6 .
Figure 6.A synthesized 3D Microstructure sample with spherical inclusions and sampled 2D microstructural images for training the 2D-DGM.

Figure 7 .
Figure 7. 2D-to-3D reconstruction results of microstructure with spherical inclusions using Micro3Diff with different numbers of harmonizing steps.

Figure 8 .
Figure 8. Two-point correlation functions ( 2 ) and lineal path functions () for different types of microstructures: a) case I: spherical inclusions, b) case II: polycrystalline grains, c) case III: NMC battery cathode, and d): case IV: carbonates.

Figure 9 .
Figure 9. Error rate of two-point correlation function (  ) and lineal path function (  ) for different cases.For the cases where 2D samples from 3D data (i.e., 3D micro-CT data) were used for training the 2D-DGM (Case I and Case III), the error rate is evaluated in 3D space.For the cases where 2D samples were used for training the 2D-DGM (Case II and Case IV), the error rate is evaluated in 2D space.

Figure 10 .
Figure 10.Synthesized 2D microstructure samples of polycrystalline grains for training the 2D-DGM.Figure10shows a generated 3D volume with polycrystalline grains using Micro3Diff, along with its sectional views at each orthogonal plane.To show the connectivity and the grain structures clearly, the 3D volumetric view includes the normal view, where only the grain boundaries (i.e., white voxels) are visualized, and its inverted view.At first glance, the grain boundaries appear to be well connected in 3D space, which is the important feature of polycrystalline materials.However, the 2D sectional views show that there are some disconnected and curved grain boundaries in the generated 3D volume.This error is likely due to the lack of intermediate images in the training dataset for connecting different slices in the 3D volume, which makes the model to generate sample with lower probabilities in ( 0 ).In addition, the occurrence of disconnected grain boundaries has been commonly reported even in the recent studies with the descriptor-based MCR method[17,19] and GAN-based reconstruction[34].Meanwhile, the error rates for  2 and  are 6.43% and 5.88%, respectively, indicating the good agreement between the original and generated data (Figure8band Figure9).It is worth noting that, since the source of the training data for case II is 2D images (Figure10), the 2D correlation functions were computed to evaluate the generated sample quality at the three orthogonal planes.Then, the mean values of the computed correlation functions were used to evaluate the error rate.To enhance the performance of 2Dto-3D reconstruction for microstructures of polycrystal materials such as alloys, the authors suggest considering preprocessing and augmentation of dataset to address the lack of intermediate images, as well as optimizing the hyperparameters (e.g.,  ℎ and )

Figure 12 .
Figure 12.An experimentally obtained 3D microstructure sample of NMC battery cathode and sampled 2D microstructural images for training the 2D-DGM.

Figure 13 .
Figure 13.2D-to-3D reconstruction results for microstructure of NMC battery cathode using Micro3Diff.

Figure 14 .
Figure 14.An experimentally obtained single 2D microstructure image of carbonates and random sampling of 2D images for training the 2D-DGM.

Figure A 1 .
Figure A 1. Exemplary 2D-to-3D reconstruction results of different microstructures using Micro3Diff: a) spherical inclusions, b) polycrystalline grains, c) polycrystalline grains (inverted), d) NMC battery electrode, and d) carbonate Create a class for re-noising a sample from  −1 to   with the pre-defeind noising schedule i.e., (  | −1 ) 2: Initialize   with a dimension of  ×  ×  from a standard Gaussian distribution Require: the trained 2D diffusion model   (  , ) for generating  ×  sized samples, the number of forward/reverse diffusion steps (  ), the number of harmonizing step ( ℎ ) (i.e., resampling step), the dimension of the volume to be generated ( ×  ×  voxels), pre-defined noising schedule with   , 1: