Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling

The crystal diffusion variational autoencoder (CDVAE) is a machine learning model that leverages score matching to generate realistic crystal structures that preserve crystal symmetry. In this study, we leverage novel diffusion probabilistic (DP) models to denoise atomic coordinates rather than adopting the standard score matching approach in CDVAE. Our proposed DP-CDVAE model can reconstruct and generate crystal structures whose qualities are statistically comparable to those of the original CDVAE. Furthermore, notably, when comparing the carbon structures generated by the DP-CDVAE model with relaxed structures obtained from density functional theory calculations, we find that the DP-CDVAE generated structures are remarkably closer to their respective ground states. The energy differences between these structures and the true ground states are, on average, 68.1 meV/atom lower than those generated by the original CDVAE. This significant improvement in the energy accuracy highlights the effectiveness of the DP-CDVAE model in generating crystal structures that better represent their ground-state configurations.


I. INTRODUCTION
Advances in computational materials science has enabled the accurate prediction of novel materials possessing exceptional properties.Remarkably, these computational advancements have facilitated the successful experimental synthesis of materials that exhibit the anticipated properties.Some predicted materials, such as nearroom-temperature superconductors, have been successfully synthesized under high-pressure conditions, with their superconducting temperatures in accordance with density functional theory (DFT) calculations [1,2].To achieve accurate predictions, a priori knowledge of plausible molecular and crystal structures play a vital role in both theoretical and experimental studies.Several algorithms, such as evolutionary algorithms, swarm particle optimization, random sampling method, and etc., have been employed for structure prediction [3][4][5].These algorithms rely on identifying local minima on the potential energy landscape obtained from DFT calculations [6,7].In the case of crystal structures, where atoms are arranged in three-dimensional space with periodic boundaries, additional criteria are necessary to enforce crystal symmetry constraints [5].* Correspondence to: thiti.b@chula.ac.thRecent approach for structure prediction employs denoising diffusion models to perform probabilistic inference.These models sample molecular and crystal structures from a probability distribution of atomic coordinates and types [8][9][10][11], bypassing the computationally intensive DFT calculation to tediously determine the potential energy landscape.By leveraging sufficiently large datasets containing various compounds, this method enables the generation of diverse compositions and combinations of elements simultaneously.Furthermore, the models allow for the control of desired physical properties of the generated structures through conditional probability sampling [12][13][14].These machine learning-based algorithms also hold promise for solving inverse problem efficiently, resolving structures from experimental characterizations, e.g., x-ray absorption spectroscopy and other techniques, a challenging problem in materials science [15][16][17].
There are two primary types of denoising diffusion models: score matching approach and denoising diffusion probabilistic models (DDPM) [18][19][20].These two models can denoise (reverse) a normal distribution such that the distribution gradually transforms into the data distribution of interest.The score matching approach estimates the score function of the perturbed data directing the normal distribution toward the data distribution and employing large step sizes of variance.In contrast, DDPM gradually denoises the random noise through a joint distribution of data perturbed at different scales of variance.
Since atomic positions in crystal structures are periodic and can be invariant under some rotation groups depending on their crystal symmetry, the core neural networks should favourably possess roto-translational equivariance [21][22][23].Xie et al. [8] has proposed a model for crystal prediction by a combination between variational autoencoder (VAE) [24] and the denoising diffusion model, called crystal diffusion VAE (CDVAE).The model employs the score matching approach with (annealed) Langevin dynamics to generate new crystal structures from random coordinates [18].The neural networks for an encoder and the diffusion model are rototranslationally equivariant.As a result, CDVAE can generate crystal structures with realistic bond lengths and respect crystal symmetry.
Because of the periodic boundary condition imposed on the unit cell, gradually injecting sufficiently strong noises (in the forward process) to the fractional coordinates can lead to the uniform distribution of atomic positions at late times, the consequence of ergodicity in statistical mechanical sense.Rather than beginning with a Gaussian distribution and denoising it as in the original CDVAE formulation, Jiao et al. [25] perturbed and sampled atomic positions beginning with a wrapped normal distribution which satisfies the periodic boundary condition.With this approach, the reconstruction performance has been significantly improved.Other circular (periodic) distributions, e.g., the wrapped normal and von Mises distributions, are not natural for DDPM framework since there is no known analytical method to explicitly incorporate such distributions into the framework.There, one needs to resort to an additional sampling procedure to construct the DDPM [26].
In this work, we introduce a crystal generation framework called diffusion probabilistic CDVAE (DP-CDVAE).Similar to the original CDVAE, our model consists of two parts: the VAE part and the diffusion part.The purpose of the VAE part is to predict the lattice parameters and the number of atoms in the unit cell of crystal structures.On the other hand, the diffusion part utilizes the diffusion probabilistic approach to denoise fractional coordinates and predict atomic coordinates.By employing the DDPM instead of the score matching approach, the DP-CDVAE model shows reconstruction and generation task performances that are statistically comparable to those obtained from original CDVAE.Importantly, we demonstrate the significantly higher groundstate generation performance of DP-CDVAE, through the distance comparison between generated structures and those optimized using the DFT method.We also analyze the changes in energy and volume after relaxation to gain further insights into models' capabilities.

II. RESULTS
The performances of DP-CDVAE models are herein presented.There are four DP-CDVAE models, differed by the choice of the encoder (see Fig. S1).DimeNet ++ has been employed for the main encoder for every DP-CDVAE models [27].We then modify the encoder of DP-CDVAE to encode the crystal structure by two additional neural networks: a multilayer perceptron that takes the number of atoms in the unit cell (N a ) as an input, and a graph isomorphism network (GINE) [28].Their latent features are combined with the latent features from DimeNet ++ through another multilayer perceptron.The N a is encoded such that the model can decode the N a accurately, and GINE encoder is inspired by GeoDiff [10] whose model is a combination of SchNet [29] and GINE which yields better performance.
Three datasets, Perov-5 [30,31], Carbon-24 [32], and MP-20 [33], were selected to evaluate the performance of the model.The Perov-5 dataset consists of perovskite materials with cubic structures, but with variations in the combinations of elements within the structures.The Carbon-24 dataset comprises carbon materials, where the data consists of carbon element with various crystal systems obtained from ab initio random structure searching algorithm at pressure of 10 GPa [32].The MP-20 dataset encompasses a wide range of compounds and structure types.

A. Reconstruction performance
The reconstruction performance is determined by the similarity between reconstructed and ground-truth structures.The similarity can be evaluated using Niggli's algorithm implemented in StructureMatcher method from pymatgen library [34].The reconstructed and ground-truth structures are similar if they pass the criteria of StructureMatcher which are stol=0.5,angle tol=10, ltol=0.3.Match rate is the percentage of those structures passed the criteria.If the reconstructed and ground-truth structures are similar under the criteria, the root-mean-square distance between their atomic positions is computed and then normalized by 3 V /N a , where V is the unit-cell volume, and N a is the number of atoms in the unit cell.An average of the distances of every pair of structures (⟨δ rms ⟩), computed from Niggli's algorithm, is used as the performance metric.
Table I presents the reconstruction performance of different models for three different datasets: Perov-5, Carbon-24, and MP-20.For the Perov-5 dataset, the DP-CDVAE model achieves a match rate of 90.04%, indicating its ability to reconstruct a significant portion of the ground-truth structures.This performance is slightly lower than the CDVAE model but still demonstrates the effectiveness of our model.In terms of ⟨δ rms ⟩, the DP-CDVAE model achieves a value of 0.0212, comparable Here, Az enables the sampling of atomic types (Zt), and all the decoded features enable the reconstruction of crystal structures.
The input fractional coordinates r f undergo perturbation (dash-dotted line) at time step t and then are transformed by π(•) to satisfy the periodic boundary condition (dotted line), serving as the coordinates for the reconstructed crystal structures.These reconstructed structures, (Lzr f t , Zt, z, t), are subsequently fed into the diffusion network (D θ (Lzr f t , ft)), where ft is a node feature composing of Zt, z, and t.The diffusion network predicts the noise added to the fractional coordinates (ϵ θ ) as well as the one-hot vector of atomic types (A θ ), see Sec.IV C. Dashed-line boxes represent the unit cells of the crystal structures.
to the FTCP model [35], but slightly higher than the CDVAE model.Similarly, for the Carbon-24 and MP-20 datasets, the DP-CDVAE model performs well in terms of both match rate and ⟨δ rms ⟩.It achieves match rates of 45.57% and 32.42% for Carbon-24 and MP-20, respectively.The corresponding ⟨δ rms ⟩ values for Carbon-24 and MP-20 are 0.1513 and 0.0383, respectively, comparable to the CDVAE model.Regarding the DP-CDVAE+N a model, the additional encoding of N a into the model leads to improved match rates for all datasets, with an increase of 2-5%.This enhancement can be attributed to the accurate prediction of N a .However, in terms of ⟨δ rms ⟩, only the Perov-5 dataset shows an improvement, with a value of 0.0149.On the other hand, for the Carbon-24 and MP-20 datasets, the ⟨δ rms ⟩ values are higher compared to the DP-CDVAE model.
For the DP-CDVAE+GINE and DP-CDVAE+N a +GINE models, the additional encoding of GINE into the models leads to a substantial drop in match rates compared to the DP-CDVAE model, particularly for the Perov-5 and Carbon-24 datasets.In contrast, there is a moderate increase in the match rates for the MP-20 dataset.The ⟨δ rms ⟩ values for the Perov-5 and Carbon-24 datasets are comparable to those of the DP-CDVAE model.However, for the MP-20 dataset, the ⟨δ rms ⟩ is noticeably higher in the models with GINE encoder compared to the DP-CDVAE model.
Overall, while the reconstruction performance of the DP-CDVAE model may be slightly lower than the CD-VAE model in terms of match rate, it still demonstrates competitive performance with relatively low ⟨δ rms ⟩.The match rate can be enhanced by additionally encoding the N a , but the performance is traded off by the increase in ⟨δ rms ⟩.

B. Generation performance
We follow the CDVAE model that used three metrics to determine the generation performance of the models [8].The first metric is Validity percentage.The structures that are valid for structure and composition tests must satisfy two criteria: distances of every pair of atoms are larger than 0.5 Å and the total charge in the unit cell is neutral.The second metric is called coverage (COV), which utilizes structure and composition fingerprints to evaluate the similarity between the generated and ground-truth structures.COV-R (Recall) represents the percentage of ground-truth structures covered by the generated structures.COV-P (Precision) represents the percentage of generated structures that are similar to the ground-truth structures, indicating the quality of the generation.The third metric is the Wasserstein distance between property distributions of generated and groundtruth structures.Three property statistics are density (ρ), which is total atomic mass per volume (unit g/cm 3 ), formation energy (E f orm , unit eV/atom), and the number of elements in the unit cell (# elem.).A separated and pre-trained neural network is employed to predict E of the structures where the detail of the pre-training can be found in Ref. [8].The first and second metrics are computed over 10,240 generated structures, and 1000 structures are randomly chosen from the generated structures that pass the validity tests to compute the third metric.The ground-truth structures used to evaluate the generation performance are from the test set.
In Table II, the DP-CDVAE model achieves a validity rate of 100% for the Perov-5 dataset and close to 100% for the Carbon-24 and MP-20 datasets in terms of structure.The validity rate for composition is comparable to that of the CDVAE model.The DP-CDVAE model also demonstrates comparable COV-R values to the CDVAE model across all three datasets.Furthermore, the DP-CDVAE models with N a and/or GINE encoders exhibit similar Validity and COV-R metrics to those of the DP-CDVAE model.However, for COV-P, all DP-CDVAE models yield lower values compared to CDVAE.
On the other hand, our models show significant improvements in property statistics.In the case of the MP-20 dataset, the DP-CDVAE models, particularly those with the GINE encoder, yield substantially smaller Wasserstein distances for ρ, E f orm , and the number of elements compared to other models.For the Carbon-24 dataset, our models also exhibit a smaller Wasserstein distance for ρ compared to the CDVAE model.

C. Ground-state performance
Another objective of the structure generator is to generate novel structures that also are close to the ground state.To verify that, the generated structures are relaxed using the DFT calculation where the relaxed structures exhibit balanced internal stresses with external pressures and reside in local energy minima.These relaxed structures are then compared with the generated structures to evaluate their similarity.In this study, we have chosen a set of 100 generated structures from each of CD-VAE, CDVAE+Fourier, and DP-CDVAE models for re-laxation where CDVAE+Fourier model is CDVAE model with Fourier embedding features of the perturbed coordinates.However, relaxation procedures for multi-element compounds can be computationally intensive.To address this, we have specifically selected materials composed solely of carbon atoms, using the model trained on Carbon-24 dataset.This selection ensures a convergence of the self-consistent field in DFT calculation.Moreover, in the relaxation, we consider the ground state of the relaxed structures at a temperature of 0 K and a pressure of 10 GPa since the carbon structures in the training set are stable at 10 GPa [32].
We here introduce a ground-state performance presented in Table III.The StructureMatcher with the same criteria as in the reconstruction performance is used to evaluate the similarity between the generated and relaxed structures.The relaxed structure was used as a based structure to determine if the generated structure can be matched.Four metrics used to determine the similarity are 1) match rate, 2) ⟨δ rms ⟩, 3) ∆V rms and 4) ∆E rms .The ∆V rms and ∆E rms represent the root mean square differences in volume and energy, respectively, between the generated structures and the relaxed structures in the dataset.
In Table III, the DP-CDVAE model achieves the highest match rate and the lowest ⟨δ rms ⟩ and ∆E rms .Although the CDVAE+Fourier model achieves the lowest ∆V rms , the DP-CDVAE model demonstrates the ∆V rms that is comparable to that of the CDVAE+Fourier model.

III. DISCUSSION
The DP-CDVAE models significantly enhance the generation performance, particularly in terms of property statistics, while maintaining comparable COVs to those of CDVAE.Specifically, for Carbon-24 and MP-20 datasets, the density distributions between the generated and ground-truth structures from DP-CDVAE models exhibit substantially smaller Wasserstein distance compared those of the CDVAE model (see Table II).The ∆V rms of the DP-CDVAE model presented in Table III is significantly lower than that of the original CDVAE.This is corresponding to smaller Wasserstein distance of ρ shown in Table II.The DP-CDVAE model also demonstrates significantly smaller ⟨δ rms ⟩ than the original CD- VAE.These suggest that our lattice generation closely approximates the relaxed lattice, while also achieving atomic positions that closely resemble the ground-state configuration.Additionally, the distribution of the number of elements in the unit cells is relatively similar to that of the data in the test set, particularly in the results from the models with GINE encoder.This could be attributed to the capability of GINE to search for graph isomorphism [36].
Moreover, ∆E is the energy difference between the generated structures and their corresponding relaxed structures.The ground-state energy represents a local minimum that the generated structure is relaxed towards.A value of ∆E close to zero indicates that the generated structure is in close proximity to the ground state.In Table III, it can be observed that our model achieves the ∆E rms value of 400.7 meV/atom which is about 68.1 meV/atom lower than the ∆E rms of CDVAE.The mode of ∆E of our model is 64 -128 meV/atom, which is lower than its root-mean-square value (see Fig. S2).Nevertheless, both the ∆E rms and the mode of ∆E exhibit relatively high values.In many cases, the formation energy of synthesized compounds is reported to be above the convex hull less than 36 meV/atom [37][38][39].To obviate the need for time-consuming DFT relaxation, it is essen-tial for the generated structures to be even closer to the ground state.Therefore, achieving lower ∆E rms values remains a milestone for future work.

A. Diffusion probabilistic model
In the diffusion probabilistic model, the data distribution is gradually perturbed by noise in the forward process until it becomes a normal distribution at late times.In this study, the distribution of the fractional coordinate (r f ) is considered since their values of every crystal structures distribute over the same range ,i.e., r f ∈ [0, 1) 3 .The Markov process is assumed for the forward diffusion such that the joint distribution is a product of the conditional distributions conditioned on the knowledge of the fractional coordinate at the previous time step: where r 0 ∼ q(r f ) which is the data distribution of the fractional coordinate, t is the discretized diffusion time step, T is the final diffusion time, α t is a noise schedule with a sigmoid scheduler [40], and the conditional q(•|•) is a Gaussian kernel due to the Markov diffusion process assumption.Then r t can be expressed in the Langevin's form through the reparameterization trick as where ϵ ∼ N (0, I), and ᾱt = t i=1 α i .This update rule does not necessitate r t to remain in [0, 1) 3 ; however, we can impose the periodic boundary condition for the fractional coordinate so that The reverse conditional distribution can be trained by minimizing the Kullback-Leibler divergence between p θ (r t−1 |r t ) and q(r t−1 |r t , r 0 ), the posterior of the corresponding forward process [20].We use GemNetT for the diffusion network to train the parametrized noise ϵ θ [41].Then, the coordinate in the earlier time can be sampled from r t−1 ∼ p θ (r t−1 |r t ), whose corresponding reverse Langevin's dynamics reads where ϵ ′ ∼ N (0, I).Crucially, we empirically found that the final reconstruction performance is considerably improved when we impose the periodic boundary condition on the fractional coordinate at every time step such that r t−1 ∼ p θ (r t−1 |r ft ) and α t in the first term of Eq. ( 6) is replaced by ᾱt .Namely, in our modified reverse process, the coordinate is sampled from An illustration of denoising atomic coordinates with Eq. ( 7) is demonstrated in Fig. 2. The model performance using Eq. ( 6) is shown in Table S1, whereas the performance using Eq. ( 7) is shown in Table I.

B. Graph neural networks
Graph neural networks architecture facilitate machine learning of crystal graphs G = (V, E), graph representations of crystal structures.V and E are sets of nodes and edges, respectively, defined as where n and m are indices of atoms in a crystal structure, f n is a vector of M features of an atom in the unit cell, T is a translation vector, and L is a lattice matrix that converts a fractional coordinate r fn into its atomic Cartesian coordinate r cn .The atomic features, fractional coordinates, and atomic Cartesian coordinates of the crystal structure are vectorized (concatenated) as f = (f 1 , . . ., f Na ) ∈ R Na×M , r f = (r f1 , . . ., r f Na ) ∈ R Na×3 , and r c = (r c1 , . . ., r c Na ) ∈ R Na×3 .Three graph neural networks implemented in this work are DimeNet ++ [27], GINE [28], and GemNetT [41].DimeNet ++ and GINE are employed for encoders, and GemNetT is used for a diffusion network.DimeNet ++ and GemNetT, whose based architecture concerns geometry of the graphs, are rotationally equivariant.GemNetT has been devised by incorporating the polar angles between four atoms into DimeNet ++ .This development grants Gem-NetT a higher degree of expressive power compared to DimeNet ++ [42].Furthermore, GINE has been developed to distinguish a graph isomorphism, but not graph geometry nor the distance between nodes, which is important for our study.Thus we supplement the edge attributes into GINE with the distances between atoms, i.e.E = {||∆r (T) cmn ||}.

C. DP-CDVAE's architecture
The forward process of DP-CDVAE model is illustrated in Fig. 1.The model is a combination of two generative models, which are VAE and diffusion probabilistic model.The pristine crystal structures consist of the fractional coordinate (r f ), the lattice matrix (L), ground-truth atomic type (Z), and the number of atoms in a unit cell (N a ).For crystal graphs of the encoders, their node features are f = Z.The number of atoms in a unit cell N a is encoded through multilayer perceptron before concatenated with the latent features from other graph encoders.They are encoded to train µ ϕ and logvar ϕ where ϕ is a learnable parameter of the encoders.The latent variables (z) can be obtained by where ϵ ′′ ∼ N (0, I).Then, z will be decoded to compute the lattice lengths and angles, which then yield the lattice matrix (L z ), N a , and A z .In the original CDVAE, A z is the probability vector indicating the fraction of each atomic type in the compound and is used to perturb Z by where M is a multinomial distribution, A is a one-hot vector of ground-truth atomic type Z, and σ ′ t is the variance for perturbing atomic types at time t, which is distinct from σ t used for perturbing the atomic coordinates.Similar to the original CDVAE, σ ′ t is selected from the range of [0.01, 5].
For the diffusion network, the input structures are constructed from r ft , Z t , and L z where the (Cartesian) atomic coordinates at time t are computed by r ct = L z r ft .These are then utilized by the crystal graphs for the diffusion network, whose node features are f t = (Z t , F t , z, t) where F t is a Fourier embedding feature of r t (see section S3).As proposed by Ho et al. [20], we use the simple loss to train the model such that Since the diffusion model is trained to predict both ϵ and A, so the loss of the diffusion network is where L CE is the cross entropy loss, λ is a loss scaling factor, and t ∈ {1, ..., T } where T = 1000.In this work, t is randomly chosen for each crystal graph and randomly reinitialized for each epoch in the training process.The total loss in the trainig process is shown in Eq.S1.
In the reverse diffusion process, we measure the model performance of two tasks: reconstruction and generation tasks.For the former task, z is obtained from Eq. ( 8) by using the ground-truth structure as an input of the encoders.For the latter task, z ∼ N (0, I), which is then used to predict N a , L z , A z , and concatenate with the node feature of the crystal graph in the diffusion network.At the initial step, t = T , Z T is sampled from the highest probability of A z , and the final-time coordinate is obtained from sampling a Gaussian distribution, i.e. r T ∼ N (0, I).The coordinates can be denoised using Eq. ( 7), and the predicted atomic types are updated in each reversed time step by argmax(A θ ).

D. DFT calculations
The Vienna ab initio Simulation Package (VASP) was employed for structural relaxations and energy calculations based on DFT [43,44].The calculations were conducted under the generalized gradient approximation (GGA) and the project augmented wave (PAW) method [45,46].The thresholds for energy and force convergence were set to 10 −5 eV and 10 −5 eV/ Å, respectively.The plane-wave energy cutoff was set to 800 eV, and the Brillouin zone integration was carried out on a k-point mesh of 5 × 5 × 5 created by the Monkhorst-Pack method [47,48].

S3. FOURIER FEATURES
We appended the Fourier features to the node attributes for the input of the diffusion network.The Fourier features are the concatenation of sin(2 n πr t ) and cos(2 n πr t ) where n ∈ {n min , ..., n max }.For every DP-CDVAE model in the main text, n min = 3 and n max = 8.

S4. LOSS FUNCTIONS
Similar to the original CDVAE, the total loss function to train the model consists of 5 sub-loss functions: the Kullback-Leibler divergence loss (L KLD ) for training µ ϕ and logvar ϕ , the lattice loss (L latt ) for training lattice parameters, the composition loss (L comp ) for training A z , the loss for training the number of atoms (L Na ), and the loss from the diffusion network (L dif f ) for training ϵ θ and A θ .In particular, the total loss is where λ 1 , λ 2 , λ 3 , and λ 4 are tunable loss scaling factors, L latt is computed from the mean square error of lattice parameters, and L comp and L Na are cross-entropy losses.

FIG. 2 .
FIG. 2. The schematic depicting the reverse diffusion process of the DP-CDVAE model.Initially, atomic coordinates are sampled from a normal distribution and subsequently mapped into the unit cell (dashed-line box) using the periodic boundaryimposing function π(•).White circles outside the unit cell depict the atomic coordinates prior to the periodic boundary condition is imposed, while colored circles represent atoms that are inside the unit cell of interest.The action of π(•) on the atoms outside the unit cell is represented by an arrow that translates the white circles into colored circles in the unit cell.Left to right show the reverse direction of the arrow of time, depicting the reverse diffusion process.

ACKNOWLEDGMENTS
This research project is supported by the Second Century Fund (C2F), Chulalongkorn University.This Research is funded by Thailand Science research and Innovation Fund Chulalongkorn University (IND66230002) and National Research Council of Thailand (NRCT): (NRCT5-RSA63001-04).T.C. acknowledges funding support from the NSRF via the Program Management Unit for Human Resources and Institutional Development, Research and Innovation [grant number B05F650024].The authors acknowledge high performance computing resources including NVIDIA A100 GPUs from Chula Intelligent and Complex Systems Lab, Faculty of Science, and from the Center for AI in Medicine (CU-AIM), Faculty of Medicine, Chulalongkorn FIG. S2.Histograms of energy difference between generated and relaxed structures of (a) CDVAE, (b) CDVAE+Fourier, and (c) DP-CDVAE models.The modes (highest counting number) of CDVAE, CDVAE+Fourier, and DP-CDVAE models are 19.2 -32.0, 68.0 -128, and 64.0 -128 eV/atom, respectively, where the bin size is 64.0 meV/atom.
, ,          , FIG.1.The schematic summarizing the architecture for training the DP-CDVAE model.Multiple sub-networks are trained to minimize the total loss function of Eq.(S1).The encoder (G ϕ (Lr f , Z, Na)) compresses input pristine crystal structures into the latent feature (z).The predicted lattice parameters (Lz), the predicted number of atoms (Nz), and Az are decoded from z.

TABLE I .
Reconstruction performance.

TABLE II .
Generation performance