Abstract
Despite the enormous success and popularity of densityfunctional theory, systematic verification and validation studies are still limited in number and scope. Here, we propose a protocol to test publicly available pseudopotential libraries, based on several independent criteria including verification against allelectron equations of state and planewave convergence tests for phonon frequencies, band structure, cohesive energy and pressure. Adopting these criteria we obtain curated pseudopotential libraries (named SSSP or standard solidstate pseudopotential libraries), that we target for highthroughput materials screening (“SSSP efficiency”) and highprecision materials modelling (“SSSP precision”). This latter scores highest among opensource pseudopotential libraries available in the Δfactor test of equations of states of elemental solids.
Introduction
In the last three decades, atomistic electronicstructure methods have radically changed the way we think at materials theory and simulations. The 1998 Nobel prize in Chemistry given for densityfunctional theory (DFT) acknowledges this shift of paradigm. The ability to reduce the complexity of the manybody quantummechanical problem involving interacting electrons and nuclei into more tractable forms and algorithms allowed to leverage inexpensive and exponentially growing computational power, in order to provide sufficiently precise predictions for a great number of materials. What followed has been a flourishing of methods to compute more and more complex materials properties, and most notably spectroscopies (such as Raman, IR, ARPES, EELS, NMR and more). Nowadays, the (over)confidence in firstprinciples methods is such that they are routinely used to help interpreting experiments and guide the discovery and understanding of novel materials. In particular, systematic DFTbased computational materials screening is a fastgrowing field of research, as reflected by the creation, in the last few years, of several research centres worldwide on computational materials discovery. Up to now, the most visible output has been the creation of large databases of materials properties obtained from first principles, to be compared with or to augment experimental databases such as the Pauling file (http://paulingfile.com/), COD (http://www.crystallography.net/cod/) or ICSD (http://www2.fizkarlsruhe.de/icsd_home.html/). Even from a purely financial perspective, the personnel cost for planewave studies is of the order of 1 billion US$ per year, in purchasing power parity (PPP) terms, matched by substantial hardware usage (see Supplementary Note 1). It is ever so more remarkable that in spite all of this, the efforts of verification of the precision of the underlying pseudopotentials (PSPs) or projector augmentedwave (PAW) approximations have been minimal. Only in 2016 a multigroup effort was able to establish a baseline in the calculations of the equations of state of elemental crystals.^{1} In addition to the issue of precision, or of verification (i.e. insuring that the removal of the core electrons from the calculations performs with the required tolerance), the issue of performance looms large—a softer, smoother PSP will allow straightforwardly faster calculations, both because the basis set is decreased as the computational cost scales with the square of the basis size, and because the minimisation or iterative approaches can become more efficient or better preconditioned with a smaller basis.
Here, we introduce a PSP testing protocol based on extensive DFT and densityfunctional perturbation theory (DFPT) calculations of elemental solids, and select the optimal PSP for 85 elements of the periodic table. Our protocol, named standard solidstate pseudopotential (SSSP) testing protocol, is made of a verification part, based on the Δfactor (i.e. the difference between allelectron and PSP equations of state),^{1,2} and an extensive performanceoriented part based on planewave convergence tests for phonon frequencies, band structures, cohesive energies and stress tensors. We underline here that the SSSP testing protocol is a protocol based on verification and not on validation (following the nomenclature of ref. ^{1}). Indeed our primary goal is to test the precision (verification) of the PSPs and thus to furnish, through our testing protocol, PSP libraries that give results as close as possible to the “exact” theoretical results for the PBE functional, as they would be obtained through a perfectly converged allelectron calculation. We do not perform any comparison with respect to experimental results, i.e. we do not test the accuracy (validation) of the PSPs.
We consider up to eight (depending on the element) publicly available PSP libraries for the PBE functional^{3} and test them with the PWscf and Phonon codes of the Quantum ESPRESSO (QE) distribution^{4} in an automated fashion within the framework of the AiiDA^{5} infrastructure for reproducible computational science. AiiDA also allows straightforward dissemination of results through the Materials Cloud web platform (https://www.materialscloud.org/), a cloud service designed to enable data sharing in computational materials science.
In this paper, first we describe the test set of physically relevant quantities defined for our SSSP testing protocol and what are the corresponding selection criteria used in order to select the best PSPs. Then, we discuss the need for testing several properties, showing how different PSPs may yield similar results and convergence behaviours for one property but different ones for another property. Finally, we propose two optimal PSP libraries chosen according to the SSSP testing protocol and criteria.
Results
SSSP testing protocol
In this work, we investigate the precision and performance of several PSPs libraries available for the QE distribution.^{4} QE is an integrated suite of opensource codes for electronicstructure calculations based on DFT which uses planewaves as basis set and PSPs to represent the electrostatic electron–ion interactions. Nowadays QE is one of the most popular DFT codes adopted by researchers for the firstprinciples study of materials properties, with more than 2000 citations last year, according to Google Scholar™.
All the tested PSP libraries are based on the generalised gradient approximation (GGA) for the exchangecorrelation functional of Perdew, Burke and Ernzerhof (PBE)^{3} and they include the three main pseudization approaches: normconserving (NC),^{6} ultrasoft (US)^{7} and projectoraugmented wave (PAW)^{8} (in the following, we refer both to PSP and PAW approaches as PSP methods). In particular, we investigate three PAW PSP libraries (pslibrary.0.3.1,^{9} pslibrary.1.0.0 high accuracy^{10} and the library proposed by Topsakal and Wentzcovitch for the rareearth elements^{11}), five US libraries (GBRV (versions 1.2, 1.4 and 1.5),^{12} pslibrary.0.3.1,^{9} and pslibrary.1.0.0 high accuracy^{10}) and two NC libraries (SG15, versions 1.0 and 1.1^{13}). For a few selected elements, i.e. N, O, F and Hf, in which all the PSP libraries above perform less well in the Δfactor test we also consider the recent NC Pseudo Dojo library^{14} and, only for N, our own set (called THEOS) of US PSPs. Besides, for the elements from H to Ne in the periodic table, we test the NC library proposed by Willand et al.^{15} which is tailored for systems made of light elements (see Table 1 for a list of all the PSP libraries tested).
Equation of state
In order to assess the precision of PSPs, we compute the Δfactor, i.e. the integral of the difference between the equations of state calculated with PSP simulations and with reference allelectron results. For this purpose we use the protocol introduced in 2014 by Lejaeghere et al.^{2} This protocol was recently exploited to compare 15 different DFT codes, including both allelectron and PSP codes, in order to verify the reproducibility of the PBE equations of state of elemental crystals across different methods and implementations.^{1}
The protocol consists in calculating the energyversusvolume at seven equidistant points centred around the reference equilibrium volume and then performing a Birch–Murnaghan fit. From the parameters of the fit some important physical quantities related to the structural and elastic properties of the system are extracted: the equilibrium volume V_{0}, the bulk modulus B_{0} and the first derivative of the bulk modulus B_{1}. The Δfactor, that is reported in units of meV/atom, gives an overall estimate of the discrepancy between PSPs and allelectron results in terms of these structural properties.
However, as originally noticed by Jollet et al.,^{16} the Δfactor is a stiffnessdependent quantity, being proportional to B_{0}. Indeed, very soft materials, as for example, the noblegas solids, are generally associated with small values of the Δfactor, even for significant volume differences. On the other hand for very hard materials the opposite situation occurs, i.e. small volume differences give rise to large values of the Δfactor. To solve this problem the alternative Δ′factor was introduced, which corresponds to a Δfactor “renormalized” to reference values of V_{0} and B_{0} as described in ref. ^{16}. In Fig. 1 it is shown how the Δ′factor is indeed very strongly correlated with the differences in equilibrium volume, δV_{0}, obtained from the equations of state (note that the data for Δfactor, Δ′factor and δV_{0} reported in the scatter plots of Fig. 1 are taken between pairs of PSPs for the elemental crystals tested in this work and not between PSPs and allelectron results). Instead the Δfactor is scattered along a wide range of slopes that correspond to different values of the bulk modulus B_{0}: small (large) bulk moduli correspond to small (large) slopes. In our protocol we adopt the Δfactor (and not the Δ′factor), as much more Δfactor reference data has been produced in the literature^{1} for several PSP libraries and codes. However, for completeness, we compute also the renormalized Δ′factor and report this value as well. Broadly speaking two compared equations of state can be considered undistinguishable if the Δfactor is smaller than 1 meV/atom (valid for most of the elemental solids but with the notable exception of very soft materials) or if the Δ′factor is smaller than 3 meV/atom, where the latter corresponds to a variation in the equilibrium volume of <0.5% for all elemental solids (see Fig. 1).
The reference allelectron results of the equation of states chosen in this work are the ones of the WIEN2k code^{17} reported in ref. ^{1} with the exception of the rareearth nitrides for which we use the WIEN2k results reported in ref. ^{11}. All PSP calculations needed for the Δfactor estimation are performed at the reference wavefunction cutoff of 200 Ry using a dense Monkhorst–Pack^{18} kgrid of 20 × 20 × 20 and a Marzari–Vanderbilt smearing^{19} of 2 mRy. Magnetism is included for the equations of state of oxygen and chromium (antiferromagnetism), manganese (antiferrimagnetism) and iron, cobalt, nickel and the rareearth nitrides (ferromagnetism).
Within the SSSP testing protocol we study the convergence of four different quantities as a function of the wavefunction cutoff E_{c}, i.e. of the number of planewaves used in the expansion of the Kohn–Sham states. The tested quantities are phonons frequencies at the zoneborder, cohesive energies, pressure and band structures.
All the calculations are performed on the groundstate structures of elemental crystals at 0 K, as provided in http://molmod.ugent.be/deltacodesdft/ with the exception of fluorine for which the SiF_{4} structure is used because of convergence issues of the elemental fluorine structure and of lanthanides that are not included in the test set of http://molmod.ugent.be/deltacodesdft/ and for which the nitride structures of ref. ^{11} are used. In total we test 85 different elements of the periodic table.
In all PSP frameworks, a planewave representation of the charge density requires a cutoff, E_{ρ}, higher than the wavefunction cutoff, E_{c}. It should be noted that here charge densities and wavefunctions actually stand for their pseudopotential counterparts, often called pseudocharge densities and pseudowavefunctions. The true “allelectron” quantities can in some cases be reconstructed, as for instance in the PAW formalism. Typically, convergence tests are performed by varying E_{c} and keeping the dual, i.e. the ratio E_{ρ}/E_{c}, fixed. For instance, in the NC scheme the charge density is simply the modulus squared of the singleparticle wavefunctions, summed over all the electrons, and in reciprocal space it reads:
where sums run over the occupied bands with index n, Bloch vectors k and reciprocal lattice vectors G′. Hence, the largest Gvector appearing in the charge density has modulus twice as large as the largest Gvector appearing in the wavefunction and, as planewave energies scale quadratically, the dual should be equal to 4 to guarantee that all Fourier components are represented. So for NC PSPs we always adopt a dual of 4, although it is known that in some cases calculations could be efficiently converged with a lower dual. In the PAW and US formalisms, wavefunctions are designed to be slowly varying in real space (i.e. to be soft) requiring substantially fewer planewaves to be represented with respect to NC pseudowavefunctions. The price for working with a reduced basis set is the additional complexity in deriving expressions for observables, including the charge density which cannot be simply computed using Eq. (1). However the charge density ρ is the fundamental quantity in DFT. The physics of the problem and the energy functional determine the spatial variation of ρ and thus the cutoff E_{ρ}, independently of the PSP scheme that is adopted. Fundamentally, that is why PAW and US PSPs require relatively high duals, where according to common knowledge a choice of a dual of eight is usually reasonable to efficiently achieve good precision. However, the convergence patterns at duals equal to 12 and 16 are also checked for selected elements that show a particularly high wavefunction cutoff, namely manganese, iron, cobalt, hafnium and oxygen, as also suggested by a convergence study on iron.^{20}
The convergence patterns are obtained at fixed duals as specified above and \(E_{\mathrm {c}}^{{\mathrm{ref}}} = 200\,{\mathrm {Ry}}\) as the reference wavefunction cutoff. All the quantities are considered as differences with respect to the corresponding reference value calculated at \(E_{\mathrm {c}}^{{\mathrm{ref}}}\). An example of the calculated convergence pattern plot is shown in Fig. 2 for the case of palladium. We choose 200 Ry as the reference wavefunction cutoff because, for each element, all the quantities tested in the SSSP testing protocol typically converge well before that value for at least one PSP (with radon being the only exception, as discussed in the Supplementary Figure 1). Therefore, even if for some hard PSPs the convergence plots could be marginally different by using a larger value for \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), those modifications would not be relevant for the conclusions of our work and, in particular, the selection of the SSSP libraries would not be affected.
We perform all the tests on the elemental crystals using a relatively coarse 6 × 6 × 6 Monkhorst–Pack kgrid (except for oxygen and all the lanthanides where a 10 × 10 × 10 kgrid is used instead) because in our protocol for convergence we are not directly interested in the absolute values of the tested quantities but rather on their difference with respect to the reference values computed at \(E_{\mathrm {c}}^{{\mathrm{ref}}}\). We also disregard spinpolarisation in all the convergence tests but we have verified for the magnetic structures that the convergence patterns are not substantially altered by the inclusion of magnetism (see Supplementary Figures 2–7).
Phonon frequencies
The convergence of vibrational properties of elemental crystals is performed by calculating, within the framework of DFPT, the phonon frequencies at the zoneborder of the Brillouin zone, i.e. at the point \({\mathbf{Q}} = \left( {\frac{1}{2},\frac{1}{2},\frac{1}{2}} \right)\) in relative coordinates of the reciprocal lattice vectors. While the Δfactor test is related to the structural and elastic properties of the system, by considering phonon frequencies at the border of the Brillouin zone we have access to information related to both acoustic and optical modes.
The number of phonon frequencies depends on the number of atoms in the unit cell, and so on the element under investigation. In the convergence pattern plots of the SSSP testing protocol we condense the information related to the several phonon frequencies into a single number \(\delta \bar \omega\). It is defined as a relative average deviation (in percentage) among all the phonon frequencies ω_{i} calculated at Q for each wavefunction cutoff E_{c}:
where N is the total number of phonon frequencies. The maximum relative deviation is similarly defined as
and it is represented as an half error bar in the convergence pattern plots.
If the highest phonon frequency ω_{max} of an elemental crystal at Q is smaller than 100 cm^{−1} at \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), the absolute average deviation and the corresponding maximum deviation are computed instead of the relative ones, since a precision of a few cm^{−1} is often the reasonable target for a DFPT calculation.
Cohesive energies
We investigate the convergence of the energy difference between the crystalline solid and the corresponding individually isolated atoms, i.e. the cohesive energy of the elemental crystals. Since periodic boundary conditions are used in the calculations, the isolated atom is placed in a cell of lattice parameter equal to 12 Å to avoid spurious interactions with the periodic images. The quantity δE_{coh} considered in the SSSP testing protocol is defined as the absolute difference between the cohesive energy at a given cutoff E_{c} and the one at the reference wavefunction cutoff \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), i.e. 200 Ry (in units of meV per atom).
Pressure
We evaluate the convergence of the stress by computing the hydrostatic pressure, which is defined as P = 1/3Tr(σ), where σ is the stress tensor. Rather than checking convergence directly on the pressure itself (the magnitude of which depends strongly on the stiffness of the material) we evaluate it through its conversion into an equivalent volume. This allows the definition of a stiffnessagnostic and hence material’s independent convergence criterion. Starting from the Birch–Murnaghan equation of state for the pressure fitted on the reference allelectron calculations
we define the deviation volume V′ as the one closest to the equilibrium volume V_{0} such that P_{BM}(V′) = δP where \(\delta P = P(E_{\mathrm {c}})  P(E_{\mathrm {c}}^{{\mathrm{ref}}})\) is the residual pressure of a calculation performed at the cutoff E_{c}. With this definition, fully converged values of pressure give δP = 0 and therefore V′ = V_{0}. Once V′ is known we can eventually find the relative volume deviation (in percentage) due to the residual pressure: δV_{press} = (V′ − V_{0})/V_{0}, which is the quantity considered in the SSSP testing protocol.
Band structure
The tests discussed so far deal with groundstate quantities only, computed either using DFT or DFPT. However, PSP calculations are often employed to study optical, transport and other properties that involve charged or neutral excitations. The majority of excitedstates calculations are based on manybody perturbation theory (MBPT), e.g. G_{0}W_{0} and selfconsistent GW,^{21,22,23} the Bethe–Salpeter equation (BSE),^{24} or dynamical mean field theory (DMFT),^{25} and are performed ontop of a DFT calculation, which provides the starting point for both selfconsistent and oneshot approaches. Hence, we include band structures in our testing protocol, taking into account both the occupied bands and some of the lower lying unoccupied bands. Here, we outline a protocol for performing both convergence tests and verification of band structures by defining a bands distance (a similar idea has been proposed independently in ref. ^{26}). The aim is to quantify how much two band structures “differ” by introducing a simple and computationally inexpensive metric in the band structures space. We call our bands distance η and consider two cases that are distinguished solely by the number of bands taken into account. The η_{v} (or “eta valence”) considers the occupied bands only, while in the η_{10} (or “eta conduction 10”) all the bands up to 10 eV above the Fermi level are considered. We always use a robust (0.3 eV) Fermi–Dirac smearing to deal with partially occupied bands, while to compute η_{v} for insulators we use no smearing. We choose a 6 × 6 × 6 uniform kgrid, in the full Brillouin zone and with no symmetry reduction. Choosing a highsymmetry path could result in an unsatisfactory arbitrary choice, as different recipes for the standardisation of paths have been introduced in the recent literature^{27,28} and interesting features of the band structure may occur far from the highsymmetry lines (such as Weyl points).^{29,30} A uniform mesh is also more appropriate from the point of view of electron’s nearsightedness:^{31} if the energy eigenvalues are known on a sufficiently fine uniform kpoints mesh, it is possible to get an exact realspace representation of the Hamiltonian in a Wannier function basis^{32} and then interpolate to an arbitrary fine mesh.
Let us suppose we have two sets of bands \(\varepsilon _{n{\mathbf{k}}}^A\) and \(\varepsilon _{n{\mathbf{k}}}^B\); we define the distance between the two sets of (valence) bands as
where
f_{nk}(ε, σ) being the Fermi–Dirac distribution and σ the smearing width. The Fermi energies, \(\varepsilon _{\mathrm {F}}^{(A,B)}\), for the two band structures A and B are obtained from the relation \(N_{{\mathrm {el}}}^{(A,B)} = \mathop {\sum}\nolimits_{n{\mathbf{k}}} f_{n{\mathbf{k}}}^{(A,B)}(\varepsilon _{\mathrm {F}}^{(A,B)},\sigma )\), where N_{el} is the number of electrons. In order to properly align the two sets of bands, η_{v} is defined as the minimum with respect to a rigid energy shift ω.
We now consider also the low lying conduction bands by introducing η_{10}, defined as in Eq. (5) but with a Fermi level up shift of 10 eV. In this way, η_{10} measures the bands distance of the valence bands plus the conduction bands up to 10 eV above the Fermi energy.
Finally, we also take into account the possibility that significant differences between band structures may occur only in subregions of the Brillouin zone or in small energy ranges. After computing the η, we check the slowest converging band by computing max η, defined as
and request that is has to be converged with a slightly higher threshold than η itself.
In the SSSP testing protocol we use η_{10} and max η_{10} (in units of meV) as criteria to quantitatively study the convergence of band structures.
Discussion
SSSP selection criteria
We discuss now the selection criteria used to build our optimal PSP libraries, namely the SSSP efficiency and SSSP precision libraries (version 1.1). As mentioned in the Introduction, our primary goal is to define tested PSP libraries with a focus on efficiency and precision for highthroughput calculations and to suggest converged wavefunction cutoffs. The main idea behind the SSSP precision library is to provide the PSPs that are the closest to allelectron calculations in terms of Δfactor computed at the reference wavefunction cutoff \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), without much consideration on the computational cost and the wavefunction cutoffs actually needed to converge all relevant quantities. On the other hand, the SSSP efficiency library is designed for practical applications that should remain affordable, and therefore PSPs are chosen such that wavefunction cutoffs are as low as possible while keeping the precision reasonable.
The selection criteria are listed in Table 2. For SSSP efficiency, when possible we select PSPs with a rather small Δfactor (below 1 meV/atom). The phonons \(\delta \bar \omega\) should be converged within 2% (or within 2 cm^{−1} if the highest phonon frequency is smaller than 100 cm^{−1}), the cohesive energy δE_{coh} within 2 meV/atom, the pressure within 1% for δV_{press} (i.e. 0.33% on the lattice parameter of a cubic crystal) and the band structure within 10 meV for η_{10} and within 20 meV for max η_{10}. For the SSSP precision, the criteria are slightly stricter (see Table 2) and we systematically opt for the PSP with the smallest Δfactor. Therefore the wavefunction cutoffs of the SSSP precision are typically higher than the ones proposed for the SSSP efficiency.
We underline here that in a few difficult cases the SSSP libraries are built following these criteria as general guidelines and not using these as strict rules. In practice, this means that the PSPs are chosen onebyone through human inspection and not with an automatic procedure. This flexible approach is necessary because the convergence of some of the tested quantities is sometimes slow and/or irregular. For example, it can happen that the selection criteria are not all together satisfied at a reasonable wavefunction cutoff for any of the PSPs of a given element or that the convergence patterns show outlier data points or oscillations. A clear example of this situation is given by the extremely soft noblegas elemental solids for which the convergence of the tested quantities, in particular of the stress tensor and of the phonon frequencies, can be very noisy due to numerical instabilities. In these cases it is therefore necessary to make compromise choices that can sacrifice or increase some of the thresholds imposed by the SSSP selection criteria, if no other possibilities are available or in order to keep the computational cost reasonable.
Ghost states
We use the bands distance η_{10} defined above not only for the convergence tests but also to compare the band structures of the tested PSPs for all the elemental crystals considered. However, different PSPs are often generated with different combinations of semicore states in the valence band. Hence, we compare only the bands they have in common, by taking the minimum number of electrons of all the set and cutting the exceeding lowenergy bands accordingly. By means of this additional criterion it is possible to automatically detect ghost states^{33} in a PSP in the valence and in the conduction up to the chosen threshold (here 10 eV above the Fermi energy), as they are signalled by extremely large values (of the order of eV or more) of the bands distances when computed with respect to other ghostfree PSPs (see Fig. 3 for an example). A list of the tested PSPs having ghost states in the empty conduction bands is reported in Table 3. However, it is worth noting that standard DFT calculations for groundstate properties are unaffected by ghost states above the Fermi level. Nonetheless they could be a possible source of problems for applications related to excitedstate properties (e.g. in MBPT calculations such as GW or BSE). As expected, none of the PSPs considered has ghost states in the valence as they would give unphysical results also for groundstate properties and they would be easily spotted.
We stress here that the bands distance could in principle also be used for verification studies because it would allow for a quantitative comparison of PSPs band structures with reference allelectron band structures.
Correlations among tested quantities
Before giving the list of PSPs chosen by following the SSSP testing protocol, we show with an analysis of our results that an accurate selection of PSPs for generic applications in computational material science needs several independent criteria to be satisfied, based on the estimation of different physical properties. This is done also with the purpose of furnishing an a posteriori justification of the protocol we established for PSP testing. In particular we show that PSPs that give very similar results for a certain tested quantity can give, in a nonnegligible number of cases, significant discrepancies in the estimation of some other quantity.
We compare the PSP results for the physical properties considered in the SSSP testing protocol by calculating the discrepancies between all the available PSPs for a given element, using the data obtained at the reference wavefunction cutoff of 200 Ry. An example is shown in Fig. 4 where the differences between the equations of state (through the use of the Δfactor) and between band structures (through the use of η_{10}) are compared for the case of sodium. We notice that, even if all the sodium PSPs we considered produce very similar equations of states—with Δfactors always smaller than 0.3 meV—the band structure of a particular PSP shows instead substantial differences up to η_{10} = 65 meV.
More generally, we can study correlation between pairs of quantities by looking at scatter plots, where the differences between all possible couples of PSPs for all the 85 elements are considered. In Fig. 5 we show as an example the correlation plot between the equation of state (obtained through both the Δfactor and the Δ′factor) and pressure (δV_{press}), cohesive energy (δE_{coh}), highest phonon frequency (δω_{max}) and valence band structure (η_{v}). Figure 5 shows how for all such properties correlation is very weak, suggesting that the precision of a PSP is property dependent. However, we also notice (see Fig. 5) that the correlation between the Δ′factor and δV_{press} is higher than for the Δfactor. This observation can be rationalised in terms of the Δ′factor renormalisation, that provides an estimate of the difference between two equations of state that is more materialindependent and straightforwardly related to volume differences (see Fig. 1 and discussion therein). From our results we conclude that there is no strong correlation between pairs of tested quantities entering our selection criteria, hence the similarity between PSPs is strongly property dependent.
Up to now, the Δfactor is the only verification test present in the literature to assess the precision of DFT calculations of solids. Given the small correlations among the SSSP criteria, we stress here the importance for the electronic structure community to head for the creation of an heterogeneous set of validated allelectron reference data, which would ideally include other physical properties beyond the equation of state, such as phonons or band structures (such effort is currently coordinated by S. Cottenier). This would allow the extension of the available PSP verification tests beyond the Δfactor for elemental crystals, potentially improving the assessment of PSPs precision.
Similarly, we show that the convergence with respect to the wavefunction cutoff of a given physical property usually has a different and uncorrelated behaviour if compared to other tested quantities (see Fig. 6 for a comparison of the wavefunction cutoffs at which two tested quantities in the SSSP testing protocol reach the required precision for each PSP). Indeed, differences in the mathematical expression adopted and/or in the codespecific implementations that are needed to compute the tested quantities can result in different and independent convergence patterns, so that each quantity reaches the required precision at different wavefunction cutoffs. For example, the derivatives involved in the calculation of the stress tensor or the phonon frequencies either through direct, finite differences methods or linear response theory, can introduce different numerical noise and display a slower convergence if compared to other properties, such as equations of state or band structures, that do not require calculations of derivatives.
In general, it is therefore necessary to study the convergence of each relevant quantity separately, in order to correctly estimate the optimal number of plane waves that gives results converged within the required precision for all the properties of interest.
Exchangecorrelation functionals
Our study on the precision and efficiency of PSPs is restricted to the PBE functional as it is among the most popular ones in the electronic structure community and the only functional for which a verified set of reference allelectron results for solids exists.^{1} So the Δfactor test for verification can be performed, at the moment, only with the PBE functional.
Still, it is worth to comment on the transferability of the convergence tests performed in the SSSP testing protocol among different functionals. For this purpose, we consider the revised PBE GGA for solids, namely the PBEsol^{34} functional, and one of the most widely used functionals for the local density approximation (LDA), i.e. the PZ^{35} functional. By testing some elemental crystals for the GBRV library (see Fig. 7 for the case of Ga and the Supplementary Figures 8 and 9 for a few more systems), we find that the convergence patterns turn out to be very similar if the PSPs are generated with the same atomic parameters (such as electronic configuration, cutoff radii, etc.), thus showing a good transferability of the convergence tests among different local and semilocal functionals. However, performing consistent tests for the transferability of more complex and nonlocal functionals, such as SCAN^{36} (metaGGA) or HSE^{37} (exactexchange), is less straightforward because, as of today, no PSP libraries for these kind of functionals exist. Indeed, in these cases, the common approach followed in all PSP DFT codes is to use PSPs generated with local or semilocal functionals, e.g. PBE, and to then “switch” to the complex functional, e.g. HSE, when performing the PSP DFT calculation including the valence electrons only. Tests performed following this approach, although useful, would not ensure the transferability of the functional under investigation and more extensive and consistent studies on the subject are therefore left to future work, in particular when HSE or SCAN PSP libraries, to only cite a few possible examples, will be available.
SSSP libraries
Table 4 and Table 5 show the two SSSP libraries, efficiency and precision (version 1.1), selected according to the SSSP selection criteria specified above. The suggested wavefunction cutoffs (in Ry) and the duals are also indicated for each PSP chosen. The SSSP periodic table with all PSPs, wavefunction cutoffs and duals is also accesible interactively online on the Materials Cloud platform (https://www.materialscloud.org/) (see Fig. 8). The average suggested wavefunction cutoffs of the SSSP efficiency and SSSP precision over all the 85 elements tested are 44 and 56 Ry, respectively. A dual of 8 has been used for all PSPs except normconserving ones where a dual of 4 is used, and iron and manganese for which a dual of 12 is suggested.
The SSSP efficiency and SSSP precision have small average Δfactors of 0.44 and 0.33 meV, respectively (where the average is performed over all elements tested excluding the 15 rareearth nitrides, following the recipe of ref. ^{1}).
The SSSP libraries have already proven to be a reliable tool for a number of computational studies: for instance the beta version (called version 0.7) of the SSSP libraries have enabled the highthroughput computational exfoliation of twodimensional materials^{38} and have supported the combined experimental and theoretical study of catalysts for oxygen evolution reaction.^{39}
On a more general level, apart from the SSSP testing protocol and libraries, our work provides a database of verification data and convergence tests that facilitates the optimal choice of PSPs and wavefunction cutoffs for custom applications. For example, some physical properties may be implemented only for some PSP types (typically only NC) or some applications may require convergence of just a subset of the quantities that we consider in the SSSP testing protocol. By a look at our plots and data, see for instance the condensed plot for palladium shown in Fig. 2, a user can quickly select the optimal PSP and wavefunction cutoff tailored for the specific application.
In summary, we propose an extensive testing protocol for PSPs to investigate precision and performance of several NC, US and PAW PSP libraries that are publicly available. We incorporate in the SSSP protocol a verification part, based on the Δfactor, and an efficiency part, based on the planewave cutoff convergence tests for phonon frequencies, cohesive energies, pressures, and band structures. Leveraging the SSSP protocol, we identify two optimal PSP libraries, named SSSP efficiency and SSSP precision (version 1.1), that provide thoroughly tested and precise PSPs for 85 elements of the periodic table, selected from publicly available PSP libraries,^{9,10,11,12,13,14,15} for which the original authors should be acknowledged. Our effort not only is particularly relevant for highthroughput computational materials screening, where the right compromise between precision and computational cost is essential, but it substantially contributes to set high the bar of the quality of PSP calculations of solidstate materials. Building on the invaluable work behind all the PSP libraries we considered, we provide a systematic survey of PSP quality across multiple physical properties and multiple libraries and techniques (NC, US and PAW). Our work shows how the assessment of both precision and efficiency of pseudopotentials is strongly propertydependent and requires a multidimensional quality gauge, pointing to the need of a verification standard in the computational solidstate community. Given the importance of PSP calculations in modern materials science, nanotechnology, chemistry and physics, our findings call for more verification efforts aimed at increasing precision and efficiency of computed quantities that are routinely used to discuss novel physics, to help interpreting experiments or even to discover and design novel materials. In particular, we hope that this work will stimulate further investigations in the allelectron community, that ideally would provide more reference data for an heterogeneous set of properties elaborating on our discussion.
Methods
All the calculations needed for this work (more than 50,000 DFT and DFPT calculations) were performed with the goal to ensure reproducibility of all the data obtained, compliant with the FAIR guiding principles for data management.^{40} This is the reason why we used AiiDA,^{5} an opensource Python infrastructure for computational science, that is specifically designed to track the provenance of data and calculations and that allow the user to implement workflows that can run complex sequences of calculations. It is therefore particularly suited for highthroughput studies, such as the deployment of the SSSP testing protocol, where a large number of simulations are involved.
In practice AiiDA can prepare and submit calculations (usually to an HPC cluster) and then retrieve and store the results inside a database, all automatically. The database can be subsequently queried by the user to extract data or other useful informations.
The complete SSSP testing protocol is implemented as an AiiDA workflow, called SsspWorkflow, that can run all the convergence tests and the Δfactor verification test. The SsspWorkflow is built on top of the PwWorkflow, a very robust lowerlevel workflow in charge of handling all the QE simulations and that can restart calculations in case of standard QE errors or, for example, if the userspecified wall time is too small.
The SsspWorkflow allows a generic user to perform all the calculations required by the SSSP protocol in a completely automatic way. With this tool, other and new pseudopotential libraries could be easily tested in the future in order to update the subsequent versions of the SSSP libraries with more precise and efficient PSPs.
Data availability
All the data produced in this work is freely available on the Materials Cloud online platform, https://www.materialscloud.org/, where the user can interactively browse the results and explore the data provenance. The full database with all the data can also be downloaded.^{41}
References
 1.
Lejaeghere, K. et al. Reproducibility in density functional theory calculations of solids. Science 351, aad3000 (2016).
 2.
Lejaeghere, K., Van Speybroeck, V., Van Oost, G. & Cottenier, S. Error estimates for solidstate densityfunctional theory predictions: an overview by means of the groundstate elemental crystals. Crit. Rev. Solid State Mater. Sci. 39, 1–24 (2014).
 3.
Perdew J., Burke K. & Ernzerhof M. Generalized gradient approximation made simple. Phys. Rev. Lett. 77, 3865–3868 (1996).
 4.
Giannozzi, P. et al. QUANTUM ESPRESSO: a modular and opensource software project for quantum simulations of materials. J. Phys.: Condens. Matter 21, 395502 (2009).
 5.
Pizzi, G., Cepellotti, A., Sabatini, R., Marzari, N. & Kozinsky, B. AiiDA: automated interactive infrastructure and database for computational science. Comput. Mater. Sci. 111, 218–230 (2016).
 6.
Hamann, D., Schlüter, M. & Chiang, C. Normconserving pseudopotentials. Phys. Rev. Lett. 43, 1494–1497 (1979).
 7.
Vanderbilt, D. Soft selfconsistent pseudopotentials in a generalized eigenvalue formalism. Phys. Rev. B 41, 7892–7895 (1990).
 8.
Blöchl, P. E. Projector augmentedwave method. Phys. Rev. B 50, 17953–17979 (1994).
 9.
Kucukbenli, E. et al. Projector augmentedwave and allelectron calculations across the periodic table: a comparison of structural and energetic properties. Preprint at arXiv:1404.3015 (2014).
 10.
Dal Corso, A. Pseudopotentials periodic table: from H to Pu. Comput. Mater. Sci. 95, 337–350 (2014).
 11.
Topsakal, M. & Wentzcovitch, R. Accurate projected augmented wave (PAW) datasets for rareearth elements (RE = La–Lu). Comput. Mater. Sci. 95, 263–270 (2014).
 12.
Garrity, K. F., Bennett, J. W., Rabe, K. M. & Vanderbilt, D. Pseudopotentials for highthroughput DFT calculations. Comput. Mater. Sci. 81, 446–452 (2014).
 13.
Schlipf, M. & Gygi, F. Optimization algorithm for the generation of ONCV pseudopotentials. Comput. Phys. Commun. 196, 36–44 (2015).
 14.
van Setten, M. et al. The pseudodojo: training and grading a 85 element optimized normconserving pseudopotential table. Comput. Phys. Commun. 226, 39–54 (2018).
 15.
Willand, A. et al. Normconserving pseudopotentials with chemical accuracy compared to allelectron calculations. J. Chem. Phys. 138, 104109 (2013).
 16.
Jollet, F., Torrent, M. & Holzwarth, N. Generation of projector augmentedwave atomic data: a 71 element validated table in the xml format. Comput. Phys. Commun. 185, 1246–1254 (2014).
 17.
Blaha, P., Schwarz, K., Sorantin, P. & Trickey, S. B. Fullpotential, linearized augmented plane wave programs for crystalline systems. Comput. Phys. Commun. 59, 399–415 (1990).
 18.
Monkhorst, H. J. & Pack, J. D. Special points for Brillouinzone integrations. Phys. Rev. B 13, 5188–5192 (1976).
 19.
Marzari, N., Vanderbilt, D., De Vita, A. & Payne, M. C. Thermal contraction and disordering of the Al(110) surface. Phys. Rev. Lett. 82, 3296–3299 (1999).
 20.
Dragoni, D., Ceresoli, D. & Marzari, N. Thermoelastic properties of αiron from firstprinciples. Phys. Rev. B 91, 104105 (2015).
 21.
Hybertsen, M. S. & Louie, S. G. Electron correlation in semiconductors and insulators: band gaps and quasiparticle energies. Phys. Rev. B 34, 5390–5413 (1986).
 22.
Onida, G., Reining, L. & Rubio, A. Electronic excitations: densityfunctional versus manybody Green’sfunction approaches. Rev. Mod. Phys. 74, 601–659 (2002).
 23.
Reining, L. The GW approximation: content, successes and limitations. Wiley Interdiscip. Rev.: Comput. Mol. Sci. 8, e1344 (2018).
 24.
Strinati, G. Application of the Green’s functions method to the study of the optical properties of semiconductors. Riv. Nuovo Cim. 11, 1–86 (1988).
 25.
Georges, A., Kotliar, G., Krauth, W. & Rozenberg, M. J. Dynamical meanfield theory of strongly correlated fermion systems and the limit of infinite dimensions. Rev. Mod. Phys. 68, 13–125 (1996).
 26.
Huhn, W. P. & Blum, V. Onehundredthree compound bandstructure benchmark of postselfconsistent spinorbit coupling treatments in density functional theory. Phys. Rev. Mater. 1, 033803 (2017).
 27.
Hinuma, Y., Pizzi, G., Kumagai, Y., Oba, F. & Tanaka, I. Band structure diagram paths based on crystallography. Comput. Mater. Sci. 128, 140–184 (2017).
 28.
Setyawan, W. & Curtarolo, S. Highthroughput electronic band structure calculations: challenges and tools. Comput. Mater. Sci. 49, 299–312 (2010).
 29.
Xu, S.Y. et al. Discovery of a Weyl fermion semimetal and topological Fermi arcs. Science 349, 613–617 (2015).
 30.
Soluyanov, A. A. et al. TypeII Weyl semimetals. Nature 527, 495–498 (2015).
 31.
Prodan, E. & Kohn, W. Nearsightedness of electronic matter. Proc. Natl Acad. Sci. USA 102, 11635–11638 (2005).
 32.
Marzari, N., Mostofi, A. A., Yates, J. R., Souza, I. & Vanderbilt, D. Maximally localized Wannier functions: theory and applications. Rev. Mod. Phys. 84, 1419–1475 (2012).
 33.
Gonze, X., Stumpf, R. & Scheffler, M. Analysis of separable potentials. Phys. Rev. B 44, 8503–8513 (1991).
 34.
Perdew, J. P. et al. Restoring the densitygradient expansion for exchange in solids and surfaces. Phys. Rev. Lett. 100, 136406 (2008).
 35.
Perdew, J. P. & Zunger, A. Selfinteraction correction to densityfunctional approximations for manyelectron systems. Phys. Rev. B 23, 5048–5079 (1981).
 36.
Sun, J., Ruzsinszky, A. & Perdew, J. P. Strongly constrained and appropriately normed semilocal density functional. Phys. Rev. Lett. 115, 036402 (2015).
 37.
Heyd, J., Scuseria, G. E. & Ernzerhof, M. Hybrid functionals based on a screened Coulomb potential. J. Chem. Phys. 118, 8207–8215 (2003).
 38.
Mounet, N. et al. Twodimensional materials from highthroughput computational exfoliation of experimentally known compounds. Nat. Nanotechnol. 13, 246–252 (2018).
 39.
Lebedev, D. et al. Highly active and stable iridium pyrochlores for oxygen evolution reaction. Chem. Mater. 29, 5182–5191 (2017).
 40.
Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3, 160018 (2016).
 41.
Prandini, G., Marrazzo, A., Castelli, I. E., Mounet, N. & Marzari, N. A Standard Solid State Pseudopotentials (SSSP) library optimized for precision and efficiency (Version 1.1, data download). Materials Cloud Archive (2018); https://archive.materialscloud.org/2018.0001/v3.
Acknowledgements
The authors warmly thank Fernando Gargiulo, Snehal Waychal and Elsa Passaro for assistance with the SSSP section of the Materials Cloud web platform, Giovanni Pizzi and Andrea Cepellotti for the help in developing AiiDA workflows, Marco Gibertini for numerous discussions and for pointing out several useful references, Nicolas G. Hörmann for further testing on the SSSP pseudopotential libraries, Stefaan Cottenier for critical reading of the manuscript. This work has been supported by NCCR MARVEL and by H2020 CoE MaX, computing time has been provided by the Swiss National Supercomputing Centre (CSCS) and by PRACE (Project Ids 2016153543 and 2016163963). We also thank Sadas Shankar and Intel Corporation for early support to this project, through their seed funding on “Validated pseudopotentials for electronicstructure simulations” (2009–11).
Author information
Affiliations
Contributions
N.Ma. and I.C. designed the study; G.P., A.M., I.C. and N.Mo. developed the workflows and performed the calculations; G.P., A.M. and I.C. wrote the manuscript, and all authors discussed the protocol, analysed the data, and commented on the manuscript. G.P., A.M. and I.C. contributed equally to this work.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Prandini, G., Marrazzo, A., Castelli, I.E. et al. Precision and efficiency in solidstate pseudopotential calculations. npj Comput Mater 4, 72 (2018). https://doi.org/10.1038/s4152401801272
Received:
Accepted:
Published:
Further reading

Magnetism and stability of all primitive stacking patterns in bilayer chromium trihalides
Journal of Physics D: Applied Physics (2021)

Synergistic effects in oxygen evolution activity of mixed iridiumruthenium pyrochlores
Electrochimica Acta (2021)

Materials Cloud, a platform for open computational science
Scientific Data (2020)

Machine learning for metallurgy I. A neuralnetwork potential for AlCu
Physical Review Materials (2020)

Controlling reaction process to realize high thermoelectric performance in filled skutterudites
Journal of Alloys and Compounds (2020)