## Abstract

Despite the enormous success and popularity of density-functional theory, systematic verification and validation studies are still limited in number and scope. Here, we propose a protocol to test publicly available pseudopotential libraries, based on several independent criteria including verification against all-electron equations of state and plane-wave convergence tests for phonon frequencies, band structure, cohesive energy and pressure. Adopting these criteria we obtain curated pseudopotential libraries (named SSSP or standard solid-state pseudopotential libraries), that we target for high-throughput materials screening (“SSSP efficiency”) and high-precision materials modelling (“SSSP precision”). This latter scores highest among open-source pseudopotential libraries available in the Δ-factor test of equations of states of elemental solids.

## Introduction

In the last three decades, atomistic electronic-structure methods have radically changed the way we think at materials theory and simulations. The 1998 Nobel prize in Chemistry given for density-functional theory (DFT) acknowledges this shift of paradigm. The ability to reduce the complexity of the many-body quantum-mechanical problem involving interacting electrons and nuclei into more tractable forms and algorithms allowed to leverage inexpensive and exponentially growing computational power, in order to provide sufficiently precise predictions for a great number of materials. What followed has been a flourishing of methods to compute more and more complex materials properties, and most notably spectroscopies (such as Raman, IR, ARPES, EELS, NMR and more). Nowadays, the (over)confidence in first-principles methods is such that they are routinely used to help interpreting experiments and guide the discovery and understanding of novel materials. In particular, systematic DFT-based computational materials screening is a fast-growing field of research, as reflected by the creation, in the last few years, of several research centres worldwide on computational materials discovery. Up to now, the most visible output has been the creation of large databases of materials properties obtained from first principles, to be compared with or to augment experimental databases such as the Pauling file (http://paulingfile.com/), COD (http://www.crystallography.net/cod/) or ICSD (http://www2.fiz-karlsruhe.de/icsd_home.html/). Even from a purely financial perspective, the personnel cost for plane-wave studies is of the order of 1 billion US$ per year, in purchasing power parity (PPP) terms, matched by substantial hardware usage (see Supplementary Note 1). It is ever so more remarkable that in spite all of this, the efforts of verification of the precision of the underlying pseudopotentials (PSPs) or projector augmented-wave (PAW) approximations have been minimal. Only in 2016 a multi-group effort was able to establish a baseline in the calculations of the equations of state of elemental crystals.^{1} In addition to the issue of precision, or of verification (i.e. insuring that the removal of the core electrons from the calculations performs with the required tolerance), the issue of performance looms large—a softer, smoother PSP will allow straightforwardly faster calculations, both because the basis set is decreased as the computational cost scales with the square of the basis size, and because the minimisation or iterative approaches can become more efficient or better pre-conditioned with a smaller basis.

Here, we introduce a PSP testing protocol based on extensive DFT and density-functional perturbation theory (DFPT) calculations of elemental solids, and select the optimal PSP for 85 elements of the periodic table. Our protocol, named standard solid-state pseudopotential (SSSP) testing protocol, is made of a verification part, based on the Δ-factor (i.e. the difference between all-electron and PSP equations of state),^{1,2} and an extensive performance-oriented part based on plane-wave convergence tests for phonon frequencies, band structures, cohesive energies and stress tensors. We underline here that the SSSP testing protocol is a protocol based on verification and not on validation (following the nomenclature of ref. ^{1}). Indeed our primary goal is to test the precision (verification) of the PSPs and thus to furnish, through our testing protocol, PSP libraries that give results as close as possible to the “exact” theoretical results for the PBE functional, as they would be obtained through a perfectly converged all-electron calculation. We do not perform any comparison with respect to experimental results, i.e. we do not test the accuracy (validation) of the PSPs.

We consider up to eight (depending on the element) publicly available PSP libraries for the PBE functional^{3} and test them with the PWscf and Phonon codes of the Quantum ESPRESSO (QE) distribution^{4} in an automated fashion within the framework of the AiiDA^{5} infrastructure for reproducible computational science. AiiDA also allows straightforward dissemination of results through the Materials Cloud web platform (https://www.materialscloud.org/), a cloud service designed to enable data sharing in computational materials science.

In this paper, first we describe the test set of physically relevant quantities defined for our SSSP testing protocol and what are the corresponding selection criteria used in order to select the best PSPs. Then, we discuss the need for testing several properties, showing how different PSPs may yield similar results and convergence behaviours for one property but different ones for another property. Finally, we propose two optimal PSP libraries chosen according to the SSSP testing protocol and criteria.

## Results

### SSSP testing protocol

In this work, we investigate the precision and performance of several PSPs libraries available for the QE distribution.^{4} QE is an integrated suite of open-source codes for electronic-structure calculations based on DFT which uses plane-waves as basis set and PSPs to represent the electrostatic electron–ion interactions. Nowadays QE is one of the most popular DFT codes adopted by researchers for the first-principles study of materials properties, with more than 2000 citations last year, according to Google Scholar™.

All the tested PSP libraries are based on the generalised gradient approximation (GGA) for the exchange-correlation functional of Perdew, Burke and Ernzerhof (PBE)^{3} and they include the three main pseudization approaches: norm-conserving (NC),^{6} ultrasoft (US)^{7} and projector-augmented wave (PAW)^{8} (in the following, we refer both to PSP and PAW approaches as PSP methods). In particular, we investigate three PAW PSP libraries (pslibrary.0.3.1,^{9} pslibrary.1.0.0 high accuracy^{10} and the library proposed by Topsakal and Wentzcovitch for the rare-earth elements^{11}), five US libraries (GBRV (versions 1.2, 1.4 and 1.5),^{12} pslibrary.0.3.1,^{9} and pslibrary.1.0.0 high accuracy^{10}) and two NC libraries (SG15, versions 1.0 and 1.1^{13}). For a few selected elements, i.e. N, O, F and Hf, in which all the PSP libraries above perform less well in the Δ-factor test we also consider the recent NC Pseudo Dojo library^{14} and, only for N, our own set (called THEOS) of US PSPs. Besides, for the elements from H to Ne in the periodic table, we test the NC library proposed by Willand et al.^{15} which is tailored for systems made of light elements (see Table 1 for a list of all the PSP libraries tested).

#### Equation of state

In order to assess the precision of PSPs, we compute the Δ-factor, i.e. the integral of the difference between the equations of state calculated with PSP simulations and with reference all-electron results. For this purpose we use the protocol introduced in 2014 by Lejaeghere et al.^{2} This protocol was recently exploited to compare 15 different DFT codes, including both all-electron and PSP codes, in order to verify the reproducibility of the PBE equations of state of elemental crystals across different methods and implementations.^{1}

The protocol consists in calculating the energy-versus-volume at seven equidistant points centred around the reference equilibrium volume and then performing a Birch–Murnaghan fit. From the parameters of the fit some important physical quantities related to the structural and elastic properties of the system are extracted: the equilibrium volume *V*_{0}, the bulk modulus *B*_{0} and the first derivative of the bulk modulus *B*_{1}. The Δ-factor, that is reported in units of meV/atom, gives an overall estimate of the discrepancy between PSPs and all-electron results in terms of these structural properties.

However, as originally noticed by Jollet et al.,^{16} the Δ-factor is a stiffness-dependent quantity, being proportional to *B*_{0}. Indeed, very soft materials, as for example, the noble-gas solids, are generally associated with small values of the Δ-factor, even for significant volume differences. On the other hand for very hard materials the opposite situation occurs, i.e. small volume differences give rise to large values of the Δ-factor. To solve this problem the alternative Δ′-factor was introduced, which corresponds to a Δ-factor “renormalized” to reference values of *V*_{0} and *B*_{0} as described in ref. ^{16}. In Fig. 1 it is shown how the Δ′-factor is indeed very strongly correlated with the differences in equilibrium volume, *δV*_{0}, obtained from the equations of state (note that the data for Δ-factor, Δ′-factor and *δV*_{0} reported in the scatter plots of Fig. 1 are taken between pairs of PSPs for the elemental crystals tested in this work and not between PSPs and all-electron results). Instead the Δ-factor is scattered along a wide range of slopes that correspond to different values of the bulk modulus *B*_{0}: small (large) bulk moduli correspond to small (large) slopes. In our protocol we adopt the Δ-factor (and not the Δ′-factor), as much more Δ-factor reference data has been produced in the literature^{1} for several PSP libraries and codes. However, for completeness, we compute also the renormalized Δ′-factor and report this value as well. Broadly speaking two compared equations of state can be considered undistinguishable if the Δ-factor is smaller than 1 meV/atom (valid for most of the elemental solids but with the notable exception of very soft materials) or if the Δ′-factor is smaller than 3 meV/atom, where the latter corresponds to a variation in the equilibrium volume of <0.5% for all elemental solids (see Fig. 1).

The reference all-electron results of the equation of states chosen in this work are the ones of the WIEN2k code^{17} reported in ref. ^{1} with the exception of the rare-earth nitrides for which we use the WIEN2k results reported in ref. ^{11}. All PSP calculations needed for the Δ-factor estimation are performed at the reference wavefunction cutoff of 200 Ry using a dense Monkhorst–Pack^{18} *k*-grid of 20 × 20 × 20 and a Marzari–Vanderbilt smearing^{19} of 2 mRy. Magnetism is included for the equations of state of oxygen and chromium (antiferromagnetism), manganese (antiferrimagnetism) and iron, cobalt, nickel and the rare-earth nitrides (ferromagnetism).

Within the SSSP testing protocol we study the convergence of four different quantities as a function of the wavefunction cutoff *E*_{c}, i.e. of the number of plane-waves used in the expansion of the Kohn–Sham states. The tested quantities are phonons frequencies at the zone-border, cohesive energies, pressure and band structures.

All the calculations are performed on the ground-state structures of elemental crystals at 0 K, as provided in http://molmod.ugent.be/deltacodesdft/ with the exception of fluorine for which the SiF_{4} structure is used because of convergence issues of the elemental fluorine structure and of lanthanides that are not included in the test set of http://molmod.ugent.be/deltacodesdft/ and for which the nitride structures of ref. ^{11} are used. In total we test 85 different elements of the periodic table.

In all PSP frameworks, a plane-wave representation of the charge density requires a cutoff, *E*_{ρ}, higher than the wavefunction cutoff, *E*_{c}. It should be noted that here charge densities and wavefunctions actually stand for their pseudopotential counterparts, often called pseudo-charge densities and pseudo-wavefunctions. The true “all-electron” quantities can in some cases be reconstructed, as for instance in the PAW formalism. Typically, convergence tests are performed by varying *E*_{c} and keeping the dual, i.e. the ratio *E*_{ρ}/*E*_{c}, fixed. For instance, in the NC scheme the charge density is simply the modulus squared of the single-particle wavefunctions, summed over all the electrons, and in reciprocal space it reads:

where sums run over the occupied bands with index *n*, Bloch vectors **k** and reciprocal lattice vectors **G**′. Hence, the largest **G**-vector appearing in the charge density has modulus twice as large as the largest **G**-vector appearing in the wavefunction and, as plane-wave energies scale quadratically, the dual should be equal to 4 to guarantee that all Fourier components are represented. So for NC PSPs we always adopt a dual of 4, although it is known that in some cases calculations could be efficiently converged with a lower dual. In the PAW and US formalisms, wavefunctions are designed to be slowly varying in real space (i.e. to be soft) requiring substantially fewer plane-waves to be represented with respect to NC pseudo-wavefunctions. The price for working with a reduced basis set is the additional complexity in deriving expressions for observables, including the charge density which cannot be simply computed using Eq. (1). However the charge density *ρ* is the fundamental quantity in DFT. The physics of the problem and the energy functional determine the spatial variation of *ρ* and thus the cutoff *E*_{ρ}, independently of the PSP scheme that is adopted. Fundamentally, that is why PAW and US PSPs require relatively high duals, where according to common knowledge a choice of a dual of eight is usually reasonable to efficiently achieve good precision. However, the convergence patterns at duals equal to 12 and 16 are also checked for selected elements that show a particularly high wavefunction cutoff, namely manganese, iron, cobalt, hafnium and oxygen, as also suggested by a convergence study on iron.^{20}

The convergence patterns are obtained at fixed duals as specified above and \(E_{\mathrm {c}}^{{\mathrm{ref}}} = 200\,{\mathrm {Ry}}\) as the reference wavefunction cutoff. All the quantities are considered as differences with respect to the corresponding reference value calculated at \(E_{\mathrm {c}}^{{\mathrm{ref}}}\). An example of the calculated convergence pattern plot is shown in Fig. 2 for the case of palladium. We choose 200 Ry as the reference wavefunction cutoff because, for each element, all the quantities tested in the SSSP testing protocol typically converge well before that value for at least one PSP (with radon being the only exception, as discussed in the Supplementary Figure 1). Therefore, even if for some hard PSPs the convergence plots could be marginally different by using a larger value for \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), those modifications would not be relevant for the conclusions of our work and, in particular, the selection of the SSSP libraries would not be affected.

We perform all the tests on the elemental crystals using a relatively coarse 6 × 6 × 6 Monkhorst–Pack k-grid (except for oxygen and all the lanthanides where a 10 × 10 × 10 k-grid is used instead) because in our protocol for convergence we are not directly interested in the absolute values of the tested quantities but rather on their difference with respect to the reference values computed at \(E_{\mathrm {c}}^{{\mathrm{ref}}}\). We also disregard spin-polarisation in all the convergence tests but we have verified for the magnetic structures that the convergence patterns are not substantially altered by the inclusion of magnetism (see Supplementary Figures 2–7).

#### Phonon frequencies

The convergence of vibrational properties of elemental crystals is performed by calculating, within the framework of DFPT, the phonon frequencies at the zone-border of the Brillouin zone, i.e. at the point \({\mathbf{Q}} = \left( {\frac{1}{2},\frac{1}{2},\frac{1}{2}} \right)\) in relative coordinates of the reciprocal lattice vectors. While the Δ-factor test is related to the structural and elastic properties of the system, by considering phonon frequencies at the border of the Brillouin zone we have access to information related to both acoustic and optical modes.

The number of phonon frequencies depends on the number of atoms in the unit cell, and so on the element under investigation. In the convergence pattern plots of the SSSP testing protocol we condense the information related to the several phonon frequencies into a single number \(\delta \bar \omega\). It is defined as a relative average deviation (in percentage) among all the phonon frequencies *ω*_{i} calculated at **Q** for each wavefunction cutoff *E*_{c}:

where *N* is the total number of phonon frequencies. The maximum relative deviation is similarly defined as

and it is represented as an half error bar in the convergence pattern plots.

If the highest phonon frequency *ω*_{max} of an elemental crystal at **Q** is smaller than 100 cm^{−1} at \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), the absolute average deviation and the corresponding maximum deviation are computed instead of the relative ones, since a precision of a few cm^{−1} is often the reasonable target for a DFPT calculation.

#### Cohesive energies

We investigate the convergence of the energy difference between the crystalline solid and the corresponding individually isolated atoms, i.e. the cohesive energy of the elemental crystals. Since periodic boundary conditions are used in the calculations, the isolated atom is placed in a cell of lattice parameter equal to 12 Å to avoid spurious interactions with the periodic images. The quantity *δE*_{coh} considered in the SSSP testing protocol is defined as the absolute difference between the cohesive energy at a given cutoff *E*_{c} and the one at the reference wavefunction cutoff \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), i.e. 200 Ry (in units of meV per atom).

#### Pressure

We evaluate the convergence of the stress by computing the hydrostatic pressure, which is defined as *P* = 1/3Tr(*σ*), where *σ* is the stress tensor. Rather than checking convergence directly on the pressure itself (the magnitude of which depends strongly on the stiffness of the material) we evaluate it through its conversion into an equivalent volume. This allows the definition of a stiffness-agnostic and hence material’s independent convergence criterion. Starting from the Birch–Murnaghan equation of state for the pressure fitted on the reference all-electron calculations

we define the deviation volume *V*′ as the one closest to the equilibrium volume *V*_{0} such that *P*_{BM}(*V*′) = *δP* where \(\delta P = P(E_{\mathrm {c}}) - P(E_{\mathrm {c}}^{{\mathrm{ref}}})\) is the residual pressure of a calculation performed at the cutoff *E*_{c}. With this definition, fully converged values of pressure give *δP* = 0 and therefore *V*′ = *V*_{0}. Once *V*′ is known we can eventually find the relative volume deviation (in percentage) due to the residual pressure: *δV*_{press} = (*V*′ − *V*_{0})/*V*_{0}, which is the quantity considered in the SSSP testing protocol.

#### Band structure

The tests discussed so far deal with ground-state quantities only, computed either using DFT or DFPT. However, PSP calculations are often employed to study optical, transport and other properties that involve charged or neutral excitations. The majority of excited-states calculations are based on many-body perturbation theory (MBPT), e.g. *G*_{0}*W*_{0} and self-consistent *GW*,^{21,22,23} the Bethe–Salpeter equation (BSE),^{24} or dynamical mean field theory (DMFT),^{25} and are performed on-top of a DFT calculation, which provides the starting point for both self-consistent and one-shot approaches. Hence, we include band structures in our testing protocol, taking into account both the occupied bands and some of the lower lying unoccupied bands. Here, we outline a protocol for performing both convergence tests and verification of band structures by defining a bands distance (a similar idea has been proposed independently in ref. ^{26}). The aim is to quantify how much two band structures “differ” by introducing a simple and computationally inexpensive metric in the band structures space. We call our bands distance *η* and consider two cases that are distinguished solely by the number of bands taken into account. The *η*_{v} (or “eta valence”) considers the occupied bands only, while in the *η*_{10} (or “eta conduction 10”) all the bands up to 10 eV above the Fermi level are considered. We always use a robust (0.3 eV) Fermi–Dirac smearing to deal with partially occupied bands, while to compute *η*_{v} for insulators we use no smearing. We choose a 6 × 6 × 6 uniform k-grid, in the full Brillouin zone and with no symmetry reduction. Choosing a high-symmetry path could result in an unsatisfactory arbitrary choice, as different recipes for the standardisation of paths have been introduced in the recent literature^{27,28} and interesting features of the band structure may occur far from the high-symmetry lines (such as Weyl points).^{29,30} A uniform mesh is also more appropriate from the point of view of electron’s nearsightedness:^{31} if the energy eigenvalues are known on a sufficiently fine uniform **k**-points mesh, it is possible to get an *exact* real-space representation of the Hamiltonian in a Wannier function basis^{32} and then interpolate to an arbitrary fine mesh.

Let us suppose we have two sets of bands \(\varepsilon _{n{\mathbf{k}}}^A\) and \(\varepsilon _{n{\mathbf{k}}}^B\); we define the distance between the two sets of (valence) bands as

where

*f*_{nk}(*ε*, *σ*) being the Fermi–Dirac distribution and *σ* the smearing width. The Fermi energies, \(\varepsilon _{\mathrm {F}}^{(A,B)}\), for the two band structures *A* and *B* are obtained from the relation \(N_{{\mathrm {el}}}^{(A,B)} = \mathop {\sum}\nolimits_{n{\mathbf{k}}} f_{n{\mathbf{k}}}^{(A,B)}(\varepsilon _{\mathrm {F}}^{(A,B)},\sigma )\), where *N*_{el} is the number of electrons. In order to properly align the two sets of bands*, η*_{v} is defined as the minimum with respect to a rigid energy shift *ω*.

We now consider also the low lying conduction bands by introducing *η*_{10}, defined as in Eq. (5) but with a Fermi level up shift of 10 eV. In this way, *η*_{10} measures the bands distance of the valence bands plus the conduction bands up to 10 eV above the Fermi energy.

Finally, we also take into account the possibility that significant differences between band structures may occur only in subregions of the Brillouin zone or in small energy ranges. After computing the *η*, we check the slowest converging band by computing max *η*, defined as

and request that is has to be converged with a slightly higher threshold than *η* itself.

In the SSSP testing protocol we use *η*_{10} and max *η*_{10} (in units of meV) as criteria to quantitatively study the convergence of band structures.

## Discussion

### SSSP selection criteria

We discuss now the selection criteria used to build our optimal PSP libraries, namely the SSSP efficiency and SSSP precision libraries (version 1.1). As mentioned in the Introduction, our primary goal is to define tested PSP libraries with a focus on efficiency and precision for high-throughput calculations and to suggest converged wavefunction cutoffs. The main idea behind the SSSP precision library is to provide the PSPs that are the closest to all-electron calculations in terms of Δ-factor computed at the reference wavefunction cutoff \(E_{\mathrm {c}}^{{\mathrm{ref}}}\), without much consideration on the computational cost and the wavefunction cutoffs actually needed to converge all relevant quantities. On the other hand, the SSSP efficiency library is designed for practical applications that should remain affordable, and therefore PSPs are chosen such that wavefunction cutoffs are as low as possible while keeping the precision reasonable.

The selection criteria are listed in Table 2. For SSSP efficiency, when possible we select PSPs with a rather small Δ-factor (below 1 meV/atom). The phonons \(\delta \bar \omega\) should be converged within 2% (or within 2 cm^{−1} if the highest phonon frequency is smaller than 100 cm^{−1}), the cohesive energy *δE*_{coh} within 2 meV/atom, the pressure within 1% for *δV*_{press} (i.e. 0.33% on the lattice parameter of a cubic crystal) and the band structure within 10 meV for *η*_{10} and within 20 meV for max *η*_{10}. For the SSSP precision, the criteria are slightly stricter (see Table 2) and we systematically opt for the PSP with the smallest Δ-factor. Therefore the wavefunction cutoffs of the SSSP precision are typically higher than the ones proposed for the SSSP efficiency.

We underline here that in a few difficult cases the SSSP libraries are built following these criteria as general guidelines and not using these as strict rules. In practice, this means that the PSPs are chosen one-by-one through human inspection and not with an automatic procedure. This flexible approach is necessary because the convergence of some of the tested quantities is sometimes slow and/or irregular. For example, it can happen that the selection criteria are not all together satisfied at a reasonable wavefunction cutoff for any of the PSPs of a given element or that the convergence patterns show outlier data points or oscillations. A clear example of this situation is given by the extremely soft noble-gas elemental solids for which the convergence of the tested quantities, in particular of the stress tensor and of the phonon frequencies, can be very noisy due to numerical instabilities. In these cases it is therefore necessary to make compromise choices that can sacrifice or increase some of the thresholds imposed by the SSSP selection criteria, if no other possibilities are available or in order to keep the computational cost reasonable.

### Ghost states

We use the bands distance *η*_{10} defined above not only for the convergence tests but also to compare the band structures of the tested PSPs for all the elemental crystals considered. However, different PSPs are often generated with different combinations of semi-core states in the valence band. Hence, we compare only the bands they have in common, by taking the minimum number of electrons of all the set and cutting the exceeding low-energy bands accordingly. By means of this additional criterion it is possible to automatically detect ghost states^{33} in a PSP in the valence and in the conduction up to the chosen threshold (here 10 eV above the Fermi energy), as they are signalled by extremely large values (of the order of eV or more) of the bands distances when computed with respect to other ghost-free PSPs (see Fig. 3 for an example). A list of the tested PSPs having ghost states in the empty conduction bands is reported in Table 3. However, it is worth noting that standard DFT calculations for ground-state properties are unaffected by ghost states above the Fermi level. Nonetheless they could be a possible source of problems for applications related to excited-state properties (e.g. in MBPT calculations such as GW or BSE). As expected, none of the PSPs considered has ghost states in the valence as they would give unphysical results also for ground-state properties and they would be easily spotted.

We stress here that the bands distance could in principle also be used for verification studies because it would allow for a quantitative comparison of PSPs band structures with reference all-electron band structures.

### Correlations among tested quantities

Before giving the list of PSPs chosen by following the SSSP testing protocol, we show with an analysis of our results that an accurate selection of PSPs for generic applications in computational material science needs several independent criteria to be satisfied, based on the estimation of different physical properties. This is done also with the purpose of furnishing an a posteriori justification of the protocol we established for PSP testing. In particular we show that PSPs that give very similar results for a certain tested quantity can give, in a non-negligible number of cases, significant discrepancies in the estimation of some other quantity.

We compare the PSP results for the physical properties considered in the SSSP testing protocol by calculating the discrepancies between all the available PSPs for a given element, using the data obtained at the reference wavefunction cutoff of 200 Ry. An example is shown in Fig. 4 where the differences between the equations of state (through the use of the Δ-factor) and between band structures (through the use of *η*_{10}) are compared for the case of sodium. We notice that, even if all the sodium PSPs we considered produce very similar equations of states—with Δ-factors always smaller than 0.3 meV—the band structure of a particular PSP shows instead substantial differences up to *η*_{10} = 65 meV.

More generally, we can study correlation between pairs of quantities by looking at scatter plots, where the differences between all possible couples of PSPs for all the 85 elements are considered. In Fig. 5 we show as an example the correlation plot between the equation of state (obtained through both the Δ-factor and the Δ′-factor) and pressure (*δV*_{press}), cohesive energy (*δE*_{coh}), highest phonon frequency (*δω*_{max}) and valence band structure (*η*_{v}). Figure 5 shows how for all such properties correlation is very weak, suggesting that the precision of a PSP is property dependent. However, we also notice (see Fig. 5) that the correlation between the Δ′-factor and *δV*_{press} is higher than for the Δ-factor. This observation can be rationalised in terms of the Δ′-factor renormalisation, that provides an estimate of the difference between two equations of state that is more material-independent and straightforwardly related to volume differences (see Fig. 1 and discussion therein). From our results we conclude that there is no strong correlation between pairs of tested quantities entering our selection criteria, hence the similarity between PSPs is strongly property dependent.

Up to now, the Δ-factor is the only verification test present in the literature to assess the precision of DFT calculations of solids. Given the small correlations among the SSSP criteria, we stress here the importance for the electronic structure community to head for the creation of an heterogeneous set of validated all-electron reference data, which would ideally include other physical properties beyond the equation of state, such as phonons or band structures (such effort is currently coordinated by S. Cottenier). This would allow the extension of the available PSP verification tests beyond the Δ-factor for elemental crystals, potentially improving the assessment of PSPs precision.

Similarly, we show that the convergence with respect to the wavefunction cutoff of a given physical property usually has a different and uncorrelated behaviour if compared to other tested quantities (see Fig. 6 for a comparison of the wavefunction cutoffs at which two tested quantities in the SSSP testing protocol reach the required precision for each PSP). Indeed, differences in the mathematical expression adopted and/or in the code-specific implementations that are needed to compute the tested quantities can result in different and independent convergence patterns, so that each quantity reaches the required precision at different wavefunction cutoffs. For example, the derivatives involved in the calculation of the stress tensor or the phonon frequencies either through direct, finite differences methods or linear response theory, can introduce different numerical noise and display a slower convergence if compared to other properties, such as equations of state or band structures, that do not require calculations of derivatives.

In general, it is therefore necessary to study the convergence of each relevant quantity separately, in order to correctly estimate the optimal number of plane waves that gives results converged within the required precision for all the properties of interest.

### Exchange-correlation functionals

Our study on the precision and efficiency of PSPs is restricted to the PBE functional as it is among the most popular ones in the electronic structure community and the only functional for which a verified set of reference all-electron results for solids exists.^{1} So the Δ-factor test for verification can be performed, at the moment, only with the PBE functional.

Still, it is worth to comment on the transferability of the convergence tests performed in the SSSP testing protocol among different functionals. For this purpose, we consider the revised PBE GGA for solids, namely the PBEsol^{34} functional, and one of the most widely used functionals for the local density approximation (LDA), i.e. the PZ^{35} functional. By testing some elemental crystals for the GBRV library (see Fig. 7 for the case of Ga and the Supplementary Figures 8 and 9 for a few more systems), we find that the convergence patterns turn out to be very similar if the PSPs are generated with the same atomic parameters (such as electronic configuration, cutoff radii, etc.), thus showing a good transferability of the convergence tests among different local and semi-local functionals. However, performing consistent tests for the transferability of more complex and nonlocal functionals, such as SCAN^{36} (meta-GGA) or HSE^{37} (exact-exchange), is less straightforward because, as of today, no PSP libraries for these kind of functionals exist. Indeed, in these cases, the common approach followed in all PSP DFT codes is to use PSPs generated with local or semi-local functionals, e.g. PBE, and to then “switch” to the complex functional, e.g. HSE, when performing the PSP DFT calculation including the valence electrons only. Tests performed following this approach, although useful, would not ensure the transferability of the functional under investigation and more extensive and consistent studies on the subject are therefore left to future work, in particular when HSE or SCAN PSP libraries, to only cite a few possible examples, will be available.

### SSSP libraries

Table 4 and Table 5 show the two SSSP libraries, efficiency and precision (version 1.1), selected according to the SSSP selection criteria specified above. The suggested wavefunction cutoffs (in Ry) and the duals are also indicated for each PSP chosen. The SSSP periodic table with all PSPs, wavefunction cutoffs and duals is also accesible interactively online on the Materials Cloud platform (https://www.materialscloud.org/) (see Fig. 8). The average suggested wavefunction cutoffs of the SSSP efficiency and SSSP precision over all the 85 elements tested are 44 and 56 Ry, respectively. A dual of 8 has been used for all PSPs except norm-conserving ones where a dual of 4 is used, and iron and manganese for which a dual of 12 is suggested.

The SSSP efficiency and SSSP precision have small average Δ-factors of 0.44 and 0.33 meV, respectively (where the average is performed over all elements tested excluding the 15 rare-earth nitrides, following the recipe of ref. ^{1}).

The SSSP libraries have already proven to be a reliable tool for a number of computational studies: for instance the beta version (called version 0.7) of the SSSP libraries have enabled the high-throughput computational exfoliation of two-dimensional materials^{38} and have supported the combined experimental and theoretical study of catalysts for oxygen evolution reaction.^{39}

On a more general level, apart from the SSSP testing protocol and libraries, our work provides a database of verification data and convergence tests that facilitates the optimal choice of PSPs and wavefunction cutoffs for custom applications. For example, some physical properties may be implemented only for some PSP types (typically only NC) or some applications may require convergence of just a subset of the quantities that we consider in the SSSP testing protocol. By a look at our plots and data, see for instance the condensed plot for palladium shown in Fig. 2, a user can quickly select the optimal PSP and wavefunction cutoff tailored for the specific application.

In summary, we propose an extensive testing protocol for PSPs to investigate precision and performance of several NC, US and PAW PSP libraries that are publicly available. We incorporate in the SSSP protocol a verification part, based on the Δ-factor, and an efficiency part, based on the plane-wave cutoff convergence tests for phonon frequencies, cohesive energies, pressures, and band structures. Leveraging the SSSP protocol, we identify two optimal PSP libraries, named SSSP efficiency and SSSP precision (version 1.1), that provide thoroughly tested and precise PSPs for 85 elements of the periodic table, selected from publicly available PSP libraries,^{9,10,11,12,13,14,15} for which the original authors should be acknowledged. Our effort not only is particularly relevant for high-throughput computational materials screening, where the right compromise between precision and computational cost is essential, but it substantially contributes to set high the bar of the quality of PSP calculations of solid-state materials. Building on the invaluable work behind all the PSP libraries we considered, we provide a systematic survey of PSP quality across multiple physical properties and multiple libraries and techniques (NC, US and PAW). Our work shows how the assessment of both precision and efficiency of pseudopotentials is strongly property-dependent and requires a multi-dimensional quality gauge, pointing to the need of a verification standard in the computational solid-state community. Given the importance of PSP calculations in modern materials science, nanotechnology, chemistry and physics, our findings call for more verification efforts aimed at increasing precision and efficiency of computed quantities that are routinely used to discuss novel physics, to help interpreting experiments or even to discover and design novel materials. In particular, we hope that this work will stimulate further investigations in the all-electron community, that ideally would provide more reference data for an heterogeneous set of properties elaborating on our discussion.

## Methods

All the calculations needed for this work (more than 50,000 DFT and DFPT calculations) were performed with the goal to ensure reproducibility of all the data obtained, compliant with the FAIR guiding principles for data management.^{40} This is the reason why we used AiiDA,^{5} an open-source Python infrastructure for computational science, that is specifically designed to track the provenance of data and calculations and that allow the user to implement workflows that can run complex sequences of calculations. It is therefore particularly suited for high-throughput studies, such as the deployment of the SSSP testing protocol, where a large number of simulations are involved.

In practice AiiDA can prepare and submit calculations (usually to an HPC cluster) and then retrieve and store the results inside a database, all automatically. The database can be subsequently queried by the user to extract data or other useful informations.

The complete SSSP testing protocol is implemented as an AiiDA workflow, called SsspWorkflow, that can run all the convergence tests and the Δ-factor verification test. The SsspWorkflow is built on top of the PwWorkflow, a very robust lower-level workflow in charge of handling all the QE simulations and that can restart calculations in case of standard QE errors or, for example, if the user-specified wall time is too small.

The SsspWorkflow allows a generic user to perform all the calculations required by the SSSP protocol in a completely automatic way. With this tool, other and new pseudopotential libraries could be easily tested in the future in order to update the subsequent versions of the SSSP libraries with more precise and efficient PSPs.

## Data availability

All the data produced in this work is freely available on the Materials Cloud online platform, https://www.materialscloud.org/, where the user can interactively browse the results and explore the data provenance. The full database with all the data can also be downloaded.^{41}

## Additional information

**Publisher’s note:** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## References

- 1.
Lejaeghere, K. et al. Reproducibility in density functional theory calculations of solids.

*Science***351**, aad3000 (2016). - 2.
Lejaeghere, K., Van Speybroeck, V., Van Oost, G. & Cottenier, S. Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals.

*Crit. Rev. Solid State Mater. Sci.***39**, 1–24 (2014). - 3.
Perdew J., Burke K. & Ernzerhof M. Generalized gradient approximation made simple.

*Phys. Rev. Lett.***77**, 3865–3868 (1996). - 4.
Giannozzi, P. et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials.

*J. Phys.: Condens. Matter***21**, 395502 (2009). - 5.
Pizzi, G., Cepellotti, A., Sabatini, R., Marzari, N. & Kozinsky, B. AiiDA: automated interactive infrastructure and database for computational science.

*Comput. Mater. Sci.***111**, 218–230 (2016). - 6.
Hamann, D., Schlüter, M. & Chiang, C. Norm-conserving pseudopotentials.

*Phys. Rev. Lett.***43**, 1494–1497 (1979). - 7.
Vanderbilt, D. Soft self-consistent pseudopotentials in a generalized eigenvalue formalism.

*Phys. Rev. B***41**, 7892–7895 (1990). - 8.
Blöchl, P. E. Projector augmented-wave method.

*Phys. Rev. B***50**, 17953–17979 (1994). - 9.
Kucukbenli, E. et al. Projector augmented-wave and all-electron calculations across the periodic table: a comparison of structural and energetic properties. Preprint at

*arXiv:1404.3015*(2014). - 10.
Dal Corso, A. Pseudopotentials periodic table: from H to Pu.

*Comput. Mater. Sci.***95**, 337–350 (2014). - 11.
Topsakal, M. & Wentzcovitch, R. Accurate projected augmented wave (PAW) datasets for rare-earth elements (RE = La–Lu).

*Comput. Mater. Sci.***95**, 263–270 (2014). - 12.
Garrity, K. F., Bennett, J. W., Rabe, K. M. & Vanderbilt, D. Pseudopotentials for high-throughput DFT calculations.

*Comput. Mater. Sci.***81**, 446–452 (2014). - 13.
Schlipf, M. & Gygi, F. Optimization algorithm for the generation of ONCV pseudopotentials.

*Comput. Phys. Commun.***196**, 36–44 (2015). - 14.
van Setten, M. et al. The pseudodojo: training and grading a 85 element optimized norm-conserving pseudopotential table.

*Comput. Phys. Commun.***226**, 39–54 (2018). - 15.
Willand, A. et al. Norm-conserving pseudopotentials with chemical accuracy compared to all-electron calculations.

*J. Chem. Phys.***138**, 104109 (2013). - 16.
Jollet, F., Torrent, M. & Holzwarth, N. Generation of projector augmented-wave atomic data: a 71 element validated table in the xml format.

*Comput. Phys. Commun.***185**, 1246–1254 (2014). - 17.
Blaha, P., Schwarz, K., Sorantin, P. & Trickey, S. B. Full-potential, linearized augmented plane wave programs for crystalline systems.

*Comput. Phys. Commun.***59**, 399–415 (1990). - 18.
Monkhorst, H. J. & Pack, J. D. Special points for Brillouin-zone integrations.

*Phys. Rev. B***13**, 5188–5192 (1976). - 19.
Marzari, N., Vanderbilt, D., De Vita, A. & Payne, M. C. Thermal contraction and disordering of the Al(110) surface.

*Phys. Rev. Lett.***82**, 3296–3299 (1999). - 20.
Dragoni, D., Ceresoli, D. & Marzari, N. Thermoelastic properties of α-iron from first-principles.

*Phys. Rev. B***91**, 104105 (2015). - 21.
Hybertsen, M. S. & Louie, S. G. Electron correlation in semiconductors and insulators: band gaps and quasiparticle energies.

*Phys. Rev. B***34**, 5390–5413 (1986). - 22.
Onida, G., Reining, L. & Rubio, A. Electronic excitations: density-functional versus many-body Green’s-function approaches.

*Rev. Mod. Phys.***74**, 601–659 (2002). - 23.
Reining, L. The GW approximation: content, successes and limitations.

*Wiley Interdiscip. Rev.: Comput. Mol. Sci.***8**, e1344 (2018). - 24.
Strinati, G. Application of the Green’s functions method to the study of the optical properties of semiconductors.

*Riv. Nuovo Cim.***11**, 1–86 (1988). - 25.
Georges, A., Kotliar, G., Krauth, W. & Rozenberg, M. J. Dynamical mean-field theory of strongly correlated fermion systems and the limit of infinite dimensions.

*Rev. Mod. Phys.***68**, 13–125 (1996). - 26.
Huhn, W. P. & Blum, V. One-hundred-three compound band-structure benchmark of post-self-consistent spin-orbit coupling treatments in density functional theory.

*Phys. Rev. Mater.***1**, 033803 (2017). - 27.
Hinuma, Y., Pizzi, G., Kumagai, Y., Oba, F. & Tanaka, I. Band structure diagram paths based on crystallography.

*Comput. Mater. Sci.***128**, 140–184 (2017). - 28.
Setyawan, W. & Curtarolo, S. High-throughput electronic band structure calculations: challenges and tools.

*Comput. Mater. Sci.***49**, 299–312 (2010). - 29.
Xu, S.-Y. et al. Discovery of a Weyl fermion semimetal and topological Fermi arcs.

*Science***349**, 613–617 (2015). - 30.
Soluyanov, A. A. et al. Type-II Weyl semimetals.

*Nature***527**, 495–498 (2015). - 31.
Prodan, E. & Kohn, W. Nearsightedness of electronic matter.

*Proc. Natl Acad. Sci. USA***102**, 11635–11638 (2005). - 32.
Marzari, N., Mostofi, A. A., Yates, J. R., Souza, I. & Vanderbilt, D. Maximally localized Wannier functions: theory and applications.

*Rev. Mod. Phys.***84**, 1419–1475 (2012). - 33.
Gonze, X., Stumpf, R. & Scheffler, M. Analysis of separable potentials.

*Phys. Rev. B***44**, 8503–8513 (1991). - 34.
Perdew, J. P. et al. Restoring the density-gradient expansion for exchange in solids and surfaces.

*Phys. Rev. Lett.***100**, 136406 (2008). - 35.
Perdew, J. P. & Zunger, A. Self-interaction correction to density-functional approximations for many-electron systems.

*Phys. Rev. B***23**, 5048–5079 (1981). - 36.
Sun, J., Ruzsinszky, A. & Perdew, J. P. Strongly constrained and appropriately normed semilocal density functional.

*Phys. Rev. Lett.***115**, 036402 (2015). - 37.
Heyd, J., Scuseria, G. E. & Ernzerhof, M. Hybrid functionals based on a screened Coulomb potential.

*J. Chem. Phys.***118**, 8207–8215 (2003). - 38.
Mounet, N. et al. Two-dimensional materials from high-throughput computational exfoliation of experimentally known compounds.

*Nat. Nanotechnol.***13**, 246–252 (2018). - 39.
Lebedev, D. et al. Highly active and stable iridium pyrochlores for oxygen evolution reaction.

*Chem. Mater.***29**, 5182–5191 (2017). - 40.
Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship.

*Sci. Data***3**, 160018 (2016). - 41.
Prandini, G., Marrazzo, A., Castelli, I. E., Mounet, N. & Marzari, N. A Standard Solid State Pseudopotentials (SSSP) library optimized for precision and efficiency (Version 1.1, data download).

*Materials Cloud Archive*(2018); https://archive.materialscloud.org/2018.0001/v3.

## Acknowledgements

The authors warmly thank Fernando Gargiulo, Snehal Waychal and Elsa Passaro for assistance with the SSSP section of the Materials Cloud web platform, Giovanni Pizzi and Andrea Cepellotti for the help in developing AiiDA workflows, Marco Gibertini for numerous discussions and for pointing out several useful references, Nicolas G. Hörmann for further testing on the SSSP pseudopotential libraries, Stefaan Cottenier for critical reading of the manuscript. This work has been supported by NCCR MARVEL and by H2020 CoE MaX, computing time has been provided by the Swiss National Supercomputing Centre (CSCS) and by PRACE (Project Ids 2016153543 and 2016163963). We also thank Sadas Shankar and Intel Corporation for early support to this project, through their seed funding on “Validated pseudopotentials for electronic-structure simulations” (2009–11).

## Author information

### Author notes

These authors contributed equally: Gianluca Prandini, Antimo Marrazzo, Ivano E. Castelli.

### Affiliations

#### Theory and Simulation of Materials (THEOS) and National Centre for Computational Design and Discovery of Novel Materials (MARVEL), École Polytechnique Fédérale de Lausanne, 1015, Lausanne, Switzerland

- Gianluca Prandini
- , Antimo Marrazzo
- , Ivano E. Castelli
- , Nicolas Mounet
- & Nicola Marzari

#### Department of Energy Conversion and Storage, Technical University of Denmark, DK-2800 Kgs., Lyngby, Denmark

- Ivano E. Castelli

### Authors

### Search for Gianluca Prandini in:

### Search for Antimo Marrazzo in:

### Search for Ivano E. Castelli in:

### Search for Nicolas Mounet in:

### Search for Nicola Marzari in:

### Contributions

N.Ma. and I.C. designed the study; G.P., A.M., I.C. and N.Mo. developed the workflows and performed the calculations; G.P., A.M. and I.C. wrote the manuscript, and all authors discussed the protocol, analysed the data, and commented on the manuscript. G.P., A.M. and I.C. contributed equally to this work.

### Competing interests

The authors declare no competing interests.

### Corresponding author

Correspondence to Nicola Marzari.

## Electronic supplementary material

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.