Nematic electronic phases with vestigial order1,2 are known from iron-based superconductors and cuprates, where it has been suggested that the nematic phase and the nearby spin- and charge-density wave states are not independent competing but intertwined electronic phases1,2,3,4,5,6,7,8,9,10. The density-wave states are the primary electronic phases and characterized by a multi-component order parameter. The nematic phase is a fluctuation-driven phase and characterized by a composite order parameter. Then the spin- or charge density-wave order melts partially, but leaves an Ising, i.e., Z2-nematic state as a vestige. Vestigial order whose primary order is superconductivity has not been observed. Such partially molten superconductivity requires a material with unconventional, multi-component order parameter, and strong pairing fluctuations.

When the topological insulator Bi2Se3 is doped with electrons, e.g., by intercalation of Cu, Sr, Nb, or other metal ions in its layered structure, a superconducting state is formed11,12. The presence of a strong spin-orbit coupling, which also manifests itself in a topological surface state of the parent insulator13, led to the proposal of unconventional pairing with an odd-parity symmetry and topological superconductivity14. The low carrier concentration, the layered structure, and the low ratio ξ/λF of the superconducting coherence length and the Fermi wavelength11,12, strongly enhance fluctuation effects. In addition, numerous experiments have shown that the superconducting state is accompanied by a spontaneous breaking of rotational symmetry with a pronounced twofold anisotropy within the Bi2Se3 basal plane15,16,17,18,19,20,21,22; see Ref. 23 for a recent review. The twofold symmetry can be observed in field-angle resolved experiments where a magnetic field is rotated in the plane with respect to the crystalline axes and the corresponding physical quantity (e.g., spin susceptibility, specific heat, magneto-resistance, upper critical field, magnetization, magnetic torque) is represented as a function of angle15,16,17,18,19,20,21,22,23. This behavior directly reflects the anisotropy of the superconducting state. Thus, doped Bi2Se3 is an unconventional nematic superconductor with a pairing wave function in either the two-component Eu or Eg point group representation, the only pairing states that spontaneously break the trifold crystal symmetry within the basal plane. The temperature dependence of the penetration depth of Ref. 24 supports point nodes, consistent with Eu odd-parity pairing.

In this article we report on high-resolution thermal expansion experiments on a superconducting mono-crystalline Nb-doped Bi2Se3 sample in combination with electrical transport, DC magnetization, and specific heat data demonstrating a Z3-vestigial nematic phase with enhanced superconducting fluctuations. We have measured the linear thermal expansion in three different crystalline directions in the Bi2Se3 basal plane and observed a strong anisotropic expansion occurring at a temperature of ~0.5 K above the superconducting transition. Our high-resolution magnetization, electrical resistivity, and specific heat data—after zooming near Tc—show that an anomaly with increasing superconducting fluctuations occurs at the nematic transition. As we will explain below, these observations are perfectly consistent with a vestigial nematic phase of symmetry-breaking pairing fluctuations, recently predicted in Ref. 25. This observation of a genuine symmetry breaking of pairing fluctuations above Tc is qualitatively distinct from the gradual onset of order-parameter fluctuations in the disordered phase26 or the crossover to Bose–Einstein condensation of pairs27. It corresponds to a sharply defined state of matter that might, e.g., undergo a separate quantum phase transition when a magnetic field is applied in the plane at low temperatures. Qualitatively similar results are obtained both on another Nb-doped Bi2Se3 single crystal and on a Cu-doped Bi2Se3 single crystal, thus, demonstrating the reproducibility and universality of the observed features.


Magnetoresistance and nematicity

The advantage of the Nb-doped Bi2Se3 system is that single crystals with a high superconducting volume fraction and a complete zero resistance can be found, as the results presented here show. All our bulk thermodynamic data (thermal expansion, magnetization, and specific heat) show relatively large anomalies at the superconducting transition.

Figure 1a shows magnetoresistance data recorded at 0.35 K with the magnetic field applied strictly parallel to the Bi2Se3 basal plane for different directions in the plane with respect to the trigonal crystalline axes. We determine the approximate Hc2 values from the fields in which 75% of the normal state resistance is reached and plot these values in Fig. 1b in a polar diagram as a function of angle ϕ. A significant angular variation of Hc2 can be observed, ranging from 0.43 T at 90° where Hc2 is minimal to 1.42 T at 0° with a clear maximum of Hc2. The data show a pronounced twofold symmetry, at odds with the trifold crystalline symmetry. This is the characteristic property of nematic superconductivity in NbxBi2Se319,20, also known from CuxBi2Se315,16,22 and SrxBi2Se317,18,21. A fit with a theoretical model for nematic SC of the form20

$$H_{{\mathrm{c}}2}\left( \phi \right) = \frac{{H_{{\mathrm{c}}2}(0)}}{{\sqrt {{\mathrm{cos}}^2\phi + {\mathrm{\Gamma }}^2{\mathrm{sin}}^2\phi } }},$$

yields \(\Gamma \approx 3.32{\mathrm{\Omega }}\) and Hc2(0) = 1.42 T as a measure of the anisotropy in the basal plane. It should be noted that the normal state resistance well above the upper critical field has no variation for the different orientations of the magnetic field in the plane, indicating an isotropic normal state within the trigonal basal plane. We have previously found that the orientation of this nematic superconducting order parameter for this sample always appears to be pinned along the same of the three equivalent crystal directions in the Bi2Se3 basal plane, even if the sample is warmed to room temperature between different experiments19. The origin of this preference for a particular direction is unknown, but likely associated with microscopic details of the sample morphology, such as internal strain or microcracks (see Supplementary Discussion and Supplementary Fig. 6 for more details).

Fig. 1: Dependence of the upper critical field Hc2 of Nb0.25Bi2Se3 on the angle ϕ in the plane.
figure 1

a Field-angle-resolved magnetoresistance data taken at 350 mK (Sample 1) for various alignments of the magnetic field in the Bi2Se3 basal plane. The additional circles mark the characteristic fields in which the magnetoresistance reaches 75% of the normal state value. The data were shifted vertically by −0.4 mΩ cm for better clarity, except for the ϕ = 98° data. b Polar plot of characteristic fields where the magnetoresistance reaches 75% of the normal state value, depending on the field direction in the Bi2Se3 basal plane. Only the full squares are real data, while the open squares are the same data shifted by 180° to better illustrate the full angular dependence. In the center, the corresponding crystal structure is added. The lines are theoretical expectations of a superconductor with trigonal symmetry without (green) and with (orange) vestigial nematic order.

Thermal expansion and vestigial order

Through thermal expansion experiments, we have a highly sensitive bulk thermodynamic probe that is not only sensitive to the anharmonicity of phononic contributions but also to electronic degrees of freedom including nematic and superconducting order. We focus here on \({\mathrm{\Delta }}L(T,H)|_\mu /L_0\) measured along different directions μ in the Bi2Se3 basal plane, which directly represents the change in length ΔL of the sample as a function of temperature or magnetic field, normalized to the length L0 at ambient temperature. This quantity is directly related to the linear thermal expansion coefficient αμ (T,H) = 1/L0 dLμ(T)/dT. Figure 2a shows the linear thermal expansion ΔL|μ/L0 measured along the three directions within the Bi2Se3 basal plane of 90°, 155°, and 215°. All data fall perfectly on each other in the normal state, but begin to deviate gradually from each other below 3.8 K. In the following, we will refer to this characteristic temperature as Tnem, because here the onset of a twofold crystalline distortion and thus nematicity occurs. The twofold crystalline distortion with a relative length change ΔL/L0 = 2 × 10−7 amounts to a distortion of less than 0.1 femtometers within the unit cell. Still, these minute changes smaller than the size of the proton, are clearly resolvable in our measurements. The distortion is correlated with the upper critical field, with a small negative length change along 90° where the Hc2 minimum occurs (Fig. 1b), and large positive anomalies at 155° and 215°, both near the mean Hc2 value. At 3.25 K, much smaller anomalies are visible that can be identified as the superconducting transition, as the comparison with the specific heat (shown in the same graph) reveals. In the specific heat, the anomaly at Tnem is obscured by the phonon background, but becomes visible in the temperature derivative of C/T (Fig. 2b), where a small step-shaped anomaly occurs. The anomalies in ∆L/L0 at Tnem show up as a somewhat broadened step. As a first-order derivative of the free energy, a step-like transition in ∆L/L0 is the characteristic signature of a first-order transition, while a second-order transition would appear as a kink. In distinction, the superconducting transition at Tc remains the standard second-order transition, as evidenced by the jump in the specific heat. Figure 2c shows the Meissner signal in the zero-field cooled and field-cooled DC magnetization, which agrees with Tc ≈ 3.25 K obtained from the specific heat. We also show the same data, but with a magnification of 105, to illustrate that an enhanced diamagnetic response, signaling superconducting fluctuations, already sets in at Tnem. The electrical resistivity in Fig. 2d shows a similar trend with a drop in resistivity consistent with paraconductivity, i.e., superconducting fluctuations, well above the main transition.

Fig. 2: Thermal expansion, specific heat, and DC magnetization of Nb0.25Bi2Se3 (Sample 1).
figure 2

a Linear thermal expansion ΔL(T)/L0 (units on the left axis) measured in three directions in the Bi2Se3 basal plane corresponding to 90°, 155°, and 215°, together with the specific heat C/T for comparison (units on the right axis). C/T shows that the superconducting transition is at ~Tc = 3.25 K, with small kink-like anomalies occurring in the thermal expansion. A pronounced anisotropy appears in ΔL(T)/L0 at a higher temperature below Tnem = 3.8 K. b First order derivative d(C/T)/dT of specific heat. A linear background fitted in the range above 4 K was subtracted for reasons of clarity. The large dip centered at ~3.25 K marks Tc, while at Tnem a tiny step-shaped anomaly is visible. c DC magnetization M showing the total Meissner signal and a magnification of 105 to demonstrate that the onset of superconducting fluctuations is at ~3.8 K. d Electrical resistivity ρ showing the main superconducting transition and a magnification, which shows that the first drop in resistance due to fluctuations occurs at ~3.8 K.

Our X-ray diffraction results (see section “Methods”) show that the doped Bi2Se3 phase of the R-3m space group is the majority phase responsible for Tc at 3.25 K. As minority phases Bi2Se3 of space group P-3m1 and NbBiSe3 were found. The latter occupy far too little volume to explain such large anomalies in thermal expansion. From this, we conclude that the observed crystalline distortion below Tnem = 3.8 K is caused by a transition separate from the main superconducting transition, but linked to the occurrence of superconducting fluctuations, which cause a weak Meissner effect and decrease in resistivity below this temperature.

Two stage transition

Our data thus show that nematic superconductivity occurs in the form of a two-stage transition, see also Fig. 3 for an illustration. The distortion forms near the higher onset temperature Tnem, where superconducting fluctuations in the magnetization and the first-order derivative of the specific heat are visible. The superconducting transition occurs at a lower temperature Tc, which corresponds to the formation of a global phase-coherent superconducting state. The signs and magnitudes of the length changes for the three measuring directions are consistent with the indicated distortions, see “Methods” section.

Fig. 3: Lattice distortion and real space image of the superconducting order parameter.
figure 3

Purple dashed lines below Tnem indicate the high-temperature atomic positions. The lattice distortion is strongly exaggerated. Arrows in the nematic phase indicate the directions of the thermal expansion measurements. In the latter, strong superconducting fluctuations break the discrete lattice symmetry without broken U(1) symmetry and superconducting coherence. A globally coherent superconducting state only sets in at the superconducting transition temperature Tc.

It could be argued that thermal expansion reveals a separate structural transition at Tnem that has nothing to do with superconductivity28. Such a sequence of independent or competing transitions would be allowed within the Landau theory of phase transitions. Obviously, the anomaly in the diamagnetic response at Tnem is already strong evidence that this is not the case. Furthermore, in Fig. 4 we show magnetostriction data for the 155° direction, along which we observed the greatest change in length in Fig. 2a. Here, ∆L/L0 was measured at a fixed temperature of 350 mK as a function of the magnetic field. A broad step-like transition occurs with a total length change of ∆L/L0 ≈ 0.22 × 10−7 with onset at ~1.0 T. Figure 1 shows that the resistively determined Hc2 for this direction occurs at 0.7 T. Magnetostriction shows that the crystalline distortion is constant up to this field where a kink occurs, then it is gradually removed up to 1.0 T. The overall anomaly represents a broad step in ∆L/L0, which is the expected characteristics of a first-order transition. A small hysteresis can be seen in the data measured upon sweeping the field up and down. Experimental artifacts as source for the hysteresis have been carefully excluded, especially since the field scanning speed was kept very slow at 0.02 T/min, which typically does not cause any hysteresis-like effects in reversible samples. Given the layered structure of the sample with the field aligned in parallel, this hysteresis is most likely a consequence of flux pinning effects, which are typically strong at such a low temperature. Therefore, the hysteresis should not necessarily be regarded as evidence of a first-order Hc2 nature. The field-induced length change at low temperature shown in Fig. 4 corresponds to the temperature-induced length change at zero field shown in Fig. 2a. This observation provides further evidence that the nematic distortion is closely linked to the superconducting state, with a separate nematic transition Hnem occurring above the main superconducting Hc2 transition, as shown in Fig. 4.

Fig. 4: Magnetostriction ΔL(T)/L0 as a function of the magnetic field.
figure 4

The data (Sample 1) was measured along the 155° direction at a fixed temperature of 350 mK. The crystalline distortion is removed at Hnem = 1 T, well above the resistively determined Hc2, which coincides with the kink at 0.7 T for this field orientation (marked as Hc2). The magnetic field was applied parallel to the measured sample direction. A weak linear normal state background was subtracted for clarity. The inset shows the magnetic phase diagram.

Further data on thermal expansion measured on a second sample of the same batch are shown in Supplementary Figs. 1 and 2 and are discussed in Supplementary Note 1. They show a similar behavior, although the crystalline distortion occurring below Tnem is weaker due to a multi-domain structure. For this sample we have also measured thermal expansion in fixed magnetic fields, and it can be seen that Tnem is suppressed by the magnetic field together with Tc, further confirming that the two transitions are closely related. We also show similar data of a Cu0.2Bi2Se3 single crystalline sample (Sample 3) in Supplementary Fig. 35 and the data are discussed in Supplementary Note 2.


The two thermodynamic properties, the specific heat and the linear thermal expansion coefficient αμ(T) = 1/L0 dLμ(T)/dT are closely related in the vicinity of a phase transition through the Clausius Clapeyron (2) and Ehrenfest (3) relation for first and second order phase transitions, respectively. The proportionality is the uniaxial pressure dependence dTc/dpμ of the transition (Tc is the critical temperature at which the phase transition occurs, pμ is the uniaxial pressure applied along a certain crystalline direction μ, Vmol is the molar volume, ΔS is the jump in entropy at a first order transition and ΔCp is the jump in the specific heat at constant pressure at a second order phase transition).

$$\frac{{{\mathrm{d}}T_{\mathrm{c}}}}{{{\mathrm{d}}p_\mu }} = \frac{{\left( {\frac{{{\mathrm{\Delta }}L}}{{L_0}}} \right)_\mu \cdot V_{{\mathrm{mol}}}}}{{{\mathrm{\Delta }}S}},$$
$$\frac{{{\mathrm{d}}T_{\mathrm{c}}}}{{{\mathrm{d}}p_\mu }} = \frac{{{\mathrm{\Delta }}\alpha _\mu \cdot T_{{\mathrm{c}} \cdot }V_{{\mathrm{mol}}}}}{{{\mathrm{\Delta }}C_{\mathrm{p}}}}.$$

A large anomaly in thermal expansion and a small anomaly in specific heat means in both cases that Tnem is strongly dependent on uniaxial pressure and the electronic nematic order is strongly coupled to the crystalline lattice. The strong crystalline distortion observed here using linear thermal expansion therefore means that the nematic transition is strongly dependent on pressure or strain. Such a behavior can also be observed, for example, in iron based superconductors, where a nematic transition occurs in the vicinity to a spin density wave transition and causes large anomalies in thermal expansion29.

Our findings can be explained in terms of vestigial order due to superconducting fluctuations. In fact, recently it has been suggested that such vestigial order should emerge from the superconducting phase in doped Bi2Se325. On the one hand, the superconducting order parameter of either the Eg or the Eu representation has two components14,30,31,32

$$\left( {{\mathrm{\Delta }}_{\mathrm{x}},{\mathrm{\Delta }}_{\mathrm{y}}} \right) = {\mathrm{\Delta }}_0e^{i\varphi }\left( {{\mathrm{cos}}\theta ,{\mathrm{sin}}\theta } \right),$$

that are characterized by the overall amplitude Δ0, the global U(1) phase φ and three distinct values of the angle \(\theta = \left\{ {\frac{\pi }{6},\frac{\pi }{2},\frac{{5\pi }}{6}} \right\}\) that select a specific crystalline axis. Superconducting fluctuations will then induce a phase transition to a vestigial nematic state at a temperature Tnem above Tc. While superconductivity is signaled by a finite expectation value of Δx and/or Δy, the nematic phase is characterized by a finite expectation value of the composite order parameter

$${\mathbf{Q}} = \left( {\begin{array}{*{20}{c}} {\left| {{\mathrm{\Delta }}_x} \right|^2 - \left| {{\mathrm{\Delta }}_y} \right|^2} & {{\mathrm{\Delta }}_x^ \ast {\mathrm{\Delta }}_y + {\mathrm{\Delta }}_y^ \ast {\mathrm{\Delta }}_x} \\ {{\mathrm{\Delta }}_x^ \ast {\mathrm{\Delta }}_y + {\mathrm{\Delta }}_y^ \ast {\mathrm{\Delta }}_x} & {\left| {{\mathrm{\Delta }}_y} \right|^2 - \left| {{\mathrm{\Delta }}_x} \right|^2} \end{array}} \right).$$

Upon increasing the temperature, superconducting fluctuations continue to break the rotational symmetry, even after restoration of global U(1) symmetry at Tc. The composite order parameter Qμν is made up of combinations of the superconducting order parameter, similar to charge-4e superconductivity proposed within the context of pair-density wave order in cuprate superconductors33,34 or proton–electron superconducting condensate in liquid hydrogen35. As a traceless second-rank tensor, \(\langle{\mathbf{Q}}_{\mu {\upnu}}\rangle = Q_0\left( {n_\mu n_{\upnu} - \frac{1}{2}\delta _{\mu {\upnu}}} \right)\) behaves, however, like a nematic order parameter with director \({\mathbf{n}} = \left( {\cos \theta ,\sin \theta } \right)\)36 and strongly couples to the strain tensor εμν via \(\kappa {\mathrm{tr}}({\mathbf{Q\varepsilon }})\) with nemato-elastic coupling constant κ. A nonzero Q0 then induces a lattice distortion εμν\({\propto} \, {\kappa} {\mathbf{Q}}_{\mu {\upnu}}\), see Fig. 3. Thus, the lattice can be utilized to detect this unconventional electronic order. The point group analysis further yields a first-order transition at Tnem into a state with Q0 ≠ 0, since it is in the three-state Potts model, i.e., the Z3 universality class. The superconducting transition continues to be of second order, all in agreement with our experimental findings. In Fig. 5a we show the nematic (Q0) and superconducting (Δ0) order parameters and in Fig. 5b the diamagnetic susceptibility obtained within the theory of Ref. 25. The susceptibility is compared with the data of Fig. 2c, where the logarithmic axis is used to illustrate the rapid growth of diamagnetic fluctuations below Tnem. While the in-plane anisotropy of the susceptibility is only finite below the nematic transition, the crystal symmetry allows χzz to be distinct already above Tnem. The magnitude of the out-of-plane anisotropy is determined by the ratio of the electron velocities in the corresponding directions. The anisotropy of \(H_{{\mathrm{c}}2}\left( \phi \right)\) shown in Fig. 1b (orange line) was also obtained within the same theory and is compared with the behavior without nematic order (Q0 = 0) where Hc2 should have sixfold symmetry24,25. Without nematic phase above Tc, the superconducting order parameter directly at \(H_{{\mathrm{c}}2}\left( \phi \right)\) is infinitesimal and no twofold rotational symmetry breaking should be visible, in clear contrast to experimental observations. In Ref. 31, the twofold symmetric behavior of \(H_{{\mathrm{c}}2}\left( \phi \right)\) only occurred after an additional symmetry-breaking strain was added. Vestigial nematic order offers a natural explanation for this strain field.

Fig. 5: Superconducting and nematic order parameters and diamagnetic susceptibility.
figure 5

a The two intertwined order parameters Δ0 (superconducting) and Q0 (nematic). b The anisotropic diamagnetic susceptibilities χxx, χyy and χzz together with the experimental magnetization data on a log-linear scale to compare with the theory.

After this work was completed, we learned about Ref. 37, where a twofold symmetry breaking above Tc is reported. Our results agree with those of Ref. 37 and make evident that the high temperature phase is separated by an actual first order transition where superconducting fluctuations are enhanced. Furthermore, Refs. 38,39 reported on the control of nematic superconductivity by uniaxial strain, which is consistent with our observation of a coupling of the nematic order parameter to the crystal lattice.

To summarize, our data demonstrate that a separate nematic transition occurs in the doped topological insulator Nb0.25Bi2Se3 at Tnem = 3.8 K, i.e., about 0.5 K above Tc, with a distinct crystalline distortion occurring in the Bi2Se3 basal plane. Tnem coincides with the onset temperature of superconducting fluctuations. The direction of the crystalline distortion is correlated with the direction of the twofold symmetry of the superconducting order parameter and is removed together with the superconductivity at or near the upper critical field Hc2. The two transitions are thus interconnected. Our observations are perfectly consistent with vestigial nematic order and a sequential restoration of U(1) and rotational symmetry. The new nematic phase is a state of matter in which Cooper pairs have lost their off-diagonal long-range order, yet fluctuate in a way that breaks the rotational symmetry of the crystalline lattice.


Sample characterization

The monocrystalline Nb0.25Bi2Se3 sample used in this study was selected because of its particularly large Tc anomalies in the specific heat, which indicates a high superconducting volume fraction, and because of its particularly large nematic in-plane Hc2 anisotropy19. Our previous work also demonstrated that it forms one large nematic domain comprising ~90% of the superconducting volume fraction in which the nematic order parameter is pinned in one crystalline direction, while the remaining 10% is due to a minority domain in which the orientation is rotated by 60°. Data from a second monocrystalline Nb0.25Bi2Se3 sample with broader transition anomalies and somewhat less nematic anisotropy are shown in Supplementary Figs. 1 and 2. We also show data of a Cu0.2Bi2Se3 single crystalline sample (Sample 3) in Supplementary Fig. 35. These samples show a qualitatively similar behavior as Sample1.

A laboratory X-ray Laue equipped with CCD camera (Photonic Science) was used to characterize the crystal quality and to determine the crystalline directions in the basal Bi2Se3 plane. The Laue images (Fig. 6) were taken on the shiny surface of the sample after cleaving off a thin layer with shooting X-ray beam along the c-axis, revealing the crystal orientation and proving the hexagonal structure. We took data on different spots on the sample surface and found that the change in the crystalline direction was less than 0.2° over a distance of 0.8 mm, proving the sufficiently good single crystalline quality of the sample.

Fig. 6: Laue X-ray diffraction image of the NbxBi2Se3 single crystal (Sample 1).
figure 6

This data allowed us to define the crystalline orientation in the Bi2Se3 basal plane.

Powder X-Ray Diffraction pattern was collected at room temperature in the Bragg-Brentano geometry using a Bruker AXS D8 Advance diffractometer equipped with a Ni-filtered Cu Kα radiation and a 1D LynxEye PSD detector (Fig. 7). The reflections were indexed to Bi2Se3 (SG R-3m; No 166) as a main phase. In addition, some minority phases were found as following: Bi2Se2 and NbBiSe3 with corresponding space groups: P-3m1 (No 164) and P212121 (No 19), respectively. This demonstrates that, while most of the Nb must be intercalated between the Bi2Se3 layers, some Nb is incorporated into the layers on the Bi sites. This agrees with literature data40. The Rietveld refinement41 of the diffraction patterns was done by the package FULLPROF SUITE42 (version July-2019) using a previously determined instrument resolution function (based on the small line width polycrystalline sample Na2Ca3Al2F14 measurements43). Refined parameters were: scale factor, zero displacement, lattice parameters, atomic positions, isotropic Debye–Waller factors, and peak shape parameters as a Thompson-Cox-Hastings pseudo-Voigt function. Determined lattice parameters of the rhombohedral Bi2Se3 are equal a = b = 4.1854(4) Å, and c = 28.4633(7) Å. Because of the habit of the powdered crystal, a preferred orientation as a March-Dollase multi-axial phenomenological model was implemented in the analysis.

Fig. 7: X-ray diffraction pattern for NbxBi2Se3 (Sample 1).
figure 7

The room temperature CuKα radiation pattern is shown by the red crosses. The black line corresponds to the best fit from the Rietveld refinement analysis. Lower vertical marks denote the Bragg peak positions of Bi2Se3 (red), Bi2Se2 (blue), and NbBiSe3 (green). The bottom, gray line represents the difference between experimental and calculated points.

Experimental techniques

The high-resolution linear thermal expansion was measured with a capacitive technique using a dilatometer, in which the sample is pressed by a fine screw mechanism against a cantilever forming one of the two plates of a capacitor. A change in the sample length leads to a change in the separation of the capacitor plates, which can be determined with a General Radio 1615A capacitance bridge in combination with a Stanford Research SR830 digital lock-in amplifier. Before the experiments we have carefully checked that the empty dilatometer does not show any anomalies in the temperature range of interest (Supplementary Fig. 7). We have measured the thermal expansion as a function of temperature in three different directions within the basal Bi2Se3 plane (90°, 155°, and 215°). 0° is the a direction normal to the mirror plane and corresponds to the magnetic-field direction in the plane providing the maximum upper critical field, while 90° corresponds to the a* direction parallel to the mirror plane. The crystalline directions have been obtained by Laue X-ray diffraction as explained above. The other directions were chosen as representative of other characteristic directions in the plane, but largely dictated by the crystal shape, which allowed a stable mounting of the sample only for certain directions. All directions were within 5° from the three different a* directions. For technical reasons, all data was taken at a slow rate of 0.02 K/min upon increasing temperature. Figure 8 shows a photograph of the mounted crystal for the three different orientations in the dilatometer. The dilatometer44 was well characterized using separate calibration measurements: e.g., a measurement with a Cu sample of 1 mm length of the same material as the dilatometer body shows only a very small temperature dependence without significant anomalies in the temperature range of interest. The absolute value of the change in length was calibrated using a 2 mm long undoped silicon sample, which gave a linear thermal expansion coefficient that agrees well with literature45 (Supplementary Fig. 8).

Fig. 8: Photographs of Sample 1 mounted in the capacitive dilatometer.
figure 8

The three panels show the NbxBi2Se3 sample mounted along the three measured directions within the Bi2Se3 basal plane. The dotted lines mark the 90° direction, which corresponds to one of the crystalline a* directions (parallel to the mirror plane). The lower row of images illustrates the corresponding crystalline orientations. The red lines mark the crystalline a directions (perpendicular to the mirror plane) as determined by Laue X-ray diffraction. The blue lines mark the measured directions.

The specific heat was measured with a home-made calorimeter, which can be used either in AC modulated temperature mode or in long relaxation mode. The long relaxation mode provides high accuracy in the absolute value of 1% precision, while the AC mode provides high relative resolution with a high density of data points of 1000 points per K. The data presented in this letter has been acquired using the AC technique, but the absolute value has been calibrated using the relaxation technique.

DC magnetization was measured using a commercial Quantum Design Vibrating Sample SQUID magnetometer and the electrical resistance was measured using a standard 4-probe technique with a Keithley 6221 AC current source combined with a SR830 digital lock-in amplifier. For the latter, a low temperature piezo rotator was used to precisely align the magnetic field along the different crystalline directions and to study the Hc2 anisotropy that reflects the nematic superconductivity. The rotator allowed milli-degree precision for relative changes in orientation. However, a systematic error of less than 5° can occur with respect to the measured crystalline axes.

Theory: q-state Potts model

A q-state Potts model describes a spin-like variable s = 1, 2, …, q that can attain q different values. In our case q = 3 and the three values label the three axes of the crystal along which one assumes a local displacement. The energy of two neighboring variables s and s’ that are the same is then lower than for distinct variables, i.e., the bond energy goes like \(- \frac{1}{2} J_0(q\delta_{s,{s^{\prime}}} - 1)\). In general, the model has to be distinguished from a q-states clock model where an angle \(\varphi \in \left[ {0,2\pi } \right]\) can take q distinct values \({\upvarphi}_{\mathrm{s}} = \frac{{2\pi s}}{q}\) and two sites interact via an energy proportional to \(- J_0{\mathrm{cos}}({\upvarphi}_{\mathrm{s}} - {\upvarphi}_{{\mathrm{s}}\prime })\). For q = 3 both models are equivalent though. Using the Potts language one can define the fraction ns of the lattice in the s-th state, which implies n1 +n2 +n3 = 1. At high temperatures one expects ms = ns \(- \frac{1}{3}\) to have zero expectation value for all s. Below the transition temperature, one of the ms becomes positive and the two others negative. Because of the condition on the sum of the ns the three ms are not independent: m1 +m2 +m3 = 0. Furthermore, the free energy of the system must be symmetric under msms. These two conditions lead to the unique free energy expansion up to quartic order

$$f = r\mathop {\sum }\limits_{s = 1}^3 m_s^2 - vm_1m_2m_3 + u\left( {\mathop {\sum }\limits_{s = 1}^3 m_s^2} \right)^2.$$

Since there are really only two independent variables one can introduce the parametrization \({\mathrm{m}}_1 = \frac{2}{{\sqrt 3 }}{\mathrm{Q}}_1\) and \({\mathrm{m}}_{2,3} = \frac{1}{2} - \frac{1}{{\sqrt 3 }}{\mathrm{Q}}_1 \pm {\mathrm{Q}}_3\). The expansion in terms of the Q1,2 leads up to constants to

$$f = r\left( {Q_1^2 + Q_2^2} \right) - v\prime Q_1\left( {Q_1^2 - 3Q_2^2} \right) + u\left( {Q_1^2 + Q_2^2} \right)^2$$

This is precisely the free energy expansion obtained in Ref. 25 in terms of the quadrupolar order parameter of Eq. (2) with \(Q_1 = \left| {{\mathrm{\Delta }}_{\mathrm{x}}} \right|^2\)\(\left| {{\mathrm{\Delta }}_{\mathrm{y}}} \right|^2\) and \(Q_2 = {\mathrm{\Delta }}_{\mathrm{x}}^ \ast {\mathrm{\Delta }}_{\mathrm{y}} + {\mathrm{\Delta }}_{\mathrm{y}}^ \ast {\mathrm{\Delta }}_{\mathrm{x}}\). This demonstrates that the problem at hand is indeed in the universality class of the q = 3 Potts or clock models.

The sketch of the distorted unit cell in Fig. 3 is deduced from the linear coupling term κtr() between the strain tensor and the composite order parameter. Thus, the nematic order parameter acts in the same way as an applied external stress field in the Eg symmetry channel and hence distorts the unit cell. Moreover, the linear coupling term does not entail a change of the unit cell volume, which is assumed to be unaltered in the following. Figure 9 shows the relative length changes in the three directions 90°, 155°, and 215° caused by an Eg unit cell deformation as a function of the lattice parameter a/a0. For the strongly exaggerated value a/a0 = 1.1 we also show the corresponding distorted hexagon in the inset, which is quantitatively similar to the distorted unit cell in Fig. 3. While the computed relative length changes qualitatively capture the measured thermal expansion behavior, the calculated magnitude of the 90o direction is slightly larger than that observed in experiment when compared with the other two directions. This could be due to the higher-order coupling that gives rise to a change of the unit cell volume at Tnem.

Fig. 9: Relative length changes in the three directions 90°, 155°, and 215°.
figure 9

The length changes are caused by an Eg unit cell deformation as a function of the lattice parameter a/a0. The inset shows a distorted hexagon for an exaggerated value of a/a0 = 1.1.