Hidden non-collinear spin-order induced topological surface states

Rare-earth monopnictides are a family of materials simultaneously displaying complex magnetism, strong electronic correlation, and topological band structure. The recently discovered emergent arc-like surface states in these materials have been attributed to the multi-wave-vector antiferromagnetic order, yet the direct experimental evidence has been elusive. Here we report observation of non-collinear antiferromagnetic order with multiple modulations using spin-polarized scanning tunneling microscopy. Moreover, we discover a hidden spin-rotation transition of single-to-multiple modulations 2 K below the Néel temperature. The hidden transition coincides with the onset of the surface states splitting observed by our angle-resolved photoemission spectroscopy measurements. Single modulation gives rise to a band inversion with induced topological surface states in a local momentum region while the full Brillouin zone carries trivial topological indices, and multiple modulation further splits the surface bands via non-collinear spin tilting, as revealed by our calculations. The direct evidence of the non-collinear spin order in NdSb not only clarifies the mechanism of the emergent topological surface states, but also opens up a new paradigm of control and manipulation of band topology with magnetism.

Rare-earth monopnictides R(Bi,Sb) (R=Ce, Nd, Sm, etc.) with a rock salt structure have been extensively studied in the past sixty years for their intricate magnetic phase diagrams [1,2,3,4].Except for the lanthanum, yttrium, and praseodymium compounds, most rare-earth monopnitides are antiferromagnets at low temperatures [1,2,3,4].Upon cooling or applying a magnetic field, many of them undergo another or even multiple antiferromagnetic transitions [5,6,7], as a consequence of competing magnetic interactions.A notable example is the "Devil's staircase" in CeSb [8] and the magnetic transitions have a significant influence on the electronic band structures [9].More recently, rare-earth monopnictides have been proposed to have topologically non-trivial band structures.A first-principle calculation predicted a systematic band inversion reduction from Ce to Yb compounds, and a topologically non-trivial to trivial transition occurs around SmSb and DyBi [10].Later angle-resolved photoemission spectroscopy (ARPES) experiments confirmed the band inversion in rare-earth monobismuthides and identified linear-dispersed Dirac-like states [11,12,13,14].Magnetotransport experiments revealed non-trivial Berry phase in SmBi [15] and SmSb [16], and possible Weyl or Dirac fermions in CeSb and NdSb when driving the samples to ferromagnetic phase with magnetic field or pressure [17,18,19].Therefore, rare-earth monopnictides are an intriguing platform to explore the interplay between magnetism, strong electronic correlation, and topological band structure.More strikingly, recent ARPES studies reveal that a number of rare-earth monopnictides, including NdBi, NdSb, and CeBi, host disconnected Fermi surface arcs emerging from an unconventional magnetic band splitting below Néel temperature (T N ) [20,21].Initial density functional theory (DFT) calculations suggest that the emergent surface states cannot be explained by the ordinary A-type antiferromagnetism observed by neutron scattering experiment [3].Later analysis suggests that multiple-wave-vector (multi-q) antiferromagnetic order [22] can reproduce the emergent surface states [23,24].So far there is no direct experimental evidence of multi-q antiferromagnetic order, which is difficult to be differentiated from multi-domain state of simple 1q order in powder neutron diffraction [3].Real-space magnetic imaging probes, such as spin-polarized scanning tunneling microscopy (SP-STM), are suitable for directly visualizing the non-collinear magnetic order [25,26] In this work, we provide real space evidence of non-collinear antiferromagnetic order in NdSb using SP-STM.We observe a checkerboard magnetic contrast on the (001) surface of NdSb that is consistent with a non-collinear spin order, which is a superposition of two A-type magnetic orders with orthogonal in-plane modulations.Furthermore, we discover a hidden spin-rotation transition (1q-2q) 2 K below the Néel ordering temperature T N that has not been reported.More interestingly, the hidden transition coincides with the onset of band splitting observed by ARPES.DFT study reveals a band inversion driven by magnetic order-induced band folding, resulting in the unconventional magnetism-induced topological surface states in the family of rare-earth monopnictides.Fig. 1a shows an STM topographic image taken at 4.5 K with a nonmagnetic tip at sample bias V bias = +1.4V.The image reveals a square lattice of Nd atoms with Nd-Nd spacing approximately 4.3 Å calculated from the Bragg peaks q lat = (1, 0) and (0, 1) in the Fourier transform map (Fig. 1c), in good agreement with prior studies [4].The bias dependence of STM topography indicates the unoccupied states are dominated by Nd orbitals while the occupied states are predominantly of Sb orbitals, which is confirmed by our DFT calculations (Extended Data Fig. 2).Interestingly, SP-STM topography in Fig. 1b reveals an additional checkerboard-like pattern at +1.4 V bias, resulting in two pairs of superlattice peaks of wave vectors q AFM = ± 1 2 , ± 1 2 as shown in Fig. 1d.The bias dependence of SP-STM topography shows that the checkerboard modulation is mainly visible in unoccupied states dominated by Nd orbitals, and gradually diminishes in occupied states with significant Sb character.The disappearance of checkerboard-like modulation is accompanied by a fractional 1  2 , 1 2 shift of the atomic lattice, presumably from Nd to Sb atoms (Extended Data Fig. 2, 3 and 4).This further confirms that the observed checkerboard pattern comes from the spin-dependent tunneling between the magnetic tip and the local spin-polarized Nd orbitals.
Close inspection of the Fourier transform map (Fig. 1d) reveals that there is an asymmetry of the magnetic superlattice peak intensity along two orthogonal directions: the intensity of q A AFM = 1 2 , 1 2 is stronger than that of q B AFM = 1 2 , − 1 2 .We also performed bias-dependent studies to confirm that asymmetry persists in the whole bias range when the magnetic modulation is visible (Extended Data Fig. 3).This magnetic asymmetry is uniform over hundreds of micrometers, indicating a global C 4 symmetry breaking.To confirm that this symmetry breaking is intrinsic, we examine the sample surface extensively over hundreds of micrometers and locate a magnetic domain boundary shown in Fig. 2c.The SP-STM image is taken at V bias = +0.3V to maximize the magnetic contrast.
The magnetic modulation asymmetry rotates 90 • across the domain boundary, as clearly demonstrated in the Fourier transform maps of two domains and the corresponding line profiles along orthogonal directions (Fig. 2b,e).This magnetic asymmetry suggests a magnetic structure are different from those proposed in prior DFT analysis where the angle between Nd spin and cubic axes is either 0 • (1q) or 45 • (simple 2q) [21,23,24].The out-of-plane spin modulation in 3q cannot be detected by SP-STM, so we focus on the in-plane multi-wave-vector modulation and do not differentiate 2q and 3q in the following text.For the 1q magnetic structure, there is no magnetic modulation on the (001) surface, thus no magnetic superlattice peak.In contrast, there is a single modulation direction on the (110) [i.e., cubic (010)] surface, resulting in one pair of magnetic superlattice peaks (Fig. 1e).For the proposed simple 2q modulation [21,23], the Nd spin moments are 45 • with respect to the cubic a, b-axes, so the 4-fold rotational symmetry (plus a fractional lattice translation) is restored.This magnetic modulation can be considered as a superposition of two orthogonal 1q modulations with equal spin components along the modulation directions, giving rise to two pairs of magnetic peaks with equal intensity on the (001) surface.Therefore, the two pairs of magnetic peaks with unequal intensity must stem from a magnetic structure with a spin configuration between 1q and the simple 2q.In other words, the magnetic structure of NdSb is a 2q modulation with broken C 4 symmetry where the angle between Nd spin moments and the cubic a, b-axes is between 0 • and 45 • .In this scenario, the spin components along q A AFM and q B AFM are proportional to their modulation amplitudes, which are proportional to the magnetic peak amplitudes.Using the ratios of superlattice peak intensities, we estimate that the Nd moment rotates θ ≈ 18 • from q A AFM .The angle is weakly temperature-dependent below 13 K (see Methods and Extended Data Fig. 6).
To understand the origin of asymmetric magnetic modulation, we perform SP-STM measurements at various temperatures to examine the temperature dependence (Fig. 3bd and Supplementary Information).At 11 K, the magnetic contrast is much weaker than that at 4.5 K, indicating a dramatic reduction of order moments.At around 14 K, the magnetic contrast is barely visible in SP-STM topography.From the Fourier transform the magnetic peaks with higher intensity (q A AFM ) remain while the ones with lower intensity (q B AFM ) disappear.Above T N (≈ 15 K), all the magnetic superlattice peaks disappear, indicating the recovery of the paramagnetic phase.Fig. 3e shows the line profile of the Fourier transform intensity along two orthogonal directions crossing q A AFM and q B AFM .
Clearly, q B AFM vanishes above 13 K while q A AFM persists to T N , indicating a 1q magnetic structure in this 2 K window.Therefore, there is a hidden 1q-2q transition at 13 K, which can be considered as anti-phase rotation of Nd spins in neighboring layers (Fig. 3a and Extended Data Fig. 8).So the 1q-2q transition is a spin-rotation transition.Just below T N , the peak intensity (q A AFM ) is weak and develops slowly.However, it grows sharply below T R accompanied by the emergence of orthogonal modulation (q B AFM ), whose intensity also increases drastically.Therefore, our SP-STM results also suggest that the magnetic order parameter is greatly enhanced below spin rotation transition T R .
The hidden spin-rotation transition provides new insight into the mechanism of the mysterious splitting of emergent surface states observed in prior ARPES studies [20,21].
In Fig. 4a, we plot the Fermi surface of NdSb measured by ARPES at 5.5 K. Consistent with previous results [21], the surface states are two-fold symmetric.In addition, our ARPES unveils a larger momentum space and the surface states also emerge at the original M -point in the Brillouin zone of the paramagnetic phase, demonstrating the band-folding caused by the magnetic modulations.Fig. 4c shows the temperature evolution of band dispersion.Besides the two split surface bands reported before [21], which we denoted as α and β, we observe a third surface band γ, which is not reported in prior studies [20,21].We extract the momentum distribution curves (MDCs) at the Fermi energy (Fig. 4e).Above T N , only a broad shoulder-like feature due to the bulk bands can be observed.Below T N , a peak appears as a signature of the emergent surface states.The intensity of the peak increases gradually upon cooling but it does not split into α and β bands until below 13 K (Fig. 4e).The γ band is almost temperature-independent.The onset temperature of band splitting agrees with the spin-rotation transition around 13 K observed in our SP-STM measurements.We note that similar behavior of peak splitting has been observed in sister compound NdBi where T N (≈ 24 K) is higher.Interestingly, the emerging surface states do not split until below 20 K [20], suggesting that there is a similar hidden spin-rotation at 20 K in NdBi.Therefore, the hidden transition is likely a universal phenomenon for the Nd monopnictides.
To understand the fundamental nature of emergent surface states and their splitting, we perform DFT calculations to understand the bulk and surface states evolution from PM to 1q and from 1q to 2q transitions.The PM phase of NdSb is topologically trivial, unlike NdBi.In contrast, the 1q phase leads to a band anti-crossing near the Fermi energy (E F ) due to folding along Γ − X in the bulk.Here, spins align along the x-direction in the 1q phase.Consequently, two surface bands emerge from the upper and lower bulk band edges involved in the anti-crossing gap and disperse in the direction of Γ − X (Fig. 4d).
The surface band splitting is small because of the small anti-crossing gap.Because NdSb exhibits multiple band crossings at different energies and momenta in the full Brillouin zone as a metal, the global topological invariant is trivial.However, this does not neces-  AFM and q B AFM highlight the magnetic Bragg peaks along two orthogonal directions.a, Line profiles along q A AFM and q B AFM .c, A spin-polarized topographic image with a magnetic domain wall highlighted by the white dashed line.Tunneling condition: V bias =0.3 V, I =0.3 nA.d, e, Similar to a and b but obtained from the squared area on the right of the domain wall in c.The insets are the corresponding FFT and the positions of the magnetic Bragg peaks are highlighted by the red (q A AFM ) and blue (q B AFM ) squares.e, The temperature-dependent line profiles along q A AFM (red) and q B AFM (blue) magnetic Bragg peaks.f, The normalized magnetic Bragg peak intensity of q A AFM and q B AFM as a function of temperature.The blue and red lines are guides to the eyes.to match the ARPES finding that the paramagnetic phase is trivial.The properties of Nd f electrons were corrected with a Hubbard U = 6.3 eV and J = 0.7 eV using Dudarev's method [32].The plane wave states were then projected onto a tight-binding basis of Nd d, f and Sb p orbitals using Wannier90 [33].Surface states were calculated using efficient, iterative Green's function methods as implemented in WannierTools [34].
NdSb single crystal growth.NdSb single crystals were synthesized by a Sn flux method [35].High-quality powders of Nd, Sb, and Sn in a ratio of 1:1:20 were put into an alumina crucible which was subsequently sealed within a vacuumed quartz tube.The prepared ampoule was heated up to 1150 • C and then slowly cooled down to 750 • C at a rate of 3 • C/h, followed by decanting of the excess liquid using a centrifuge.Shining cubic crystals were obtained after breaking the crucible.
Density of states for Nd and Sb.In Extended Data Fig. 2b, c, we show the nonspin-polarized STM image taken at the tunneling bias of +1.4 V and −1.3 V.It is evident that at +1.4V the highlighted defects are centered on the protrusions whereas at −1.3 V the depressions, indicating at +1.4V and −1.3 V different atoms (Nd or Sb) are imaged.
In Extended Data Fig. 2d We plot the ab-initio calculated partial density of states, projected on the atomic sites in the 2q configuration of NdSb.We focus on the energy window probed by STM with the Fermi level set to zero.For negative voltages, the density of states is dominated by Sb p orbitals, while for positive voltages the density of states is composed primarily of Nd d, f orbitals.This suggests that at positive tunneling biases, the bright atom-like features are Nd atoms because of their large DOS above Fermi energy, while Sb atoms are imaged at negative bias.
Estimation of Nd spin moment angle from spin-polarized STM images.In Fig. 2 and Extended Data Fig. 5, we observe a magnetic domain wall separating two magnetic domains of different magnetic contrast.The amplitude asymmetry of the magnetic peaks q A and q B in the Fourier transform switches sign across the domain wall.Therefore, we can estimate the angles between Nd spin moments and the magnetic modulation using the magnetic peak intensities of these two domains.
Extended data Fig. 7a shows the schematics of AFM modulations (q A and q B ), tip moments (green arrows) and Nd moments (orange arrows) on the Nd lattice (black dots).
On the left domain, we denote the angle between Nd moment and q A as θ, and the angle between the tip moment and q A as α.As explained in the main text, the 2q modulation can be decomposed into two unequal 1q modulations along q A and q B , whose spin moments are along the modulation wave vectors.The sum of the spin moments of these two 1q modulations gives rise to the spin moments in 2q.Without losing generality (e.g. in Extended Fig. 5), we assume modulation amplitudes S L A = S cos θ are stronger than S L B = S sin θ on the left domain, i.e., assuming S L A > S L B .Thus, θ < 45 • by definition.
The situation is the opposite on the right domain, i.e., S R A = S sin θ and S R B = S cos θ.
Therefore, the Nd moments on the left domain are more inclined to q A , and vice versa for the right domain.Therefore, the Nd moment angle θ is related to the ratio of the modulation amplitudes S A and S B along q A and q B through the following equation: In SP-STM, the spin-dependent component of tunneling current is proportional to the projection of local spin orientation onto the tip moment [36].Therefore, the measured magnetic peak amplitudes with SP-STM are determined by projections of the two 1q modulations onto the tip moment.Extended Data Fig. 7: Estimation of Nd spin moment angle.a, Schematics of tip spin moment m tip and the Nd spin moment S on the Nd lattice (black dots).S A and S B are the projection of Nd spin onto q A and q B , respectively; α is the angle between the tip moment and q A -axis, or [100]; θ is the angle between Nd spin moment and q A -axis.b, The estimated tip moment (α) and Nd spin moment (θ) angles relative to the q A -axis.

Figure 1 :
Fig.9), in accordance with ARPES findings.The surface band split in Fig.4gcorresponds to the zero temperature case.In finite temperature, the split is expected to be proportional to the size of order moments.Consistently, the observed temperature dependence of surface band split shown in Fig.4ffollows that of spin-polarized contrast shown in Fig.3f.In addition, we verify that the Fermi surface projected on the (001) surface of 2q phase with θ = 20 • is two-fold symmetric in calculations (Fig.4b), in good agreement with the ARPES results shown in Fig.4a.Therefore, the ground state of NdSb is a 2q antiferromagnetic structure with broken C 4 symmetry, in good agreement with SP-STM results.Between 13 and 15 K, only one pair of magnetic peaks with weak intensity is observed in Fourier transform maps, corresponding to 1q antiferromagnetic order with

Figure 2 :
Figure 2: Magnetic modulations across a domain boundary of NdSb.a, FFT obtained from the squared area on the left of the domain wall in c. q A AFM and q B

Figure 3 :
Figure 3: Temperature-dependence of magnetic modulations in NdSb.a, The magnetic phase diagram in NdSb.b-d, Spin-polarized topographic images taken at the same area at 11 K (b), 14 K (c) and 16 K (d).The insets are the corresponding FFT and the positions of the magnetic Bragg peaks are highlighted by the red (q A AFM ) and blue (q B AFM ) squares.e, The temperature-dependent line profiles along q A AFM (red) and q B AFM (blue) magnetic Bragg peaks.f, The normalized magnetic Bragg peak intensity of q A AFM and q B AFM as a function of temperature.The blue and red lines are guides to the eyes.

Figure 4 :
Figure 4: ARPES spectra at different temperatures and DFT calculated band structure of NdSb (001) surface.a, Fermi surface map of NbSb (001) measured at T =5.5K.The photon energy of ∼53 eV is used.The purple solid lines outline the Brillouin zone for the paramagnetic phase of NbSb.The blue dashed lines outline the folded Brillouin zone for the multi-q antiferromagnetic phase of NbSb.α, β and γ denote three emergent surface states.b, DFT calculated Fermi surface of 2q antiferromagnetic phase projected on NdSb (001) surface.c, Temperature evolution of ARPES spectra of NbSb along the ΓX direction.d DFT calculated ΓX band structure of NdSb at varying angles of Nd spin moments.e, Temperature-dependence of the MDCs at the Fermi level, as marked by a red arrow in c.The peaks of α and β surface states are fitted by the Lorentzian function.f, Temperature-dependence of the MDC peak splitting (black) and the sum of peaks area for the α and β surface states (blue).g, The calculated band splitting of α and β surface state as a function of the angle between Nd spin moments and a, b-axes.

. 1 :dExtended Data Fig. 3 :Extended Data Fig. 4 :
Preparation and demonstration of spin-polarized STM tip on FeTe.a, A 30×30 nm 2 topographic image of FeTe taken with a non-magnetic tip.V bias =+60 mV, I=1 nA.The inset is the FFT.Only two pairs of lattice Bragg peaks are present.b, A 30×30 nm 2 topographic image of FeTe taken with a spin-polarized tip.V bias =+100 mV, I=1 nA.The inset is the FFT.There is an additional pair of peaks corresponding to the bi-collinear AFM order.Extended DataFig.2: Large-area and bias-dependent topographic image of NdSb.a, A 60×60 nm 2 topographic image of NdSb.V bias =+1.4 V, I=1.2 nA.b, c, Zoom-in images of the green box in a. b is taken at V bias =+1.4 V, I=1.2 nA and c at V bias =+1.4 V, I=1.2 nA.The blue dashed lines highlight the center of defects on the surface.d, The orbital-decomposed density of states for Nd and Sb sites in the 2q configuration of NdSb.The yellow, orange and blue lines describe the density of Nd d, Nd f and Sb p orbitals, respectively.Bias-dependent spin-polarized topographic images at 4 K. Bias-dependent spin-polarized topographic images from −1 V to +1.4 V.The diminished magnetic contrast around −1 V indicates that at large negative bias, the atom-like features are nonmagnetic Sb atoms, while at positive bias the atom-like features are magnetic Nd atoms.Comparison between spin-polarized topographic images at +1.4 V and −1 V.A 30×30 nm 2 spin-polarized topographic image whose lower part is taken at −1 V and upper part +1.4 V.The green dashed lines track the lattice shift between +1.4 V and −1 V.The spin-polarized images reproduce the lattice shift observed in nonmagnetic images.Tunneling current I=1 nA.

. 5 :. 6 :
Temperature evolution of adjacent magnetic domains.Tunneling condition: V bias =+0.3 V, I=0.5 nA Temperature evolution of normalized magnetic peak intensities of adjacent magnetic domains.The intensities are obtained from Extended Data Fig.5.The magnetic peak intensities in both domains decay as the temperature rises.

Extended Data Fig. 8 :. 9 :. 10 :
Schematics of Nd spin moments rotation below T R .The Nd spin moments on the neighboring layers rotate in opposite directions, transforming from the collinear 1q to non-collinear 2q magnetic order.Evolution of surface states, bulk bands and magnetic order in NdSb.a-g, surface states (top) and bulk band band structures along the X −Γ−Y line with the magnetic moment angle relative to the x-axis increasing as 0, 5, 10, 20, 30, 40, 45 degrees, respectively.a corresponds to the 1q configuration.g corresponds to 2q with magnetic moments along the diagonal in the (001) plane.The experimental determination of the magnetic moment puts the angle close to 20 degrees, represented by panel (d).Paramagnetic surface states and bulk bands, and demonstration of the band inversion for θ = 0 (1q) along Γ − X. a, Surface bands along X − Γ − Y for the paramagnetic phase for the conventional cubic cell.Note the absence of any surface states.b, Bulk bands along the X − Γ − Y path showing trivial crossing due to folding within the conventional cell.a and b do not contain any f electron contribution, as these are kept frozen in the core for purposes of accurately capturing the non-magnetic ground state.c, Bulk band inversion for θ = 0, 1q state.The highlighted bands (in red) correspond to the position in momentum space where bands anti-cross.The irreducible representations of C 4x are superimposed next to the bands, showing the topological character of the gap.