Introduction

Rare-earth monopnictides R(Bi,Sb) (R = Ce, Nd, Sm, etc.) with a rock salt structure have been extensively studied in the past 60 years for their intricate magnetic phase diagrams1,2,3,4. Except for the lanthanum, yttrium, and praseodymium compounds, most rare-earth monopnitides are antiferromagnets at low temperatures1,2,3,4. Upon cooling or applying a magnetic field, many of them undergo another or even multiple antiferromagnetic transitions5,6,7, as a consequence of competing magnetic interactions. A notable example is the “Devil’s staircase” in CeSb8 and the magnetic transitions have a significant influence on the electronic band structures9. More recently, rare-earth monopnictides have been proposed to have topologically non-trivial band structures. A first-principle calculation predicted a systematic band inversion reduction from Ce to Yb compounds, and a topologically non-trivial to trivial transition occurs around SmSb and DyBi10. Later angle-resolved photoemission spectroscopy (ARPES) experiments confirmed the band inversion in rare-earth monobismuthides and identified linear-dispersed Dirac-like states11,12,13,14. Magnetotransport experiments revealed non-trivial Berry phase in SmBi15 and SmSb16, and possible Weyl or Dirac fermions in CeSb and NdSb when driving the samples to ferromagnetic phase with magnetic field or pressure17,18,19. Therefore, rare-earth monopnictides are an intriguing platform to explore the interplay between magnetism, strong electronic correlation, and topological band structure.

Remarkably, recent ARPES studies reveal that a number of rare-earth monopnictides, including NdBi, NdSb, and CeBi, host disconnected Fermi surface arcs emerging from an unconventional magnetic band splitting below Néel temperature (TN)20,21. Initial density functional theory (DFT) calculations suggest that the emergent surface states cannot be explained by the ordinary A-type antiferromagnetism observed by neutron scattering experiment3. Later analysis suggests that multiple-wave-vector (multi-q) antiferromagnetic order22 can reproduce the emergent surface states23,24. So far there is no direct experimental evidence of multi-q antiferromagnetic order, which is difficult to be differentiated from multi-domain state of simple 1q order in powder neutron diffraction3. Real-space magnetic imaging probes, such as spin-polarized scanning tunneling microscopy (SP-STM), are suitable for directly visualizing the non-collinear magnetic order25,26.

In this work, we provide real space evidence of non-collinear antiferromagnetic order in NdSb using SP-STM. We observe a checkerboard magnetic contrast on the (001) surface of NdSb that is consistent with a non-collinear spin order, which is a superposition of two A-type magnetic orders with orthogonal in-plane modulations. Furthermore, we discover a hidden spin-rotation transition (1q-2q) 2 K below the Néel ordering temperature TN that has not been reported. More interestingly, the hidden transition coincides with the onset of band splitting observed by ARPES. DFT study reveals a band inversion driven by magnetic order-induced band folding, resulting in the unconventional magnetism-induced topological surface states in the family of rare-earth monopnictides.

Results

Figure 1a shows an STM topographic image taken at 4.5 K with a nonmagnetic tip at sample bias Vbias = + 1.4 V. The image reveals a square lattice of Nd atoms with Nd-Nd spacing ~4.3 Å calculated from the Bragg peaks qlat = (1, 0) and (0, 1) in the Fourier transform map (Fig. 1c), in good agreement with prior studies4. The bias dependence of STM topography indicates the unoccupied states are dominated by Nd orbitals while the occupied states are predominantly of Sb orbitals, which is confirmed by our DFT calculations (Supplementary Fig. 2d). Interestingly, SP-STM topography in Fig. 1b reveals an additional checkerboard-like pattern at + 1.4 V bias, resulting in two pairs of superlattice peaks of wave vectors \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}=\left(\pm \frac{1}{2},\pm \frac{1}{2}\right)\) as shown in Fig. 1d. The bias dependence of SP-STM topography shows that the checkerboard modulation is mainly visible in unoccupied states dominated by Nd orbitals, and gradually diminishes in occupied states with significant Sb character. The disappearance of checkerboard-like modulation is accompanied by a fractional \(\left(\frac{1}{2},\frac{1}{2}\right)\) shift of the atomic lattice, presumably from Nd to Sb atoms (Supplementary Figs. 2–4). This further confirms that the observed checkerboard pattern comes from the spin-dependent tunneling between the magnetic tip and the local spin-polarized Nd orbitals.

Fig. 1: STM and SP-STM topographic images of NdSb at 4.5 K.
figure 1

a, b Non-spin-polarized and Spin-polarized topographic images of NdSb. Tunneling condition: Vbias= + 1.4 V, I = 1 nA. c, d FFT of raw data of (a) and (b) respectively. The green circles highlight the lattice Bragg peak. The red squares highlight the magnetic Bragg peak. e top: The schematics of a spin-polarized STM tip scanning on the surface of NdSb with 1 q spin structure; middle: the top view of the 1 q spin structure. The red stripes illustrate the modulation along \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\); bottom: the expected Fourier map of the 1 q spin structure. f schematics of the 2 q spin structure and expected Fourier map. The spins are not along the diagonal direction. It can be considered as the superposition of two unequal 1 q modulations along \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) and \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\). The resulting magnetic peaks are asymmetric.

Full size image

Close inspection of the Fourier transform map (Fig. 1d) reveals that there is an asymmetry of the magnetic superlattice peak intensity along two orthogonal directions: the intensity of \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}=\left(\frac{1}{2},\frac{1}{2}\right)\) is stronger than that of \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}=\left(\frac{1}{2},-\frac{1}{2}\right)\). We also performed bias-dependent studies to confirm that asymmetry persists in the whole bias range when the magnetic modulation is visible (Supplementary Fig. 3). This magnetic asymmetry is uniform over hundreds of micrometers, indicating a global C4 symmetry breaking. To confirm that this symmetry breaking is intrinsic, we examine the sample surface extensively over hundreds of micrometers and locate a magnetic domain boundary shown in Fig. 2c. The SP-STM image is taken at Vbias = + 0.3 V to maximize the magnetic contrast. The magnetic modulation asymmetry rotates 90° across the domain boundary, as clearly demonstrated in the Fourier transform maps of two domains and the corresponding line profiles along orthogonal directions (Fig. 2b, e).

Fig. 2: Magnetic modulations across a domain boundary of NdSb.
figure 2

a FFT obtained from the squared area on the left of the domain wall in c. \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) and \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\) highlight the magnetic Bragg peaks along two orthogonal directions. b Line profiles along \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) and \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\). c A spin-polarized topographic image with a magnetic domain wall highlighted by the white dashed line. Tunneling condition: Vbias = 0.3 V, I = 0.3 nA. d, e Similar to (a) and (b) but obtained from the squared area on the right of the domain wall in (c).

Full size image

This magnetic asymmetry suggests a magnetic structure that is different from that proposed in prior DFT analysis where the angle between Nd spin and cubic axes is either 0° (1 q) or 45° (simple 2 q)21,23,24. The out-of-plane spin modulation in 3 q cannot be detected by SP-STM, so we focus on the in-plane multi-wave-vector modulation and do not differentiate 2 q and 3 q in the following text. For the 1 q magnetic structure, there is no magnetic modulation on the (001) surface, thus no magnetic superlattice peak. In contrast, there is a single modulation direction on the (110) [i.e., cubic (010)] surface, resulting in one pair of magnetic superlattice peaks (Fig. 1e). For the proposed simple 2 q modulation21,23, the Nd spin moments are 45° with respect to the cubic a, b-axes, so the 4-fold rotational symmetry (plus a fractional lattice translation) is restored. This magnetic modulation can be considered as a superposition of two orthogonal 1 q modulations with equal spin components along the modulation directions, giving rise to two pairs of magnetic peaks with equal intensity on the (001) surface. Therefore, the two pairs of magnetic peaks with unequal intensity must stem from a magnetic structure with a spin configuration between 1 q and the simple 2 q. In other words, the magnetic structure of NdSb is a 2 q modulation with broken C4 symmetry where the angle between Nd spin moments and the cubic a, b-axes is between 0° and 45°. In this scenario, the spin components along \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) and \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\) are proportional to their modulation amplitudes, which are proportional to the magnetic peak amplitudes. Using the ratios of superlattice peak intensities, we estimate that the Nd moment rotates θ ≈ 18° from \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\). The angle is weakly temperature-dependent below 13 K (see Methods and Supplementary Fig. 10b).

To understand the origin of asymmetric magnetic modulation, we perform SP-STM measurements at various temperatures to examine the temperature dependence (Fig. 3b–d and Supplementary Information Figs. 5–7). At 11 K, the magnetic contrast is much weaker than that at 4.5 K, indicating a dramatic reduction of order moments. At around 14 K, the magnetic contrast is barely visible in SP-STM topography. From the Fourier transform the magnetic peaks with higher intensity (\({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\)) remain while the ones with lower intensity (\({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\)) disappear. Above TN( ≈ 15 K), all the magnetic superlattice peaks disappear, indicating the recovery of the paramagnetic phase. Figure 3e shows the line profile of the Fourier transform intensity along two orthogonal directions crossing \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) and \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\). Clearly, \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\) vanishes above 13 K while \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) persists to TN, indicating a 1 q magnetic structure in this 2 K window. Therefore, there is a hidden 1 q-2 q transition at 13 K, which can be considered as anti-phase rotation of Nd spins in neighboring layers (Fig. 3a and Supplementary Fig. 11). So the 1 q-2 q transition is a spin-rotation transition. Just below TN, the peak intensity (\({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\)) is weak and develops slowly. However, it grows sharply below TR accompanied by the emergence of orthogonal modulation (\({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\)), whose intensity also increases drastically. Therefore, our SP-STM results also suggest that the magnetic order parameter is greatly enhanced below spin rotation transition TR.

Fig. 3: Temperature-dependence of magnetic modulations in NdSb.
figure 3

a The magnetic phase diagram in NdSb. b–d Spin-polarized topographic images taken at the same area at 11 K (b), 14 K (c) and 16 K (d). The insets are the corresponding FFT and the positions of the magnetic Bragg peaks are highlighted by the red (\({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\)) and blue (\({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\)) squares. e The temperature-dependent line profiles along \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) (red) and \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\) (blue) magnetic Bragg peaks. f The normalized magnetic Bragg peak intensity of \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{A}}}}}}}}}\) and \({{{{{{{{\bf{q}}}}}}}}}_{{{{{{{{\rm{AFM}}}}}}}}}^{{{{{{{{\bf{B}}}}}}}}}\) as a function of temperature. The blue and red lines are guides to the eyes.

Full size image

The hidden spin-rotation transition provides new insight into the mechanism of the mysterious splitting of emergent surface states observed in NdBi and NdSb crystals20,21. In Fig. 4a, we plot the Fermi surface of NdSb measured by ARPES at 5.5 K. Consistent with previous results21,24, the surface states are two-fold symmetric. In addition, the surface states also appear at the original M-point in the Brillouin zone of the paramagnetic phase. This observation agrees with prior ARPES results14,24,27, confirming that the surface states are correlated with the band-folding induced by magnetic modulations24,27. Interestingly, two recent ARPES studies report an additional domain state with new 4-fold symmetric bulk Fermi surface in which the surface states are absent27,28. However, we didn’t observe such a domain state in our NdSb crystal samples by either APRES or SP-STM measurements. Figure 4c shows the temperature evolution of band dispersion. Besides the two split surface bands (denoted as α and β) reported before21, we observe a third surface band γ in agreement with recent reports14,24. We extract the momentum distribution curves (MDCs) at the Fermi energy as shown in Fig. 4e. Above TN, only a broad shoulder-like feature due to the bulk bands can be observed. Below TN, a peak appears as a signature of the emergent surface states. The intensity of the peak increases gradually upon cooling but it does not split into α and β bands until below 13 K, in good agreement with a recent APRES study27. Interestingly, the onset temperature of surface band splitting coincides with TR, further substantiating that the spin-rotation transition is an intrinsic phenomenon of NdSb. Note that the same behavior of surface band splitting below TN ( ≈ 24 K) has been observed in the sister compound NdBi, where the emerging surface states split below 20 K20. This indicates a similar hidden spin-rotation at in NdBi, which deserves a future exploration. Our results suggest that the hidden transition is likely a universal phenomenon for the Nd monopnictides.

Fig. 4: ARPES spectra at different temperatures and DFT calculated band structure of NdSb (001) surface.
figure 4

a Fermi surface map of NdSb (001) measured at T = 5.5 K. The photon energy of ~ 53 eV is used. The purple solid lines outline the Brillouin zone for the paramagnetic phase of NdSb. The blue dashed lines outline the folded Brillouin zone for the multi-q antiferromagnetic phase of NdSb. α, β and γ denote three emergent surface states. b DFT calculated Fermi surface of 2q antiferromagnetic phase projected on NdSb (001) surface. c Temperature evolution of ARPES spectra of NdSb along the ΓX direction. d DFT calculated ΓX band structure of NdSb at varying angles of Nd spin moments. e Temperature-dependence of the MDCs at the Fermi level, as marked by a red arrow in (c). The peaks of α and β surface states are fitted by the Lorentzian function. f Temperature-dependence of the MDC peak splitting (black) and the sum of peaks area for the α and β surface states (blue). The error bars are obtained from the Lorentzian fittings in (e). g The calculated band splitting of α and β surface state as a function of the angle between Nd spin moments and a, b-axes.

Full size image

To understand the fundamental mechanism of emergent surface states and their splitting, we perform DFT calculations to understand the bulk and surface states evolution from PM to 1 q and from 1 q to 2 q transitions. The PM phase of NdSb is topologically trivialbecause of the lack of direct band inversion between the Nd 5 d and Sb 3 p orbitals (Supplementary Figs. 13a, b and 14a). In contrast, the 1 q or 2 q phase leads to a band folding along Γ − X in the bulk, resulting in the band inversion between the Nd 5 d and Sb 3 p orbitals (Supplementary Fig. 14b). Here, spins align along the x-direction in the 1 q phase. The reduced symmetry also allows the opening of a topological band anti-crossing gap near the Fermi energy (EF). Consequently, two surface bands emerge from the upper and lower bulk band edges involved in the anti-crossing gap and disperse in the direction of Γ − X (Fig. 4d). The surface band splitting is small in 1 q phase because of the small anti-crossing gap.

Because of multiple band crossings at different energies and momenta in the full Brillouin zone, the global topological invariant of NdSb is trivial even in the antiferromagnetic phases. However, this does not necessarily negate the existence of topological surface states near EF in the local momentum region around Γ, which is distinct from the case of a trivial insulator as shown in Supplementary Fig. 13a, b.

Below the 1 q-2 q transition, spins rotate in the xy plane to form a non-collinear order. Such spin rotation increases the anti-crossing gap and thus splits the surface bands further (Fig. 4d, g, and Supplementary Fig. 12) according to our DFT calculations, which correspond to the zero temperature limit. In finite temperatures, the split is expected to be reduced because thermal fluctuations suppress the magnitude of the order moment. Consistently, the observed temperature dependence of surface band splitting from ARPES results (Fig. 4f) follows that of spin-modulation amplitude in Fig. 3f, which is a measure of the order moment size. Near and above the 1 q-2 q transition, the spin-modulation amplitude is so weak that the surface band splitting is smaller than peak width, resulting in a single surface band observed in ARPES measurements20,27. In other words, the surface band splitting below the 1 q-2 q transition is due to the enhancement of the anti-crossing gap in the non-collinear antiferromagnetic state (2q) because of an enhancement of order moment. Besides understanding the origin of surface band splitting, our DFT calculations also confirm that the Fermi surface projected on the (001) surface of 2 q phase with rotation angle (θ = 20°) similar to the experimental value is two-fold symmetric (Fig. 4b), in good agreement with the ARPES results shown in Fig. 4a. Therefore, the ground state of NdSb is a 2 q antiferromagnetic structure with broken C4 symmetry.

The combined SP-STM, ARPES, and DFT studies of NdSb establish that the band folding due to antiferromagnetic ordering induces topological band inversion at Brillouin zone boundaries, resulting in emergent topological surface states. Furthermore, our studies unveil a hidden spin-rotation transition from 1 q to 2 q spin orders 2 K below TN, which is responsible for the enhanced order moment and the resultant unconventional magnetic band splitting. The non-collinear spin order and hidden spin-rotation transition might also exist in other rare-earth monopnictides. Besides clarifying the mechanism of topological surface states, our work provides an example of magnetism-induced topological phase transition, thus enabling an exciting possibility of tuning band topology with magnetism.

Methods

STM and SP-STM measurements

The STM measurements were carried out in an Omicron LT-STM with base pressure better than 1 × 10−11 mbar. For non-spin-polarized STM, electrochemically etched tungsten tips were conditioned on a clean Au (111) surface before experiments. For SP-STM experiments, the spin-polarized tip is functionalized by repeatedly scanning the nonmagnetic tungsten tip on cleaved surfaces of FeTe single crystals29,30. The spin-polarized tip is confirmed by visualizing the in-plane collinear antiferromagnetic order on FeTe (Supplementary Fig. 1). NdSb single crystals were cleaved in-situ around 5 K and then immediately inserted into the cold STM head. All STM images in the paper are drift-corrected using the Lawler-Fujita algorithm31. High-frequency noise was removed by the low-passed filter in the Fourier maps (Supplementary Fig. 5).

ARPES measurements

The ARPES measurements were carried out at Beamline 10.0.1, Advanced Light Source, Lawrence Berkeley National Laboratory. A fresh NdSb (001) surface was achieved by cleaving the NdSb crystal at T = 20 K. One surface termination was exposed for the cubic structure of NdSb, similar to other Rare-earth monopnictides. The base vacuum of the ARPES chamber is better than ~ 5 × 10−11 mbar. A hemispherical Scienta R4000 analyzer was used in our ARPES measurements. The energy and angle resolution was set to ~ 15 meV and ~ 0.1°, respectively. The spot size of the photon beam is ~ 100 × 100 μm2.

DFT calculations

We carried out DFT calculations using the Vienna ab-initio software package (VASP)32,33. The lattice constant adopted for NdSb is a = 6.70 Å, in order to match the ARPES finding that the paramagnetic phase is trivial. The properties of Nd f electrons were corrected with a Hubbard U = 6.3 eV and J = 0.7 eV using Dudarev’s method34. The plane wave states were then projected onto a tight-binding basis of Nd d, f and Sb p orbitals using Wannier9035. Surface states were calculated using efficient, iterative Green’s function methods as implemented in WannierTools36.

NdSb single crystal growth

NdSb single crystals were synthesized by a Sn flux method37. High-quality powders of Nd, Sb, and Sn in a ratio of 1:1:20 were put into an alumina crucible which was subsequently sealed within a vacuumed quartz tube. The prepared ampoule was heated up to 1150°C and then slowly cooled down to 750°C at a rate of 3°C/h, followed by decanting of the excess liquid using a centrifuge. Shining cubic crystals were obtained after breaking the crucible.

Density of states for Nd and Sb

In Supplementary Fig. 2b, c, we show the non-spin-polarized STM image taken at the tunneling bias of + 1.4 V and − 1.3 V. It is evident that at + 1.4 V the highlighted defects are centered on the protrusions whereas at − 1.3 V the depressions, indicating at + 1.4V and − 1.3 V different atoms (Nd or Sb) are imaged. In Supplementary Fig. 2d We plot the ab-initio calculated partial density of states, projected on the atomic sites in the 2q configuration of NdSb. We focus on the energy window probed by STM with the Fermi level set to zero. For negative voltages, the density of states is dominated by Sb p orbitals, while for positive voltages the density of states is composed primarily of Nd d, f orbitals. This suggests that at positive tunneling biases, the bright atom-like features are Nd atoms because of their large DOS above Fermi energy, while Sb atoms are imaged at negative bias.

Estimation of Nd spin moment angle from spin-polarized STM images

In Fig. 2 and Supplementary Fig. 8, we observe a magnetic domain wall separating two magnetic domains of different magnetic contrasts. The amplitude asymmetry of the magnetic peaks qA and qB in the Fourier transform switches sign across the domain wall. Therefore, we can estimate the angles between Nd spin moments and the magnetic modulation using the magnetic peak intensities of these two domains.

Supplementary Fig. 10a shows the schematics of AFM modulations (qA and qB), tip moments (green arrows) and Nd moments (orange arrows) on the Nd lattice (black dots). On the left domain, we denote the angle between Nd moment and qA as θ, and the angle between the tip moment and qA as α. As explained in the main text, the 2q modulation can be decomposed into two unequal 1q modulations along qA and qB, whose spin moments are along the modulation wave vectors. The sum of the spin moments of these two 1q modulations gives rise to the spin moments in 2q. Without losing generality (e.g. in Supplementary Fig. 8), we assume modulation amplitudes \({S}_{A}^{L}=S\cos \theta\) are stronger than \({S}_{B}^{L}=S\sin \theta\) on the left domain, i.e., assuming \({S}_{A}^{L} > {S}_{B}^{L}\). Thus, θ < 45° by definition. The situation is the opposite on the right domain, i.e., \({S}_{A}^{R}=S\sin \theta\) and \({S}_{B}^{R}=S\cos \theta\). Therefore, the Nd moments on the left domain are more inclined to qA, and vice versa for the right domain. Therefore, the Nd moment angle θ is related to the ratio of the modulation amplitudes SA and SB along qA and qB through the following equation:

$$\frac{{S}_{A}}{{S}_{B}}=\frac{S\cos \theta }{S\sin \theta }=\frac{1}{\tan \theta }$$
(1)

In SP-STM, the spin-dependent component of tunneling current is proportional to the projection of local spin orientation onto the tip moment38. Therefore, the measured magnetic peak amplitudes with SP-STM are determined by projections of the two 1q modulations onto the tip moment.

For the left domain:

$$\frac{{I}_{A}^{L}}{{I}_{B}^{L}}=\frac{{S}_{A}^{L}\,{m}_{{{{{{{{\rm{tip}}}}}}}}}\,\cos \alpha }{{S}_{B}^{L}\,{m}_{{{{{{{{\rm{tip}}}}}}}}}\,\sin \alpha }=\frac{S\cos \theta \cos \alpha }{S\sin \theta \sin \alpha }=\frac{1}{\tan \theta \cdot \tan \alpha }$$
(2)

For the right domain:

$$\frac{{I}_{A}^{R}}{{I}_{B}^{R}}=\frac{{S}_{A}^{R}\,{m}_{{{{{{{{\rm{tip}}}}}}}}}\,\cos \alpha }{{S}_{B}^{R}\,{m}_{{{{{{{{\rm{tip}}}}}}}}}\,\sin \alpha }=\frac{S\sin \theta \cos \alpha }{S\cos \theta \sin \alpha }=\frac{\tan \theta }{\tan \alpha }$$
(3)

where \({I}_{A}^{L},{I}_{B}^{L},{I}_{A}^{R},{I}_{B}^{R}\) are the magnetic peak amplitudes along qA and qB directions obtained from the Fourier transform of the left and right domains in SP-STM images, respectively. We use the following two relations to compute α and θ:

$$\left\{\begin{array}{l}\theta=\arctan \left(\sqrt{\frac{{I}_{A}^{R}}{{I}_{B}^{R}}\cdot \frac{{I}_{B}^{L}}{{I}_{A}^{L}}}\right) \\ \alpha=\arctan \left(\sqrt{\frac{{I}_{B}^{R}}{{I}_{A}^{R}}\cdot \frac{{I}_{B}^{L}}{{I}_{A}^{L}}}\right) \end{array}\right.$$
(4)

The temperature-dependence of the angle θ between the Nd spin moments and the in-plane a, b-axes is plotted as orange squares in Supplementary Fig. 10b. The angle is around 18° from 4.5 K up to 12 K ( ≲ TR).

Topological features of the 1q to 2q transition

To verify the topological nature of the surface states, we connect the theoretically calculated and experimentally observed states with bulk band crossings. In our calculations of the non-magnetic phase of NdSb, band structure and surface spectrum shown in Supplementary Fig. 13a, b, is topologically trivial with no surface states, consistent with Fig. 4c for T ≥ TN. Upon the onset of magnetic order, the 1q phase assumes the magnetic space group P4/mm’m’. For this configuration, folding due to magnetism occurs only along the axis parallel to the magnetic moment direction, while there is no folding along the directions perpendicular to the magnetic moment axis. Therefore, we find surface states along \(\bar{\Gamma }-\bar{X}\), but not along \(\bar{Y}-\bar{\Gamma }\). As shown in Supplementary Fig. 12a–g, the appearance of surface states is matched by the opening of a gap in the bulk near EF (dashed line). This indicates the topological origin of the states which result from the magnetic order-induced folding. When such folding is absent (or trivial), as along \(\bar{Y}-\bar{\Gamma }\), surface states do not appear. When the spins begin to rotate towards the 2q configuration, the symmetry is reduced further, and the magnetic space group is given by Pc’c’m’ for all 0 < θ < 45o. This induces folding along all axes. The emergence of these surface states is always accompanied by a bulk band inversion and gap opening, which manifests in the appearance of surfaces states which merge with the bulk at finite k. It should be noted that these states are only present as a result of the folding induced by the magnetic ordering. The splitting of the surface states along Γ − X is also strongly tied to the magnetic order. We take a line cut along Γ − X for E = 0.07 eV and plot the momentum splitting of the two surface states corresponding to the α, β bands in Fig. 4g. As the spins are rotated away from the 1q configuration, the splitting increases, as the symmetry is reduced further. Direct evidence for the band inversion can be inferred from the band characters near EF. We calculate the irreducible representations of C4x (x being the spin direction) along Γ − X. We find a band inversion (band character inversion) precisely at the gap opening in the bulk band structure. This is illustrated in Supplementary Fig. 13c. The total topological indices of the system (calculated at the TI-invariant momenta) remain zero, as in the paramagnetic state. For simplicity and continuity of illustration, all band dispersions are plotted in terms of the folded unit cell.