Introduction

Magnetism and topology can meet each other both in real space, giving rise to complex magnetic structures such as vortices or skyrmions, and in reciprocal momentum space, resulting in Weyl semimetal or magnetic topological insulator (MTI) phases. In the MTI case, the interplay between topology and magnetism provides particularly rich playground for realization of exotic physics. Below the magnetic critical temperature, the time-reversal symmetry breaks down introducing a mass term to the linear dispersion of Dirac fermions thus opening opportunities for realization of such phenomena as quantized anomalous Hall (QAH) and magnetoelectric effect, axion electrodynamics, or Majorana fermions1,2,3,4,5. These unusual properties make MTIs extremely attractive for applications in novel electronics and in the emerging 2D6,7,8 and antiferromagnetic (AFM) spintronics9,10,11.

Until recently, the magnetism in topological insulators (TIs) has been achieved using either magnetic doping3,4,12,13,14,15,16, or proximity effect17,18 as well as by construction of van der Waals (vdW) heterostructures19,20,21,22,23. This situation has changed drastically with the recent discovery of the AFM TI phase24 in stoichiometric vdW-layered antiferromagnet MnBi2Te425,26,27,28,29, which inspired a lot of research activity as it holds promise for realization of the high-temperature QAH and axion insulator states, Majorana hinge modes and other effects24,26,27,28,30,31,32,33,34.

Here we propose the MTI family by introducing the (MnBi2Te4)(Bi2Te3)m series of vdW materials that, apart from MnBi2Te4 (m = 0), contains six more topologically nontrivial compounds, namely MnBi4Te7 (m = 1), MnBi6Te10 (m = 2), MnBi8Te13 (m = 3), MnBi10Te16 (m = 4), MnBi12Te19 (m = 5), and MnBi14Te22 (m = 6). Along the series, the strength of the interlayer exchange interaction, that couples neighboring FM Mn layers, gradually decreases with the increase of m, while the overall magnetic ordering changes from an AFM (m = 0, 1, 2) to FM (m = 3). Combined with the non-trivial topology of the (MnBi2Te4)(Bi2Te3)m compounds, these magnetic states give rise to the AFM and FM TI phases for m = 0, 1, 2 and m = 3, respectively. For large m numbers a new MTI phase can be formed in which, below the respective TC, the magnetizations of the 2D FM-ordered Mn layers of the MnBi2Te4 building blocks are disordered along the [0001] direction. The topologically nontrivial nature of these compounds is confirmed by angle-resolved photoemission spectroscopy (ARPES) measurements that reveal the presence of the topological surface states (TSS) whose dispersion depends strongly on the crystal surface termination. The unusual magnetic properties make the (MnBi2Te4)(Bi2Te3)m series a unique tunable platform for creating various exotic states of matter, such as intrinsic axion or QAH insulator20,26,32, the field-induced QH phase31,32, high-order Möbius insulator35, or chiral topological superconductor33.

Results and discussion

Crystal structure and magnetic crossover

The first compound in the (MnBi2Te4)(Bi2Te3)m series is MnBi2Te4 (m = 0), which had been investigated previously and discovered to be the first AFM TI25,36. This system consists of septuple layer (SL) blocks stacked one on top of another. Each SL is a 2D ferromagnet, while the coupling between the neighboring SLs is AFM25,37. For m ≥ 1, the members of the (MnBi2Te4)(Bi2Te3)m family are comprised of alternating septuple (MnBi2Te4) and quintuple (Bi2Te3) layer blocks (see Fig. 1a). The growth details and crystal structures of thus-formed MnBi4Te7 and MnBi6Te10 compounds have been first reported in ref. 38, for which the -5-7-5-7- and -5-5-7-5-5-7- blocks sequences corresponding to m = 1 and m = 2 have been revealed, respectively.

Fig. 1
figure 1

a Compounds (MnBi2Te4)(Bi2Te3)m consist of alternating five-layer (QL) and magnetic seven-layer (SL) blocks. b, c Temperature dependence of the resistivity for MnBi4Te7 and MnBi6Te10 for various applied out-of-plane magnetic fields. d Normalized magnetic susceptibility for MnBi4Te7, MnBi6Te10, MnBi8Te13, and MnBi10Te16 as a function of temperature measured in a small magnetic field in zero-field-cooled and field-cooled conditions. d, e Field-dependent magnetization curves taken at various temperatures with out-of-plane external magnetic field for MnBi4Te7, MnBi6Te10, MnBi8Te13, and MnBi10Te16.

Full size image

The XRD patterns of our MnBi4Te7, MnBi6Te10, MnBi8Te13, MnBi10Te16, MnBi12Te19, and MnBi14Te22 samples, shown in Supplementary Fig. 1, confirm their \(P\bar{3}m1\) and \(R\bar{3}m\) space groups.

The temperature dependence of the resistivity measured in a zero magnetic field demonstrates the metallic-like behavior for both MnBi4Te7 and MnBi6Te10 (see Fig. 1b, c). At low temperatures, the well-defined kinks at 13.2 K (MnBi4Te7) and 11.9 K (MnBi6Te10) are observed for both compounds. As the external magnetic field is switched on and increased these features are washed out, pointing towards their magnetic origin.

The field-cooled (FC) and zero-field cooled (ZFC) magnetic susceptibilities measured at 0.1 T for MnBi4Te7 are shown in Fig. 1d (blue curve). The high temperature paramagnetic behavior ends up with a pronounced peak at 13 K, that is characteristic of an AFM ordering. The fitting of the susceptibility by the Curie–Weiss law yields the Curie–Weiss temperature (ΘCW) of 13.2 K. Positive sign and relatively high value of ΘCW (cf. 3–6 K for MnBi2Te425) indicate the presence of strong ferromagnetic interactions in MnBi4Te7 in spite of the overall AFM behavior. The Néel temperature of only 13 K for MnBi4Te7 versus 25 K for MnBi2Te4 reveals a strong weakening of the interlayer AFM coupling due to the insertion of the non-magnetic quintuple layer block (QL) between neighboring SLs. The onset of the ferromagnetic interactions is also manifested in splitting of the FC and ZFC curves at low temperature (spin-glass-like transition).

Increasing by one the number of QLs between SLs and forming MnBi6Te10 leads to further weakening of the interlayer exchange interaction. The magnetic susceptibility data for this compound, shown in Fig. 1d (red curve), exhibit similar cusp at low temperature, signaling the AFM ordering, but the Néel temperature decreases to 11 K in this case. Furthermore, below 10 K FC and ZFC curves splitting becomes larger than for MnBi4Te7, thus showing the enhanced competition between AFM and FM couplings.

The m = 3 and 4 members of the series, i.e. MnBi8Te13 and MnBi10Te16, show an overall ferromagnetic behavior below the Curie temperatures of about 12 K, as revealed by the magnetization measurements (Fig. 1d); a similar behavior is expected for MnBi12Te19 (m = 5) and MnBi14Te22 (m = 6) because of the vanishing interlayer exchange interaction.

The M(H) curves for m = 1–4 compounds taken at various temperatures are presented in Fig. 1e–g. Slightly below the Néel temperature (10 K) for MnBi4Te7 a typical AFM M(H) behavior takes place, with a spin–flip upturns appearing at 0.2 T, the latter value being much lower than that for MnBi2Te4 (3.5 T)25. The gradual reduction of interlayer exchange interaction leads to further decrease of this value, the M(H) dependence for MnBi6Te10 compound (red curve in Fig. 1e) exhibits the spin–flip transition at about ~0.07 T. The next members of the MBT family, i.e. MnBi8Te13 and MnBi10Te16 demonstrate rather conventional ferromagnetic s-like curves.

At very low temperatures (T = 5 and 2 K), the spin–flip hysteresis opens up in the magnetic field interval from 0.1 to 0.3 T for m = 1 system, as seen in Fig. 1f, g. The m = 2 compound exhibits the same behavior, but at lower external fields. Moreover, at 2 K, in the absence of external field, the magnetization does not fall to zero and forms a ferromagnetic hysteresis loop with small and large remanence for m = 1 and 2, respectively. Such a dual complex metamagnetic behavior can be explained by the presence of domains with ferromagnetic and AFM ordering between neighbor SLs. However, in case of m = 2 the remanence close to saturation shows that forced FM state can survive at zero and even at small negative magnetic fields when the thermal fluctuations are weak enough. Similar results with the metamagnetic behavior of MnBi4Te7 and MnBi6Te10 have been reported recently in refs. 39,40,41,42,43. The m = 3 (MnBi8Te13) compound clearly demonstrates an almost pure ferromagnetic hysteresis loops, only some small deviations from s-like curve can be distinguished at T = 5 K. Note, that 3D FM ordering for MnBi8Te13 has been claimed in ref. 44. The presented data testify the weakening of 3D AFM coupling with m number increase and strong 2D FM intralayer coupling for all compounds.

The described experimental picture is consistent with that yielded by theory. Namely, the DFT exchange coupling parameters calculations reveal, first, a stable tendency to the intralayer FM ordering in the SLs of MnBi4Te7 and, second, a strong drop of the interlayer exchange coupling (1–2 orders of magnitude) as compared to MnBi2Te4 (see Supplementary Note 3). Highly accurate total-energy calculations give an energy difference between the FM and interlayer AFM configurations of 0.25 meV per Mn pair in favor of the interlayer AFM one. The magnetic anisotropy energy, Ea, is positive and equals to 0.12 meV (i.e. the easy axis is out-of-plane), which is roughly two times smaller than in bulk MnBi2Te4. In good agreement with the resistivity and magnetization measurements, our Monte Carlo simulations yield a Néel temperature of 13.6 K for the bulk MnBi4Te7. The drop of the Néel temperature from about 25 K in MnBi2Te4 to ≃13 K in MnBi4Te7 is precisely caused by the weakening of the interlayer exchange coupling.

The DFT total-energy calculations performed for MnBi6Te10 show that the interlayer coupling weakens with respect to the MnBi4Te7 case, the difference between the AFM and FM states being 0.1 meV in favor of the AFM ordering. These results are highly consistent with the experiment, where an AFM ground state has been found for MnBi6Te10, while a small external field of ~0.1 T is enough to force the FM ordering, that can survive at zero and some negative magnetic fields at low temperature.

In ref. 26, the Curie temperature of the free-standing MnBi2Te4 SL has theoretically been predicted to be 12 K, which coincides with the corresponding values for MnBi8Te13 and MnBi10Te16 (see Fig. 1d). These facts allow us to conclude that the interlayer exchange coupling between SLs vanishes in (MnBi2Te4)(Bi2Te3)m family for m > 3 and in the case of absence of the free carriers their magnetic structure can be described as a set of decoupled 2D ferromagnets whose magnetizations are randomly oriented— either parallel or antiparallel to the [0001] direction. Owing to the n-doping of the samples, the long-range RKKY exchange interaction may still weakly couple the magnetic layer blocks even for m > 3, although no indication of that has been observed in the experiment. Notably, some small concentration of Bi/Mn defects (~3%, see Supplementary Note 2 and ref. 45) can slightly influence magnetic properties, that requires further investigations. Thus, the interlayer exchange interaction between the 2D FM SLs in the (MnBi2Te4)(Bi2Te3)m series is AFM for m = 0, 1, 2 (MnBi2Te4, MnBi4Te7, and MnBi6Te10), FM for m = 3 (MnBi8Te13) and then progressively weakens with increase of m (MnBi10Te16, MnBi12Te19, and MnBi14Te22). As we show next, these crossovers have profound consequences for the topological properties of the materials in this series.

Topological surface states

We now turn to the electronic structure study of the compounds in the (MnBi2Te4)(Bi2Te3)m series. As it has been mentioned above, the MnBi2Te4 (m = 0) compound was shown to be an AFM TI. The ARPES dispersion relations for the (0001) surface of the MnBi4Te7 compound (m = 1) measured at T = 60 K are shown in Fig. 2a. Within the binding energy (BE) window from 0.5 to 4 eV, a quite complex bandstructure, characteristic of bismuth chalcogenides is seen46,47. However, unlike nonmagnetic TIs, a weakly dispersing state is discernible in Fig. 2a at the BE of about 3.8 eV, which is attributed to the Mn 3d-states, whose intensity is enhanced at the Mn-resonant photon energy of 50 eV.

Fig. 2
figure 2

a, d ARPES dispersion relations for the large energy and momentum region of MnBi4Te7 and MnBi6Te10 taken at 60 K at a photon energy of 50 eV. b, e ARPES dispersion relations of the Dirac cone region taken at a photon energy of 30 eV at 60 K (BCB (BVB) denotes bulk conduction (valence) band). c DFT-calculated layer-resolved electronic structure of the MnBi4Te7(0001) surface. The size of color circles reflects the weight of the state in the topmost block of the SL (green) and QL (blue) termination. The orange circles show the contribution of the subsurface SL block of the QL termination. The gray shadings correspond to the bulk bandstructure projected onto the 2D Brillouin zone. f The same for MnBi6Te10(0001) surface, the size of green, blue, and red circles reflects the weight of the state in the topmost block of the 7-5-5-, 5-7-5-, and 5-5-7- terminations, respectively.

Full size image

Detailed ARPES dispersion relations in the low-energy part of the spectrum are presented in Fig. 2b (hν = 30 eV). Around the \(\overline{\Gamma }\)-point, there are two electron pockets that can be attributed to the bulk conduction bands of MnBi4Te7. Thus the material is n-doped which is typical of Bi-containing TIs48. The third electronic band, crossing the Fermi level, has largely a linear dispersion except for the close vicinity of the \(\overline{\Gamma }\)-point, where it becomes practically flat at the BE of 0.27 eV. At slightly higher BE, there is a M-shaped state, which is accompanied by a hole-like band with which it is degenerate at the Brillouin zone (BZ) center. Thus, between the linear and M-shaped states there is a band gap. Below the gap the electron-like part of the latter band behaves like a continuation of the linear band, as both have practically the same group velocity.

To separate the bulk and surface states, ARPES measurements with various photon energies (50, 18, and 6.3 eV (laser)) have been performed, as presented in Supplementary Fig. 6. The dispersion of the linear band does not show any photon energy dependence, which points towards its surface or 2D character. This allows suggesting that the linear band is the TSS, while far from the \(\overline{\Gamma }\)-point the M-shaped band is likely to be the trivial surface state in the valence band that interacts with the TSS near the \(\overline{\Gamma }\)-point, leading to the appearance of the "avoided crossing” gap at the BE of 0.29 eV. In this case, the degeneracy point between the M-band and the close-lying hole band can be identified as the Dirac point (DP) of the MnBi4Te7(0001) TSS.

Since the TSS should be spin-polarized, spin-ARPES measurements have been performed. The acquired spectra (Supplementary Fig. 6) clearly demonstrate the spin polarization reversal for opposite branches of the linear band, revealing the characteristic helical spin texture of the TSS. The performed bands assignment allows us to estimate the bulk band gap of MnBi4Te7 to be about 0.15 eV. It should be noted that on the surface of MnBi4Te7, since it is built of two different crystal blocks, QL and SL, two surface terminations and, consequently, two Dirac TSSs with different dispersion are expected. The TSS signal for each termination has been obtained by means of laser-ARPES and will be analyzed below.

According to the DFT electronic structure calculations, AFM bulk MnBi4Te7 features a fundamental band gap of 0.18 eV (Supplementary Note 3). Because of its interlayer AFM ordering, MnBi4Te7 is invariant with respect to the combination of the time-reversal (Θ) and primitive-lattice translation (T1/2) symmetries, S = ΘT1/2, and thus obeys the Z2 topological classification of AFM insulators24,49. We find Z2 = 1 for MnBi4Te7, meaning that its fundamental band gap is inverted whereby, similarly to MnBi2Te4, MnBi4Te7 is an AFM TI below the Néel temperature. The AFM TI state of MnBi4Te7 along with the out-of-plane direction of the staggered magnetization’s easy axis dictate that there should be a gapped (gapless) TSS at the S-breaking (S-preserving) surface. This is exactly what we find: the TSS is gapped at both of the possible terminations of the MnBi4Te7(0001) surface, which is S-breaking. At the SL termination, the DP gap is located inside the fundamental bulk band gap and reaches a large value of 70 meV (Fig. 2c). At the QL termination, a more complex surface electronic structure is revealed, with four bands located in the region of interest. To understand this spectrum, we have performed surface electronic structure calculations at different spin–orbit coupling (SOC) strengths. As it can be seen in Supplementary Fig. 10, with no SOC included there is an exchange-split trivial surface state in the valence band's projected band gap around the \(\overline{\Gamma }\)-point at energy of about −0.3 eV. When the SOC is turned on and the system is in the topological phase, this spin-split state interacts with the TSS of the QL-termination, leading to the appearance of avoided crossings (Supplementary Fig. 10).

At the natural SOC strength (Fig. 2c), these avoided crossing effects are quite significant and the Dirac cone appears to be torn into two parts, one of which is located in the fundamental band gap, while another in the projected band gap between approximately −0.1 and −0.2 eV. The split DP of the QL termination lies at about 0.16 eV below the Fermi level. The smaller DP gap size at the QL termination (about 29 meV) is caused by the predominant localization of the TSS in the surface block, whereby the interaction of this state with the Mn layer, lying in the subsurface block, is weaker. Thus, the complex dispersion of the QL-termination TSS is caused by the interaction with the trivial surface state, located in the bulk valence band (BVB).

A comparison between the experimental and theoretical spectra allows identifying the measured surface bandstructure in Fig. 2b as that of the QL termination. In the PM state, the trivial surface state is exchange-unsplit which leads to a less complex appearance of the overall spectrum (as compared to the theoretical one): there is only one avoided crossing clearly seen in ARPES in Fig. 2b. Thus, the gap in the TSS at a BE of about 0.27 eV (Fig. 2b) is indeed related to the "cut” of the Dirac cone by a trivial surface state and therefore this gap is observed above the Néel temperature. Cooling down below the magnetic critical point does not lead to significant changes (see Supplementary Fig. 7), except for a slight broadening of the BVB, most likely related to the exchange splitting of the bulk states. Finally, our tight-binding calculations of the S-preserving (\(10\bar{1}1\)) surface electronic structure reveal a gapless TSS, as expected for an AFM TI (see Supplementary Fig. 4).

The TSSs are also observed for the next member in (MnBi2Te4)(Bi2Te3)m series, MnBi6Te10 (m = 2). The ARPES dispersion relations for this compound are shown in Fig. 2d, measured at the same conditions as the spectrum in Fig. 2a. In the BE region from 1 to 5 eV, the valence band dispersions for MnBi6Te10 look similar to those of MnBi4Te7. Apparently, the intensity of the Mn-3d resonant feature seen at the BE of 3.8 eV is slightly decreased for MnBi6Te10, in accordance with the lower relative Mn concentration. The most pronounced difference between the band structures of the two compounds takes place in the region of BE between 1 eV and EF, where the surface states dominate the photoemission signal. In particular, in Fig. 2d one can see three weakly dispersing states along the \(\overline{\Gamma }-\overline{{\rm{K}}}\) direction around the BE of 1 eV as well as three Dirac cones at the BZ center. This complex dispersion of the TSSs is resolved in the detailed ARPES image, shown in Fig. 2e, where both the lower and upper parts of the three Dirac cones can be clearly seen.

The appearance of three TSSs at the MnBi6Te10(0001) surface is caused by the peculiar crystal structure of the (m = 2) compound, built of the two types of blocks, stacked on top of each other in the -5-5-7-5-5-7- sequence. In such a case, the cleavage procedure leads to the formation of different surface terminations. Such a structure was previously observed for the PbBi6Te10(0001) surface, where the three terraces were attributed to the 5-5-7-, 5-7-5- and 7-5-5- terminations, showing different dispersions of the TSS50. Similarly to PbBi6Te10, the cleaved surface of MnBi6Te10 may exhibit three types of terminations, two of them having a QL on the surface (i.e. either 5-5-7- or 5-7-5-), while the third possible crystal truncation is that by the SL (7-5-5-). Similarly to the PbBi6Te10 case, these terraces have different Dirac cone-binding energies. The inner Dirac state (which by analogy with nonmagnetic TIs with similar structure can be assigned to the TSS generated by the SL-terminated terraces) shows the lowest BE of the DP which is located above the BVB states, while the other two cones (coming from different QL terraces) overlap with the BVB. Measurements at low temperature (1 K, Supplementary Fig. 9) exhibit some changes in the ARPES spectra due to magnetic ordering, that could be related to the gap appearance in the Dirac cone of the SL termination, similar to ref. 51.

Similarly to MnBi4Te7, the AFM MnBi6Te10 obeys the combined S = ΘT1/2-symmetry and based on our DFT calculations we classify it as an AFM TI with a bulk band gap of about 0.145 eV. The DFT-calculated (0001) surface band structure of the AFM MnBi6Te10 is presented in Fig. 2f, where three Dirac cones are clearly seen in agreement with the ARPES data. Both the 7-5-5- and 5-7-5- terminations show essentially similar behaviors to those revealed for the MnBi4Te7(0001). The TSS of the 5-5-7- termination is mostly located in the surface QL block and therefore its DP is almost unsplit.

In order to experimentally separate various terminations of MnBi4Te7 and MnBi6Te10, we used laser-based ARPES technique that spatial resolution is up to ~5 μm52, while for synchrotron-based measurements (Fig. 2) the resolution is up to ~100 μm. In Fig. 3 the corresponding dispersion relations are presented for two (upper panels) and three (lower panels) terminations of MnBi4Te7 and MnBi6Te10, respectively. In contrast to the synchrotron-based measurements the SL-terminated Dirac cone for MnBi4Te7 has been revealed (Fig. 3a). Spectra from QL terminated surfaces (5-7-, 5-7-5- and 5-5-7- (see Fig. 3b, e, f), correspondingly) are in good agreement with theoretical predictions and synchrotron measurements (see Fig. 2), as well as with the recent experimental works53,54,55. The distinct DP positions can be easily determined for 5-7-5- and 5-5-7- terminations of MnBi6Te10, which has larger BE for 5-7-5- termination, in accordance with the theory.

Fig. 3
figure 3

a, b Laser-based ARPES dispersion relations of indicated terminations of MnBi4Te7 along \(\mathrm{\overline{K}-\overline{\Gamma}-\overline{K}}\) direction of the Brillouin zone and c the Fermi surface of 7-5- termination taken at 15 K at a photon energy of 6.3 eV. d–f Laser-based ARPES dispersion relations of various terminations of MnBi6Te10 taken at the same conditions as a, b.

Full size image

The electronic structure of SL terminated surfaces of MnBi4Te7 and MnBi6Te10 (Fig. 3a, d) have similar features and the most important is the Dirac cone, located within the bulk band gap, with the DP at ≃0.28 eV of BE, in agreement with the DFT-derived band structure (Fig. 2c, f). Nevertheless, besides the Dirac cone and bulk bands the laser-based spectra exhibit the wide parabolic-like state, dispersing from ~0.25 eV at the \(\overline{\Gamma }\)-point to the Fermi level at ~0.17 Å−1, that is absent in ab initio calculations (see Fig. 2c, f and refs. 53,56,57). Similar band dispersions for SL-terminated terrace have been observed in refs. 54,56,57. The origin of the parabolic-like band is under debates currently. Since ab initio calculations cannot reproduce this electronic band, its origin can lie in the deviations of the surface structure from the ideal one. As an example, recently the dramatic effect of small variation of the near surface vdW gap on the band structure of MnBi2Te4(0001) has been demonstrated58. In refs. 56,57 the idea of Rashba-like surface state on SL-termination is proposed. Indeed, surface localization of the band can be confirmed since it exists only in SL-terminations and it does not appear for QL-terminated surfaces (Fig. 3b, e, and f). The ARPES Fermi surface (Fig. 3c) exhibits a weak hexagonal warping of the band and absence of the interaction with the bulk conduction band states. Notably, similar band was not observed for SL-terminated surfaces in structurally similar compounds PbBi4Te7 and PbBi6Te1050,59. One also can notice that spectra for SL-termination in MnBi4Te7 and MnBi6Te10 slightly differ in the valence band region, where interaction of the lower part of the Dirac cone with the bulk bands is seen for the case of MnBi6Te10.

Finally, the electronic structure measurements performed for the MnBi8Te13(0001) and MnBi10Te16(0001) surfaces reveal the presence of the TSSs within the bulk band gap too (see Supplementary Figure 5), but their analysis is complicated due to increased number of surface terraces after the cleavage.

Thus, all the compounds of the presented (MnBi2Te4)(Bi2Te3)m family are magnetic TIs, whose topological class changes with m. While MnBi2Te4, MnBi4Te7, and MnBi6Te10 (m = 0, 1, 2) are 3D AFM TIs, MnBi8Te13 (m = 3) turns out to be a FM TI with weak interlayer exchange coupling. For m > 3 no signatures of the interlayer exchange interaction could be detected in our experiments, which has a very interesting consequence: MnBi10Te16, MnBi12Te19, and MnBi14Te22 could be the first examples of stoichiometric 3D MTI compounds in which the magnetizations of the 2D FM-ordered layers are disordered along the [0001] direction below the corresponding critical temperature. Similar idea has been discussed in ref. 60 where starting from m = 3 the so-called "single layer magnet” phase is proposed.

The peculiar magnetic properties of the (MnBi2Te4)(Bi2Te3)m family combined with the nontrivial topology of its constituents enable observation of interesting effects for all of the presented members of the series. Those effects would take advantage of the interlayer exchange coupling tunability along the series, which is feasible through the changing of the number of QLs separating the SLs. This allows one to tailor both the strength and character of the interlayer exchange coupling and even to switch it off for large m numbers. One particular consequence of it is that the magnetic structure, and therefore the topological class of the compounds can be tuned by an external magnetic field. This property is especially attractive in the 2D limit, where the [MnBi2Te4]1SL/(Bi2Te3)mQL(s)/[MnBi2Te4]1SL sandwiches, that can be obtained by careful exfoliation of the thin flakes from the single crystal surface31,32, turn to either the intrinsic zero plateau QAH state (also known as the axion insulator state) or the Chern insulator state. For m > 3, the latter state can be achieved in zero external magnetic field as has been earlier predicted by theory20, while for MnBi2Te4 and, probably, for MnBi4Te7 an external magnetic field is needed to achieve the quantized Hall effect31,32. However, in the MnBi4Te7 and MnBi6Te10 compounds, the strength of the critical field needed to overcome the AFM interlayer exchange coupling should be much smaller than that used for MnBi2Te4. The latter fact has also been pointed out as an advantage for a possible realization of the topological superconductor state based on the (MnBi2Te4)(Bi2Te3)m family, hosting the exotic Majorana fermions33. From this point of view, the (MnBi2Te4)(Bi2Te3)m series represents a unique and highly tunable topological van-der-Waals platform for creation of both exotic topological phases and functional devices for AFM and 2D spintronics as well as for topological quantum computing.

In summary, we have reported the magnetic topological insulators (MTIs) family (MnBi2Te4)(Bi2Te3)m consisting of Mn-based magnetic SL blocks separated by different number m of non-magnetic quintuple layers. The interlayer exchange coupling between the neighboring SLs can be tuned by changing m giving rise to a crossover from the interlayer AFM ordering for m = 0, 1, 2 (MnBi2Te4, MnBi4Te7, and MnBi6Te10) to the overall ferromagnetic one for m = 3 (MnBi8Te13) and, finally, to almost complete disappearance of the interlayer interaction for m > 3 (MnBi10Te16, MnBi12Te19, and MnBi14Te22). Combined with a non-trivial topology of the (MnBi2Te4)(Bi2Te3)m compounds, proven by ARPES, these magnetic states give rise to the AFM and FM TI phases for m = 0, 1, 2 and m = 3, respectively, while for m > 3 a new MTI phase could be formed in which, below TC, the magnetizations of the 2D FM-ordered Mn layers of the MnBi2Te4 building blocks are disordered along the [0001] direction. Depending on the surface terminations, a complex bundle of the Dirac cones is resolved by means of DFT and ARPES. The tunable magnetic and topological phases in (MnBi2Te4)(Bi2Te3)m series allow engineering the promising platforms not only for QAH effect, axion insulators, and Majorana fermions, but also for emerging fields of AFM and van-der-Waals 2D spintronics.

Methods

Magnetic and transport measurements

Magnetic measurements were carried out in the resource center "Center for Diagnostics of Materials for Medicine, Pharmacology and Nanoelectronics” of the SPbU Science Park using a SQUID magnetometer with a helium cryostat manufactured by Quantum Design. The experiments were carried out in a pull mode in terms of temperature and magnetic field. The applied magnetic field was perpendicular to the (0001) sample surface. Transport measurements were done with a standard four-probe ac technique using a low-frequency (f ∝ 20 Hz) Lock-in amplifier. Contacts were attached with conducting graphite paste. The experiments were carried out in a temperature-variable cryostat at different values of magnetic field up to 8 T, generated by a superconducting solenoid and directed along the normal to the (0001) sample surface.

Photoemission measurements

The experiments were carried out at 1-cubed UE-112 beamline at BESSY II in Berlin, BaDElPh beamline61 at Elettra synchrotron in Trieste (Italy), BL-1, BL-9, and laser ARPES endstation at HiSOR in Hiroshima, and at Research Resource Center of Saint Petersburg State University "Physical methods of surface investigation” with a Scienta R4000 or a Specs Phoibos 150 energy analyzer. Samples were cleaved in situ at the base pressure of 6 × 10−11 mbar. The crystalline order and cleanliness of the surface were checked by low-energy electron diffraction (LEED) and X-ray photoelectron spectroscopy (XPS). Spin-resolved ARPES measurements were performed at BL-9 beamline of HiSOR synchrotron in Hiroshima and APE beamline at Elettra synchrotron in Trieste (Italy). The spectra were measured using VLEED spin detector. Total energy and angular resolutions were 20 meV and 1.5°, respectively.

Electronic structure and total energy calculations

Electronic structure calculations were performed using the density functional theory and the projector-augmented wave (PAW) method62, realized in the VASP code63,64. The exchange-correlation potential was described within the generalized gradient approximation in the Perdew–Burke–Ernzerhof formulation (GGA-PBE)65. The Hamiltonian contained scalar relativistic corrections and the SOC was taken into account self-consistently66. We used the DFT-D267 and the DFT-D368,69 approaches to describe the vdW interactions, that yielded similar results. The plane-waves with energies up to 270 eV were used for the wave functions expansion. Structural relaxations were done until the forces on atoms were smaller than 0.01 eV/Å. Depending on the particular task and geometry, different grids for the BZs sampling were used (see below), all of them being \(\overline{\Gamma }\)-centered.

The Mn 3d-states were described using the GGA + U method70 as formulated by Dudarev et al.71. The Ueff = U−J value for the Mn 3d-states was chosen to be equal to 5.34 eV, as in previous works on MnBi2Te419,20,21,25,26,72.

The bulk magnetic ordering was studied using total-energy calculations, performed for the FM and two different AFM states. Namely, we considered an interlayer AFM state and a noncollinear (NCL) intralayer AFM state, in which three spin sublattices form angles of 120° with respect to each other72. To model the FM and interlayer AFM structures in MnBi4Te7 and MnBi6Te10, we used cells with 24 and 34 atoms, respectively. For the MnBi4Te7 system, these calculations were performed with the respective 3D BZs sampled by the 25 × 25 × 5k-point grids. For the MnBi6Te10 compound, which exhibits extremely weak interlayer exchange coupling, the interlayer ordering was carefully studied using very fine k-meshes up to 23 × 23 × 19 points. These meshes were also used in the magnetic anisotropy energy calculations. For MnBi4Te7, the NCL intralayer AFM configuration was treated using hexagonal bulk cell containing three atoms per layer [\((\sqrt{3}\times \sqrt{3})R3{0}^{\circ }\) in-plane periodicity] and a 11 × 11 × 3 BZ sampling. For MnBi6Te10, NCL configuration has not been considered since all the available experimental and theoretical data in the literature indicate that each MnBi2Te4 SL block orders ferromagnetically, irrespectively of its structural environment.

The MnBi4Te7 and MnBi6Te10 semi-infinite surfaces were simulated within a model of repeating slabs separated by a vacuum gap of at least 10 Å. The interplanar distances were optimized for the utmost SL or QL block of each surface. The structural relaxations and static electronic structure calculations were performed using a k-point mesh of 11 × 11 × 1 in the 2D BZ.

The procedures of the magnetic anisotropy energy calculations and Monte Carlos simulations are described in detail in ref. 25.

Exchange coupling constants calculations

For the equilibrium atomic structure of MnBi4Te7 obtained with VASP, we calculated the Heisenberg exchange coupling constants J0,i ab initio using the full-potential linearized augmented plane waves (FLAPW) method73 in the FLEUR implementation (http://www.flapw.de). In the J0,i calculations SOC was neglected. We took the GGA+U approach74[,75 under the fully localized limit76, using similar settings as those in refs. 25,26. For the self-consistent FLAPW basis set, the plane waves were constructed with a 18 × 18 × 4 Monkhorst–Pack k-point sampling of the first BZ and cut-off energies of 3.4 Hartrees for the wavefunctions and 10.4 Hartrees for the density and potential expansions. Muffin tin sphere radius was set to 2.74 a.u., for Mn, while a value of 2.81 a.u. was used for Bi and Te, and the partial wave functions were expanded up to l = 8. Mn, Bi, and Te contributed 4s3d, 5s5p, and 6s6p valence electrons, respectively.

The J0,i constants were extracted by Fourier inversion of spin spirals in the reciprocal cell characterized by the q vectors of a 19 × 19 × 3 grid. These dispersion energies, calculated in the force theorem approach, converged below 0.1 meV and allowed adding up to 150 magnetic coordination spheres to the Fourier analysis.