Intrinsic magnetic topological insulator phases in the Sb doped MnBi2Te4 bulks and thin flakes

Magnetic topological insulators (MTIs) offer a combination of topologically nontrivial characteristics and magnetic order and show promise in terms of potentially interesting physical phenomena such as the quantum anomalous Hall (QAH) effect and topological axion insulating states. However, the understanding of their properties and potential applications have been limited due to a lack of suitable candidates for MTIs. Here, we grow two-dimensional single crystals of Mn(SbxBi(1-x))2Te4 bulk and exfoliate them into thin flakes in order to search for intrinsic MTIs. We perform angle-resolved photoemission spectroscopy, low-temperature transport measurements, and first-principles calculations to investigate the band structure, transport properties, and magnetism of this family of materials, as well as the evolution of their topological properties. We find that there exists an optimized MTI zone in the Mn(SbxBi(1-x))2Te4 phase diagram, which could possibly host a high-temperature QAH phase, offering a promising avenue for new device applications.

such as a typical size, a photograph and the fabrication process of the sample. 3) Related question: what is the layer number of the sample in Figs. 2d-2f? In particular, a ferromagnetic hysteresis is observed? 4) Figure 4e: The size of the band gap changes by increasing Sb due to the reduction of spin-orbit interaction. What is the origin of the peak at x = 0.9? Why it shows a non-monotonic behavior? 5) Although the authors claim that a topological phase transition lies at x = 0.55 from the band calculation, the samples shown in the manuscript are only x < 0.5. They should discuss the phase transition by providing ARPES or transport data for heavily Sb-doped samples. 6) Related to the above, the authors mention in Fig. 3b that "We therefore believe that the magnetic transition point from AFM to paramagnetism in Mn(Sb,Bi)2Te4 lies outside the range studied here". What is the evidence of the magnetic transition to paramagnetism by Sb-doping? The authors should provide a reference or an experimental data. The authors studied a series of Mn(SbxBi1-x)2Te4 samples, by angle-resolved photoemission spectroscopy, transport measurements and DFT calculations. ARPES and transport data show a quite convincing n-p transition, and transport data show AFM magnetic transition in all studied samples (x <= 0.5). Together with DFT calculations, the authors concluded that an ideal MTI zone can be reached by tuning the Sb/Bi ration, where topological nature, bulk carrier suppression, and spontaneous magnetization can be satisfied at the same time, and x ~ 0.3 is likely the best composition. These evidences are comprehensive. However, the main purpose of this study is to find the ideal MTI, and the authors concluded it is x ~ 0.3. But they showed almost no data for x ~ 0. 3 Similar ARPES spectra were already measured by previous papers. Especially, Fig.1 is similar to Fig.3 in arXiv 1809.07389(Ref.33), though experimental quality in Ref.33 seems to be better. The x dependence of ARPES in Fig.2 is new, but spectra are not clear. The ARPES data on x ~ 0.3 and its topological surface state is necessary.
There is no clear evidence of topological surface state in ARPES. Spin resolved ARPES is powerful in order to reveal the topological surface state. There have been already similar ARPES data in the previous papers. The authors should try spin resolved ARPES as a new experiment. Some comments: (1) For the ARPES kz data on x = 0 (Fig. 1g), is there any surface state in the second derivative plot? arXiv 1809.07389(Ref.33) observed both bulk bands and surface bands with hv = 9 eV in the second derivative. If not, what is the possible difference?
(2) According to the calculation (Fig. 1f), the bulk bands show very small kz dispersion (possibly due to the large lattice constant c). There should be no obvious kz change for bulk bands in ARPES. In Fig.  1(g), why the data from hv = 13.75 eV is quite different from others? In summary, there is no strong support for the topological nature and the related QAH for the ideal MTI (x ~ 0.3). The data on the chemical potential shift and AF magnetization with Sb substitution are solid. But these solid data are not be enough for Nature Communications.
following the reviewer's request, we provide the AFM photograph of an 8 SL sample in Fig. 4a  Comments: 2. In Page 10, the authors claim that the abnormal resistivity increase of x = 0.3 when cooling near 200 K originates from bulk carrier suppression or the canting of the magnetic moment for the transition from paramagnetism to anti-ferromagnetism. But the Neel temperature of Mn(Sb,Bi)2Te4 is between 20 to 30 K, much lower than 200K. Why the resistivity increase is related to the magnetic transition? The authors should give a more detailed explanation.
Reply: The reviewer comments that the resistivity increase in the R-T curve of x = 0.3 sample cannot result from the magnetic transition. We thank the reviewer for the comments. Here we provide the detailed explanation as follows. For the resistivity increase, we cannot exclude the possibility of some spin related origins because of two reasons. Firstly, according to previous research in some magnetic materials, the increasing of resistivity in R-T curve is reported to result from the influence of magnetic moments, although the temperature is still much higher than the critical temperature 1 . Secondly, a recent experimental report points out that the spin fluctuation in MnBi2Te4 is still considerable even at the room temperature 2 . Therefore, we cannot rule out the possibility that the resistivity increase is a spin related feature.
We apologize that we didn't make a clear explanation on this resistivity increase in the original manuscript. Following the reviewer's suggestion, more explanation is provided in paragraph 1 page 11 of the revised manuscript: "This resistivity increase possibly originates from bulk carrier suppression by fine tuning the ratio of antimony substituting. It also may be a spin related feature emerging when bulk metallic behavior is suppressed, since considerable spin fluctuation in MnBi2Te4 is reported even at the room temperature 2 , and the resistivity is possibly influenced."  Fig.   R2(a). The values of the coercive field of each kink point (Hc1 and Hc2) and the x dependence evolution are consistent with the ones in ρxx data, as seen in Fig. R2(b). In the revised manuscript, we separate the thin film device transport data from the original Fig. 3 and provide M-H curves and the extracted Hc1, Hc2 in Fig. 3b and 3d in the revised manuscript, respectively. Comments: 4. What's the sample thickness used for the transport and magnetic measurements? As MnBi2Te4 is an antiferromagnetic TI composed of ferromagnetic SLs with neighboring SLs coupled antiferromagnetically, there should be oscillation in magnetic properties as the thickness changes between even and odd SLs). Except for the 9 SL and 12 SL, the authors should show more data for the other thickness.
Reply: The reviewer asked the thickness of the sample for transport and magnetic measurements. We are sorry that we didn't make a clear explanation in the original manuscript. The transport data displayed in Figs. 2d-2f and 3b in the original manuscript were taken from the plate-like bulk crystals with the thickness of 50 ~ 200 μm, which are cleaved by a knife and the size of these plates are 0.5 to 2 mm. In samples with this kind of thickness, only bulk crystal properties can be detected. In the magnetic measurement, we use a vibrating sample magnetometer (VSM) to measure the magnetization of the samples. In order to obtain better signals, the samples for VSM measurement are much bigger and thicker, with about 0.5 ~ 1 mm in thickness and 2 ~ 3 mm in side length. In the revised manuscript, we make a clear explanation of the size of the bulk samples used in transport measurement in the Methods session. A detailed explanation of the magnetic measurement and the thin device transport measurement, including the equipment we used, the typical size of the samples and the fabrication process are also amended into the Methods session. The reviewer also suggests us to provide more data for the samples with other thickness. We agree with the reviewer that this point is important and thank for the kind advice. Following the advice, we further fabricate thin films devices with various thicknesses from 5 to 9 SLs. The transport data of Hall resistance at 2K are provided in Fig. R3(a) above. One can see that the devices with odd SLs (5,7,9 SLs) show obvious AHE with coercive field about 0.5 to 1 T. Though the hysteresis loops can also be observed in samples with even SLs (6,8 SLs), the coercive fields of these samples are much smaller than the ones with odd SLs and the remnant magnetic signals are believed to result from the canting or disorder of the AFM configurations.  As shown above, we have clarified all raised questions. We thank the reviewer for the detailed instructions, which help us to improve our manuscript greatly. We believe the manuscript has been improved and cordially wish that our revised manuscript is acceptable to Nature Communications.  MnBi2Te4. The authors may need to address the following concerns before the manuscript could be considered for publication.

Reference
Reply: First of all, we are grateful to the reviewer for his (her) positive comments on our manuscript. We also thank the reviewer for the detailed instructions and the constructive suggestions, which help us to improve our manuscript greatly. According to the advices, the relevant part in the manuscript is amended and the manuscript has been improved. The questions raised by the reviewer are answered point by point as follows.
Comments: 1) The authors achieved a p-n transition by Sb-doping to MnBi2Te4.
According to the theory, the EF-tuned samples with x between 0.25 and 0.35 should show the Axion insulator state or the QAH state depending on the layer number of the sample. Namely, both \sigma_xx and \sigma_xy should be zero in the former case and \sigma_xx = 0 and \sigma_xy should be quantized to h/e^2 in the latter case. However, the transport data in Figs. 2d and 3b look far from those quantized states. The authors should mention the reason.
Reply: The reviewer asked the reason why the transport data in Figs. 2d and 3b look far from quantization.
Here, we answer this question on two aspects.
On one hand, we fully agree with the reviewer that the quantized transport is important and it is the ultimate goal of the researches on these MTI materials.
According to the earlier research experience on topological materials, it is a long and hard course from materials to devices. For example, from the discovery of TIs in On another hand, the transport data in Fig. 2d and 3b was taken from plate-like bulk crystals with the thickness of 50 ~ 200 μm. According to the previous reports, it is so far unable to completely suppress the bulk transport contribution in large size of bulk TI crystals, even in those greatly optimized TIs such as BiSbTeSe2 and Sn-Bi1.1Sb0.9Te2S. 2,5-7 Though surface transport is dominated in thin film devices 2, 5 , the bulk contribution in these materials with hundreds of micrometers in thickness is still more than 80% 7 . Similarly, in our work, although Sb substitution successfully suppresses the bulk carrier density comparing with MnBi2Te4, there are still considerable bulk contributions in the bulk crystals. We believe this is another reason why quantized transport is unable to be observed in these crystals. We are sorry that we didn't make a clear explanation of the size of the sample in the original manuscript.
Thanks for the reviewer's comments, we provide more detailed information about the sample size and thickness in the main text and the Method session of the revised manuscript.  The reviewer also suggests us to provide more detail information about the thin film devices transport measurement. We thank the reviewer for the kind suggestions.
Following the reviewer's advice, we provide an AFM photograph of a typical 8 SL sample ( 2d-2f? In particular, a ferromagnetic hysteresis is observed?
Reply: As discussed above in the reply of comment 1, the transport data displayed in We are sorry that we didn't provide detailed information about the transport measurement and confused the reviewer. We appreciate the reviewer for the reminding. In order to avoid confusion between the measurement on bulk samples and thin film devices, we separate the thin film device transport data from the original Fig Reply: The reviewer asks why there is a peak for the calculated bandgap at x = 0.9.
Thanks for the reviewer's comment. Here we provide more detailed explanations. We think that the origin of this peak is the competition of two effects. The first is the spin orbit coupling (SOC) effect. We know that the SOC reduces with increasing Sb. The energy gap expects to enlarge by reducing the SOC. In Fig. R2(d), we calculate the energy gap with different SOC for MnBi2Te4, which really presents that the energy gap becomes larger with reducing the SOC. The second is the chemical bonding effect.
To see this effect, we calculate the band structures of MnSb2Te4 and MnBi2Te4 by setting the SOC to be zero, (or saying without the SOC), respectively, shown in Fig   unable to be detected. Therefore, according to the recent report, we believe it is unable to experimentally identify the topological nature in heavily Sb-doped samples by either ARPES or transport. Thanks for the reminding of the reviewer, we also cite this recent arXiv report in our revised manuscript as a reference.
Comments: 6) Related to the above, the authors mention in Fig. 3b that "We therefore believe that the magnetic transition point from AFM to paramagnetism in Mn(Sb,Bi)2Te4 lies outside the range studied here". What is the evidence of the magnetic transition to paramagnetism by Sb-doping? The authors should provide a reference or an experimental data.
Reply: Thanks for the reviewer's reminding. We believe the reviewer misunderstands our meaning here. We are sorry for the inappropriate expression of this sentence which is easy to be misunderstood. In fact, we would like to express that although the AFM exchange coupling is reduced by Sb substituting, there is still no magnetic transition from AFM to paramagnetism within the range of x < 0.5 in Mn(SbxBi1-x)2Te4. In fact, in the recent arXiv report mentioned above (arXiv: 1905.00400), the magnetism of MnSb2Te4 has been investigated. Though the Neél temperature decreases to 19 K, the spontaneous AFM order in MnSb2Te4 still maintains. That is to say, in the whole range of x = 0 to 1, there is no magnetic transition from AFM to paramagnetism in Mn(SbxBi1-x)2Te4. Thanks for the mention from the reviewer. This sentence has been amended in the revised manuscript as: "We therefore believe that there is no magnetic transition from AFM to paramagnetism and the spontaneous magnetization is maintained in Mn(SbxBi1-x)2Te4 within the range of x Reply: We apologize for the low-contrast figure in the original manuscript. Thanks for the reviewer's advice. We adjust the colors of Figure 2b in the main text, and each EDS curve is shifted for clearness, the modified figure is shown in Fig. R3 below. We thank the reviewer for this useful suggestion to improve our manuscript.  To summarize, we sincerely thank the reviewer for the detailed instructions.
Following his/her suggestions and comments, we provide more detailed information about our experimental measurements, which helps the readers to better understand our work. Some inappropriate expressions and low-contrast figures are also amended.
We think that all the suggestions and comments have been properly dealt in this revision, and the manuscript has been improved. We cordially wish the present version of the manuscript is satisfying for the acceptance.  As discussed above, for thin film devices, x ~ 0.1 is a more appropriate substituting ratio than x ~ 0.3. We thank the reviewer for the detailed instructions, which guide us to further investigate the thin film devices and find more new results.
We discuss the thin film devices in more detail in the revised manuscript. We provide more detailed devices transport data and re-organize them into Fig. 4 of the revised manuscript. The phase diagram in Fig. 5e (4e in the original manuscript) is also amended to add the information of the optimized range of x for thin film devices. The R-Vg curves for different Mn(SbxBi1-x)2Te4 thin film devices displayed in Fig. R1 above are also provided in the Supplementary Information.
Comments: Similar ARPES spectra were already measured by previous papers.
Especially, Fig.1 is similar to Fig.3  previously. We make intense modifications on the related part in the revised manuscript and we sincerely thank the guidance from the reviewer. Below, we will discuss on these new results in detail.
In the new round of ARPES measurement, the condition of the equipment is better and allows us to use a lower temperature of 8 K to collect the spectra, and the time spending for signal integration is longer to collect the spectra with better quality.
Especially, smaller angle step is used when looking for the position of Г point. Since  In fact, we also find a recent experimental report discussing about thin film MnBi2Te4 devices transport. In fabricated MnBi2Te4 devices, the measured exchange gap at B = 12 T is only = 21 K detected by transport measurement which is much smaller than the one expected by calculation, and the exchange gap is even smaller under zero magnetic field than under B = 12 T. 1 The transport result in this report is contrast to the big gap opening with dozens of meV observed in our previous ARPES measurement. Thus, our new ARPES result with no observable bandgap at the Dirac point provides a new understanding on this material and reflects a more realistic picture of the surface states in real synthesized MnBi2Te4 crystals.
In addition, the reviewer also points out the poor quality of the ARPES data in samples with different x values and requests the ARPES data of the surface states on x = 0.3. We fully agree with the reviewer that the high-quality ARPES data is important.
According to the request, we further precisely optimize the crystal growth parameters and achieve single crystals with x = 0.2, 0.3 and 0.4 with larger crystal sizes and flatter surfaces, which are better for ARPES measurement. Then, we use the ARPES equipment located at the Shanghai Synchrotron Radiation Facility to re-collect the ARPES spectra of these crystals. The collection of the photoelectron is much efficient, and the intensity of the spectra we get is almost two orders of magnitude larger than the previous spectra displayed in Fig. 2c in the original manuscript. The new ARPES data are displayed in Fig. 2c of the revised manuscript (Fig. R3 below). The Fermi level tuning by x values can be clearly identified in these new data. For the sample of x = 0.3, due to the higher-quality data, the Fermi level of this sample can be observed much clearer. One can see that the Fermi level is just above the bulk valence band, and the surface state in the band gap is just unable to be detected by ARPES. Reply: Thanks for the reviewer's reminding. We strongly agree with the reviewer that the spin ARPES is a powerful technique to identify the surface state by analyzing the spin-momentum lock texture. However, the intensity of the surface states in MnBi2Te4 is sensitive to the photon energy and can be only clearly detected at hv ~ 7 eV. We feel very sorry that a spin ARPES equipment which can work under this range of photon energy is beyond the experimental resource we can obtain. Thus, unfortunately, we are unable to satisfy the reviewer's request for spin ARPES measurement.
Alternatively, following the reviewer's suggestion, we try to find other evidence of the topological surface states. As discussed above, we re-collect the ARPES data of MnBi2Te4 under 7.25 eV more precisely, and data with better quality is achieved. We the time for collecting this spectrum is too short to achieve enough intensity of signals from surface states, because the surface states intensity is weak at 9 eV and much longer time for signal integral is needed. In addition, from our ARPES measurement, we notice that the surface states intensity changes dramatically near the photon energy of ~9 eV. Thus, slight disagreement on the photon energy may also cause the relatively big difference on the intensity of the surface states. We believe these two points mentioned above are the possible reasons causing the difference between the two spectra from two research groups. Anyway, though the quality of the ARPES spectrum under 9 eV in the reference is better than ours, these two results are still qualitatively consistent with each other. Comments: (2). According to the calculation (Fig. 1f), the bulk bands show very small kz dispersion (possibly due to the large lattice constant c). There should be no obvious kz change for bulk bands in ARPES. In Fig. 1(g), why the data from hv = 13.75 eV is quite different from others?
Reply: The reviewer feels confused about the large kz dispersion detected by ARPES and the unique spectra at hv =13.75eV comparing with other photon energies. We thank the reviewer for the comments. Following the comment, we carry out further first-principle calculations and provide more data on the ARPES measurement. Here, we respond the reviewer from two aspects as follows.
Firstly, from our experimental results, we emphasize that kz dispersion is exactly the reason why the spectra at hv = 13.75 eV looks different from the ones under other photon energies. For more detailed information of the ARPES measurement, please look at Fig. R5 displayed below. We provide more ARPES data under different photon energies from 11 eV to 17 eV. The EDC at k∥ = 0 (solid lines) and the corresponding peak positions of the conduction band minimum and the valance band maximum (dashed lines) are also displayed. One can clearly see the evolution of the conduction and valence band under different photon energies. When hv = 13.75 eV, the detected bulk bandgap is smaller than other photon energies in Fig. 1(g) (9, 11, 15, eV). This is the reason why the data at hv = 13.75 eV in Fig. 1g of the main text looks unique.
From Fig. R5 we find that kz dispersions of valence band is about 0.05 eV at k∥ = 0, which is much obvious than the relatively small kz dispersion of the conduction band.   Fig. 1(d). The bulk gap should reduce with Sb substitution.
How to explain this contradiction?
Reply: We thank the reviewer for the comments. As discussed above, higher-quality ARPES spectra of the samples with x = 0 and 0.2 are obtained, as shown in Fig. R7.
According to the EDC extracted at k∥ = 0 from our new ARPES data, the bulk bandgap of pure MnBi2Te4 and Mn(Sb0.2Bi0.8)2Te4 are similar (~160 meV) as shown in Fig. R7 below, there are no observable difference between these two samples considering the resolution limit of our ARPES measurement (~10 meV). In the previous ARPES data with lower quality, a cone shaped band marked by the red dashed line in Fig. 2c(v) is considered to be the surface state, however, according to our new ARPES results. This band in Fig. 2c(v) in the previous manuscript is more likely to be the bulk valence band, thus the bulk bandgap measured in Fig. 2c(   Fig. 4(e). Why the AHE curves of even layers show such a strange shape? Why the AHE of 8 SL changes sign from negative to positive? The authors should give an explanation for this before publication.
Reviewer #2 (Remarks to the Author): Because the questions/comments we asked are well addressed, we recommend the publication with no further comments for the authors. The biggest concern of the reviewer 3 is that the topological properties of the materials (especially the x=0.3 compound) are not sufficiently characterized. And the reviewer suggests to carry out spinresolved ARPES or show transport results with quantized Hall resistance.
I partially agree with the reviewer. Fig. 2c clear demonstrates the chemical potential shift with various x. However, it would be great, if the author can present better ARPES data with clear surface bands in the samples with x = 0.2 or 0.25. These data will be strong evidence for the success of preserving the topological properties of the samples during the control of doping level. To obtain better signal of the surface bands, it might be helpful to select the photon energy at 7 eV rather than 14 eV.
On the other hand, I think that the manuscript has shown a good approach to get intrinsic magnetic topological insulator. The authors already demonstrate an efficient way to fine tune the chemical potential and magnetism of the 124 compound. I understand that it will be a long way to realize QAH in transport measurements. At the current stage, the results are enlightening and important.
One suggestion: The absence of gap opening at the Dirac point might also originate from the change of magnetic moments on the surface of Mn-124. The author may want to add this possibility to the main text. For the strange shape of the AHE curves of even layers, we believe there are two possible origins. Firstly, we need to mention that for the data in Fig. 4 Following the reviewer's advice, a new statement "Since the Fermi level is close to the charge neutral point, the reversal of AHE signals in Fig. 4e may come from the competition between the intrinsic Berry curvature and Dirac-gap enhanced extrinsic skew scattering in this material." has been added into the manuscript.
To summarize, we thank the reviewer for the detailed instructions. We sincerely hope that the reviewer is satisfied with our replies. Thanks to his/her nice suggestions.
raised by the reviewer.
The biggest concern of the reviewer 3 is that the topological properties of the materials (especially the x=0.  (Fig. R1(b)). Then we try to change the incident photon energy a little, using 8.4 eV (xenon lamp as the light source) to collect the spectra of x = 0.2 sample, as shown in Fig. R1(c). One can see that the spectrum changes greatly from 7.25 eV to 8.4 eV. This kind of big change unlikely comes from the bulk states since the kz dispersion in this family of material is small, and the spectra in Fig. R1(c) is  We admit that the quality of the spectra in Fig. R1(c) is low and hard to distinguish the surface dispersions. We believe it is due to the relatively low luminous flux of the xenon lamp, and 8.4 eV may still not the optimal photon energy to observe the surface dispersion in x = 0.2. More detailed and systematical measurements based on the synchrotron radiation ARPES equipment is needed for the further investigation on the surface band dispersions in x = 0.2 samples. Unfortunately, due to the equipment maintenance recently and the limited machine-hour, it will take more than three months for us to access the synchrotron radiation ARPES sources. However, we think the ARPES data of 8.4 eV is a convincing evidence to prove the existence of the surface states at x = 0.2 sample. We truly wish the existing data can meet the reviewer's requirements.
Comments: On the other hand, I think that the manuscript has shown a good approach to get intrinsic magnetic topological insulator. The authors already demonstrate an efficient way to fine tune the chemical potential and magnetism of the 124 compound. I understand that it will be a long way to realize QAH in transport measurements. At the current stage, the results are enlightening and important. To summarize, we sincerely thank the reviewer for the detailed instructions. We think that all the suggestions and comments have been properly dealt in this revision.
We cordially wish the present version of the manuscript is satisfying for the acceptance.