Introduction

Three-dimensional topological insulators such as Bi2Te3 and Sb2Te3 are established to have a topologically protected Dirac-like surface state with linear dispersion1,2. Infusing these materials with magnetism breaks the time-reversal symmetry and opens a magnetic exchange gap at the Dirac point that gives rise to a plethora of exotic quantum states, such as the quantum anomalous Hall (QAH) effect, which can be exploited in quantum electronic devices3. The experimental signature of the QAH state has first been observed experimentally in chromium or vanadium-doped topological insulators of the Bi2-xSbxTe3 (BST) class4,5,6,7. In these compounds, the random distribution of a few percent of magnetic dopants opens a magnetic exchange gap, while Sb substitution allows tuning the charge carrier density so that the Fermi energy \({E}_{{{{{{\rm{F}}}}}}}\) is located in vicinity to the exchange gap8,9. However, the nature of the randomly distributed magnetic dopants in these compounds leads to variations in the exchange gap size, as well as the doping level throughout the material, both of comparable magnitude8,9. In transport experiments, these variations result in the formation of charge puddles, which inherently limit the device performance9. As a result, the QAH effect can only be observed at ultra-low temperatures (\(\lesssim 1\,{{{{{\rm{K}}}}}}\)), i.e. much lower than the materials’ magnetic transition temperatures (\(\sim 10\,{{{{{\rm{K}}}}}}\)) and the achieved local exchange gap sizes (\(\sim 100\,{{{{{\rm{K}}}}}}\))9. Recently, it was discovered that stoichiometric MnBi2Te4 (MBT) is an intrinsic magnetic topological insulator in which Mn atoms are embedded as uniform layers in the crystal structure, forming a natural heterostructure10,11,12,13,14,15,16,17,18. The resulting exchange gap is predicted to be larger than in magnetically doped TIs and thus supposed to be more resilient to inhomogeneities12,14. Similar to Bi2Te3, the charge carrier concentration in MBT can be tuned by Sb substitution, resulting in MnBi2-xSbxTe4 (MBST), with charge neutrality achieved at \(x\approx 0.64\) (Refs. 19,20). As a result, the compensated compound MnBi1.36Sb0.64Te4 is a promising candidate to overcome the technical limitations of magnetically doped BST. However, the experimentally observed surface state gaps in both MBT and MBST are varying widely from tens of meV all the way down to zero14,19,20,21,22,23. A likely reason for this discrepancy is the effect of intrinsic defects, such as vacancies and anti-sites, which give rise to local variations in the electronic structure. However, the roles of the different defect types are not yet fully understood. In this letter, we elucidate the influence of intrinsic defects on the local electronic structure of compensated MnBi1.36Sb0.64Te4 bulk crystals by scanning tunnelling microscopy/spectroscopy (STM/S). While we find that substitutional Sb atoms are incorporated uniformly and are an effective way to tune the Fermi level, the inhomogeneous distribution of (Bi,Sb)Mn anti-site defects gives rise to significant doping fluctuations, which close the transport gap.

Results

Topographic images of the MBST surface (Fig. 1a and b) display atomically flat surfaces of hexagonal atomic structure with lattice constant \(a=4.3\,{{{{{\text{\AA }}}}}}\) corresponding of the topmost Te layer24. Different kinds of defects are evident on the surface and can be identified based on the crystal structure and earlier reports on pure MnBi2Te4 (Refs. 25,26), as shown schematically in Fig. 1c. In particular, Mn(Bi/Sb) anti-sites are the dominant type of defect observed on the sample surface, which result in dark triangular features centred on the Bi/Sb atoms in the second layer (Fig. 1b). These features are similar to previous STM studies of Mn-doped Bi2Te3, where Mn atoms replace Bi atoms in the second layer, resulting in the three covalently bonded Te atoms above the substitutional atom to show a decreased topographic height27. We further observe (Bi/Sb)Te anti-sites sparsely, which replace a chalcogen atom in the topmost layer25,28 and appear as bright circular spots on the sample surface (Fig. 1a). (Bi,Sb)Mn anti-sites are another type of defect reported frequently in the literature. However, because of their location further below the surface, in the fourth atomic layer, they result in a smaller contrast in the STM topography25, which makes them hard to distinguish, especially if the defect concentration is high. Furthermore, we find that the locations of Bi and Sb atoms cannot be distinguished because their similar valence electron structures do not give rise to a significant STM contrast. The concentrations of Mn(Bi/Sb) and (Bi/Sb)Te anti-sites are estimated from topographic images to be 9.1% and 0.46%, respectively. These defect concentrations are higher than in pure MnBi2Te429 but lower than in pure MnSb2Te430 grown by the same group. This observation is in agreement with theoretical calculations that showed a lower Mn(Bi/Sb) defect formation energy in MnSb2Te4 than in MnBi2Te431. Fig. 1d shows a spatially averaged tunnelling spectrum of the MBST sample surface, which is in excellent agreement with a recent photoemission study of similar MBST crystals32. We can readily identify the bulk band gap, which is comparable to that of MBT19,33. However, in contrast to pure MBT, which is typically electron-doped with the Fermi energy \({E}_{{{{{{\rm{F}}}}}}}\) in the conduction band, in our MBST sample \({E}_{{{{{{\rm{F}}}}}}}\) is located at the top of the bulk valence band. Inside the bulk band gap, we observe the gapped surface state, which is described by the dispersion of a massive Dirac cone as \({E}_{\pm }\left(k\right)={E}_{{{{{{\rm{D}}}}}}}\pm \sqrt{{\left(\hslash {v}_{{{{{{\rm{F}}}}}}}k\right)}^{2}+{\Delta }^{2}}\), where \({v}_{{{{{{\rm{F}}}}}}}\) is the Fermi velocity, \({E}_{{{{{{\rm{D}}}}}}}\) is the Dirac point energy, i.e. the doping level, and Δ is the magnetic exchange gap9. The corresponding surface state valence and conduction band edges are \({E}_{{{{{{\rm{VB}}}}}}}={E}_{{{{{{\rm{D}}}}}}}-\Delta\) and \({E}_{{{{{{\rm{CB}}}}}}}={E}_{{{{{{\rm{D}}}}}}}+\Delta\), respectively.

Fig. 1: Surface defects and electronic spectrum of MBST.
figure 1

a, b STM images showing intrinsic defects in large-scale and atomic resolution scans. c Atomic models of typical intrinsic defects in MBST viewed from the top (top panel) and side (bottom panel). d Typical tunnelling spectrum of MBST averaged over an area of 50 × 50 nm2, plotted on a logarithmic scale, as well as a linear scale (upper inset). The flat region in the inset corresponds to the bulk band gap. In the log-scale plot, the surface state inside the bulk band gap is evident and shows another gap - the surface state exchange gap. A corresponding band schematic is shown as the left inset. We estimate the bulk band gap (\(\approx 200\,{{{{{\rm{meV}}}}}}\)) and surface state exchange gap (\(2\Delta \approx 56\,{{{{{\rm{meV}}}}}}\)) from linear extrapolation of the band edges (black and red dashed lines, respectively) to the measurement noise level (horizontal dotted line), similar to Refs. 33,47.

Full size image

In the large area topographic images (Fig. 1a) we also observe long-range height variations indicative of local doping fluctuations. Such variations can not only be caused by the defects evident on the sample surface but also by subsurface defects, such as (Bi/Sb)Mn anti-sites25 and the random distribution of the Sb dopants. Because Sb dopants and Mn(Bi/Sb) anti-sites both induce hole doping, while (Bi/Sb)Mn induces electron doping25, a revelation of the exact roles for each dopant requires further investigations. However, in contrast to pure MBT, where defect concentrations can be low enough to enable the identification of isolated defects, partial Sb substitution leads to significant defect densities, which results in an overlap of the effects of neighbouring defects. This problem is aggravated by the extended nature of (Bi/Sb)Mn anti-site features in STM, stemming from their location three layers below the sample surface. As a result, we find that atomic-scale variations in the electronic properties are small compared to the long-range variations that result from regions of different concentrations of particular defect types (see Supplementary Note 1). To resolve this issue, we use a statistical approach to analyse the influence of the defects onto the electronic properties of MBST in the following.

Statistical analysis of the electronic properties

To assess Δ and \({E}_{{{{{{\rm{D}}}}}}}\) experimentally, we record \({dI}/{dV}\) maps in a smaller energy range, corresponding to the surface state, where we find significant variations in the individual spectra (Fig. 2a, b). At each pixel of the \({dI}/{dV}\) map, we extract the valence band onset \({E}_{{{{{{\rm{VB}}}}}}}(x,y)\) and conduction band onset \({E}_{{{{{{\rm{CB}}}}}}}(x,y)\) as the energy at which the \({dI}/{dV}\) signal drops below a fixed threshold value (see Methods section for details). The resulting maps are shown in Fig. 2c, d. It is evident that there are regions that are more hole-doped (red) and electron-doped (blue) in similar locations in \({E}_{{{{{{\rm{VB}}}}}}}\left(x,y\right)\) and \({E}_{{{{{{\rm{CB}}}}}}}\left(x,y\right)\). When plotting the histogram of Fig. 2c and d, we find that both band edges exhibit variations in the range of \(\sim 50\,{{{{{\rm{meV}}}}}}\) (Fig. 2e). By fitting Gaussian distributions\(\, \sim \exp \left(-\frac{{\left(E-{\tilde{E}}\right)}^{2}}{2\sigma^{2} }\right)\), we find the average positions of the valence and conduction band edges to be \({\tilde{E}}_{{{{{{\rm{VB}}}}}}}=\tilde{E}\pm \sigma =(116\pm 24)\,{{{{{\rm{meV}}}}}}\) and \({\tilde{E}}_{{{{{{\rm{CB}}}}}}}=(172\pm 20)\,{{{{{\rm{meV}}}}}}\), respectively. Because the histograms are almost perfectly normal distributed, the fit errors for \({\tilde{E}}_{{{{{{\rm{VB}}}}}}}\) and \({\tilde{E}}_{{{{{{\rm{CB}}}}}}}\) are only \(\sim 0.06\,{{{{{\rm{meV}}}}}}\). On average, the two band edges have a separation of \({\tilde{E}}_{{CB}}-{\tilde{E}}_{{VB}}=2\tilde{\Delta }\approx 56\,{{{{{\rm{meV}}}}}}\) with respect to each other, which is consistent with the exchange gap observed in the averaged tunnelling spectra (Figs. 1d and 2b). The fact that there is an overlap between the conduction and valence band distributions in Fig. 2e necessitates the presence of charge puddles, effectively closing the transport gap34. To further analyse if this band overlap results from a local closing of the gap or comes from the varying doping level across the surface, we extract the local gap size \(\Delta (x,y)=({{E}_{{{{{{\rm{CB}}}}}}}(x,y)-E}_{{{{{{\rm{VB}}}}}}}(x,y))/2\) and present the map in Fig. 2f, where we find similar spatial variations as for the band edges. A Gaussian fit to the histogram of the extracted gap sizes (Fig. 2g) results in an average exchange gap of \(\tilde{\Delta }=27.80\,{{{{{\rm{meV}}}}}}\), with fit error \(0.06\,{{{{{\rm{meV}}}}}}\). The standard deviation of the gap is \({\sigma }_{\Delta }=9.70\,{{{{{\rm{meV}}}}}}\), with a vanishing number of data points for which the gap is fully closed. This observation indicates no significant presence of regions with such high defect density that the surface gap collapses35. The smaller standard deviation of the gap size (\(\sim 10\,{{{{{\rm{meV}}}}}}\)) compared to the propagated standard deviation of the band edges (\(\sqrt{{(20\,{{{{{\rm{meV}}}}}})}^{2}+{(24\,{{{{{\rm{meV}}}}}})}^{2}}/2\approx 16\,{{{{{\rm{meV}}}}}}\)) indicates that the observed variations in Fig. 2c, d are predominantly due to different local doping, which does not primarily alter the exchange gap size. By performing a Fourier transformation \(\Delta \left(x,y\right)\to \widehat{\Delta }({k}_{x},{k}_{y})\) we find that the exchange gap is uniform in k-space (Fig. 2h). This is in contrast to magnetically doped BST in which the gap resembles a warped Dirac cone with hexagonal symmetry9.

Fig. 2: Extraction of the surface state doping level and gap size.
figure 2

a Large-area STM topography showing long-range fluctuations on the sample surface. b Variation of the surface state spectra along the line indicated in (a) and their average. The band edge positions \({E}_{{{{{{\rm{VB}}}}}}}\) and \({E}_{{{{{{\rm{CB}}}}}}}\) are indicated. c, d Map of the variation of \({E}_{{{{{{\rm{VB}}}}}}}\) and \({E}_{{{{{{\rm{CB}}}}}}}\) in the same area as shown in (a). e Histogram of the data shown in (c, d). Gaussian fits to the peaks result in band edge positions \({\widetilde{E}}_{{{{{{\rm{VB}}}}}}}=(116\pm 24)\,{{{{{\rm{meV}}}}}}\) and \({\widetilde{E}}_{{{{{{\rm{CB}}}}}}}=(172\pm 20)\,{{{{{\rm{meV}}}}}}\). f Map of the extracted surface state gap \(\Delta =({E}_{{{{{{\rm{CB}}}}}}}-{E}_{{{{{{\rm{VB}}}}}}})/2\). g Histogram of (f) with Gaussian fit resulting in \(\widetilde{\Delta }=(28\pm 10)\,{{{{{\rm{meV}}}}}}\). h Fast Fourier transformation of (f) showing a uniform gap in k-space. Color bar units are given in the panel titles.

Full size image

To test the hypothesis of mostly rigid band shifts by Sb doping, we analyse the dependence of the gap size on the band edge positions for each data pixel. When plotting \({E}_{{{{{{\rm{CB}}}}}}}\) as a function of \({E}_{{{{{{\rm{VB}}}}}}}\) (Fig. 3a), we find that the data is described well by the equation \({E}_{{{{{{\rm{CB}}}}}}}={E}_{{{{{{\rm{VB}}}}}}}+56\,{{{{{\rm{meV}}}}}}={E}_{{{{{{\rm{VB}}}}}}}+2\tilde{\Delta }\). This linear dependence of \({E}_{{{{{{\rm{CB}}}}}}}\) on \({E}_{{{{{{\rm{VB}}}}}}}\) with a slope of unity indicates mostly rigid band shifts, which is expected for local changes in the charge doping of the crystal, such as with varying amounts of Sb doping. We also observe a group of data points deviating from this overall trend (highlighted with red ellipse in Fig. 3a), which can be ascribed to an effect of (Bi/Sb)Mn defects, as discussed below. Plotting Δ as function of the Dirac point energy \({E}_{{{{{{\rm{D}}}}}}}=({E}_{{{{{{\rm{VB}}}}}}}+{E}_{{{{{{\rm{CB}}}}}}})/2\), results in a relatively small correlation coefficient (\(r=-0.255\)), which indicates a disorder-induced variation of the gap independent of the doping level. Nevertheless, an overall trend is discernible, which can be described by a linear fit \(\Delta \left({E}_{{{{{{\rm{D}}}}}}}\right)=-0.061\cdot {E}_{{{{{{\rm{D}}}}}}}+23\,{{{{{\rm{meV}}}}}}\) (Fig. 3b). This trend of the gap size can be explained by an increase in the number of Mn(Bi/Sb) defects with increasing Sb content, which decreases the surface gap due to ferrimagnetic ordering19,21,30. Although the gap size in the observed range of \({E}_{{{{{{\rm{D}}}}}}}\) varies by \(\sim 40 \%\), it has overall weak dependence on the doping level, allowing us to conclude that Sb doping realizes an effective tuning of \({E}_{{{{{{\rm{F}}}}}}}\) near x=0.64. From the observed trend, we expect that it will be possible to tune \({E}_{{{{{{\rm{F}}}}}}}\) to mid-gap while simultaneously increasing the gap size by slightly decreasing the Sb content. At mid-gap condition, the predicted average gap size corresponds to the offset of the fit (\(\Delta =23\,{{{{{\rm{meV}}}}}}\)). We note that for even lower Sb concentration (\(x\le 0.2\)), it was reported that the gap increases with increasing Sb content36. In combination with our results, this could indicate that there is an optimum composition in the range \(0.2 < x < 0.64\) for which the surface gap is maximized.

Fig. 3: Statistical analysis of doping effect and surface state gap.
figure 3

a \({E}_{{{{{{\rm{CB}}}}}}}\) as function of \({E}_{{{{{{\rm{VB}}}}}}}\) for each pixel of Fig. 2c, d. The correlation coefficient of the data is \(r=0.639\). The indicated line of \({E}_{{{{{{\rm{CB}}}}}}}={E}_{{{{{{\rm{VB}}}}}}}+56\,{{{{{\rm{meV}}}}}}\) corresponds to isoelectronic doping and describes the data well. The red ellipse highlights a cloud of outliers resulting from (Bi/Sb)Mn defects. The black arrow indicates the axis of increasing surface state gap Δ. b The surface state gap as a function of the Dirac point energy \({E}_{{{{{{\rm{D}}}}}}}=({E}_{{{{{{\rm{VB}}}}}}}+{E}_{{{{{{\rm{CB}}}}}}})/2\) shows only a small slope and little correlation (\(r=-0.255\)), indicating a weak dependence of the two quantities.

Full size image

Quasiparticle interference

To analyse the effect of the disorder in more detail, we study the electronic surface structure of the MBST by quasiparticle interference (QPI) measurements. QPI patterns are calculated from the dI/dV maps by Fourier transformation \({dI}/{dV}(x,y,E)\to \widehat{{dI}/{dV}}({q}_{x},{q}_{y},E)\), where \({q}_{x},{q}_{y}\) are aligned with high symmetry axes of the atomic lattice. Fig. 4a, b show the dI/dV maps and QPI patterns, at energies corresponding to the bulk valence band (\(E=-150\,{{{{{\rm{meV}}}}}}\)), the surface state just below the exchange gap (\(E=50\,{{{{{\rm{meV}}}}}}\)) and the bulk conduction band (\(E=220\,{{{{{\rm{meV}}}}}}\)), respectively. We note that the dI/dV intensity fluctuations at the Dirac cone energy and the conduction band energy are qualitatively roughly opposite to each other and show a similar pattern as the doping fluctuations evident in Fig. 2c, d. In the QPI pattern of the bulk bands, we see broad features, owing to the dispersion of the corresponding bands19,20,26. In contrast, the surface state QPI pattern shows a smaller Dirac cone signature that is similar to non-magnetic BST37,38,39. However, the Dirac cone warping observed in our QPI data is smaller than in BST, in agreement with photoemission experiments on MBST20. In Fig. 4c we plot the quasiparticle dispersion \(E(q)\) by taking cross sections of the QPI patterns at each energy along the \(\bar{\Gamma }-\bar{{{{{{\rm{M}}}}}}}\) and \(\bar{\Gamma }-\bar{{{{{{\rm{K}}}}}}}\) directions and stacking them on top of each other. The bulk band gap and gapped Dirac cone of the surface state is visible in the energy range \(E=[-50\,{{{{{\rm{meV}}}}}},160\,{{{{{\rm{meV}}}}}}]\) (indicated as red dashed lines in Fig. 4c). Extracting the Dirac cone group velocity from its QPI dispersion, we find that it agrees well with photoemission spectroscopy of MBST (Refs. 40). Furthermore, the QPI intensity of the surface state is suppressed in the energy range \(E=\left[100\,{{{{{\rm{meV}}}}}},160\,{{{{{\rm{meV}}}}}}\right]\), in agreement with the presence of the exchange gap19.

Fig. 4: Quasiparticle interference mapping.
figure 4

a dI/dV maps measured in the surface region shown in Fig. 2a at three energies, corresponding to bulk valence band (\(E=-150\,{{{{{\rm{meV}}}}}}\)), bulk conduction band (\(E=220\,{{{{{\rm{meV}}}}}}\)) and surface state (\(E=50\,{{{{{\rm{meV}}}}}}\)). b QPI maps calculated from (a). The middle panel inset is a zoom into the surface state, showing the warped Dirac cone. c Quasiparticle dispersion resulting from stacking of QPI maps with indicated gapped Dirac cone (red dashed lines) and its gap (black dotted lines).

Full size image

To analyse the effect of the defects on the QPI patterns, we scrutinize regions of different local doping by classifying the dI/dV map into two subsets where \({E}_{{{{{{\rm{D}}}}}}} < 150\,{{{{{\rm{meV}}}}}}\) and \({E}_{{{{{{\rm{D}}}}}}}\ge 150\,{{{{{\rm{meV}}}}}}\), respectively, and analyzing them separately. We perform the same QPI analysis as described above for each of the two regions (red and blue areas in Fig. 5a). This approach is viable because it has been shown that even sparsely sampled dI/dV maps are sufficient to calculate representative QPI patterns41. We call this technique subset-QPI (S-QPI) in the following. Fig. 5b and c show the S-QPI dispersions and the averaged spectra for each region, respectively. We find that the dispersion of the two regions are overall similar, but shifted in energy by \((20\pm 7)\,{{{{{\rm{meV}}}}}}\) with respect to each other, owing to the more electron-doped character of region A compared to region B. In the S-QPI dispersion of region A, we further observe an additional feature in \(\bar{\Gamma }-\bar{{{{{{\rm{M}}}}}}}\) direction at \(q\approx 3\,{{{{{{\rm{nm}}}}}}}^{-1}\), which is also clearly reflected in the corresponding average spectrum as a shoulder at \(E\approx 180\,{{{{{\rm{meV}}}}}}\) (red arrow in Fig. 5b). This defect state is known to result from subsurface (Bi/Sb)Mn defects25 and we thus attribute additional counts in this energy range in Figs. 2e and 3a to this defect state. In contrast, this defect state is not observed in region B, and consequently the S-QPI displays a more ideal dispersion of the gapped Dirac cone. We conclude that the more electron-doped region A has a significantly higher concentration of (Bi/Sb)Mn defects than region B, which is in line with the reported local electron doping resulting from such defects25.

Fig. 5: Quasiparticle dispersions of regions of different doping level.
figure 5

a Classification of areas that are more electron-doped (region A, blue, \({E}_{{{{{{\rm{D}}}}}}} \, < \,150\,{{{{{\rm{meV}}}}}}\)) or hole-doped (region B, red \({E}_{{{{{{\rm{D}}}}}}}\ge 150\,{{{{{\rm{meV}}}}}}\)) superimposed on the topography from Fig. 2a. b, c Quasiparticle dispersions and average spectra of regions A and B, respectively. Horizontal dotted lines indicate the surface state gaps, respectively.

Full size image

Disentangling the defect roles

To further analyse the details of the surface state gap variations as function of \({E}_{{{{{{\rm{VB}}}}}}}\) and \({E}_{{{{{{\rm{CB}}}}}}}\), we extract the slope and intercept of the linear fit shown in Fig. 3a with respect to the indicated Δ-axis, resulting in \(\Delta \left({E}_{{{{{{\rm{VB}}}}}}}\right)=-0.239\cdot {E}_{{{{{{\rm{VB}}}}}}}+0.056\,{{{{{\rm{meV}}}}}}\), and \(\Delta \left({E}_{{{{{{\rm{CB}}}}}}}\right)=0.109\cdot {E}_{{{{{{\rm{CB}}}}}}}+0.09\,{{{{{\rm{meV}}}}}}\), respectively. As such, the data indicates that the gap tends to close from the valence band side because of the higher slope there. This observation is in agreement with the combination of the (Bi/Sb)Mn defect state being located at the conduction band edge, at \(E\approx 180\,{{{{{\rm{meV}}}}}}\), and the electron doping caused by these defects. As a result, these two effects conspire to decrease the transport gap in the vicinity of high (Bi/Sb)Mn defect concentrations, ultimately leading to the manifestation of charge puddles, when the Fermi energy is tuned into the surface state gap. However, such regions only make up a small fraction of the total number of data points (highlighted in Fig. 3a) and we find that the average gap sizes of region A and region B differ by \((3.9\pm 0.2)\,{{{{{\rm{meV}}}}}}\) while the average doping difference is \((24.4\pm 0.9)\,{{{{{\rm{meV}}}}}}\) (Fig. 5b, c and Supplementary Note 2), consistent with our results above. Since the surface state gap in MBT was shown to critically depend on the local concentration of Mn(Bi/Sb) defects21, the concentration of this type of defect is expected to be comparable in region A and B, i.e. independent of the (Bi/Sb)Mn defect distribution. We confirm this behaviour by directly estimating the defect concentration in the two regions from topography maps, where we find 9.25% and 9.02% Mn(Bi/Sb) defects for region A and B, respectively (see Supplementary Note 3). In addition, because the concentration of Mn(Bi/Sb) defects depends on the Sb content of the crystal30, we expect the Sb concentration in the two regions to be similar as well. As a result, the observed variation in the doping level of the two regions must result from the different (Bi/Sb)Mn defect concentrations (see Supplementary Note 3 for further discussion).

Last, we observe that the overall intensity of the Dirac cone in the S-QPI data of region A (Fig. 5b) is lower than in region B (Fig. 5c). Since it was shown that the surface state depth profile critically depends on the defect concentration21,42,43, we speculate that the higher concentration of (Bi/Sb)Mn defects in region A pushes the surface state into the interior of the crystal and away from its surface. As a result, the surface state contributes less to the tunnelling current, which depends exponentially on the tip-surface state distance. This effect could also explain the lower intensity of the surface state compared to pure MBT (see Ref. 19 and Supplementary Note 4), as well as earlier reports of a ‘buried’ surface state26.

Conclusion

We conclude that substitutional Sb atoms are incorporated uniformly in MBST and are an effective way to tune the Fermi level. While the effects of the individual defects in MBST have overall comparable effects as in pure MBT, the interaction of the different defect types with the Sb substitution creates a more complicated picture. In detail, the surface state exchange gap is strongly influenced by anti-site defects, where (Bi,Sb)Mn defects give rise to local electron doping, which is the main source of doping fluctuations in this compound, and Mn(Bi/Sb) defects give rise to overall hole doping and a decrease of the local surface state gap. We find the positions of (Bi,Sb)Mn defects to be independent of the Mn(Bi/Sb) defects, and the doping fluctuations caused by accumulations of (Bi,Sb)Mn defects to gives rise to the formation of charge puddles, which close the transport gap. As such, we are able to disentangle the roles of the individual types of defects on the electronic properties of MBST. Going forward, the minimization of fluctuations in the doping and surface state gap, through engineering of (Bi,Sb)Mn and Mn(Bi/Sb) defects, respectively, will be a critical task to achieve a robust transport gap in this promising material platform.

Methods

Scanning Tunnelling Microscopy/Spectroscopy

The STM/S experiments were performed in two ultra-high vacuum (UHV) systems (base pressure \(\sim {10}^{-10}\,{{{{{\rm{mbar}}}}}}\)) hosting commercial low-temperature scanning tunnelling microscopes (Scienta Omicron LT-Nanoprobe and LT-STM) both operating at liquid helium temperatures ( ~ 5 K). PtIr and W tips were obtained by electromechanical etching. PtIr tips we further sharpened by focus ion beam milling. All tips were prepared by indentation into Au(111) surfaces, where the Shockley surface state was used as a benchmark for a clean tip spectrum. dI/dV spectra were collected using standard lock-in techniques at oscillation amplitudes of \({V}_{{{{{{\rm{AC}}}}}}}=3\,{{{{{\rm{mV}}}}}}\,{{\mbox{ to }}}\,5\,{{{{{\rm{mV}}}}}}\). The spectroscopy in Figs. 2 and 5 is performed on a 256 × 256 px2 grid. Scan parameters: Fig. 1a: \({V}_{{{{{{\rm{sample}}}}}}}=(E-{E}_{{{{{{\rm{F}}}}}}})/{{{{{\rm{e}}}}}}=-1\,{{{{{\rm{V}}}}}}\), \({I}_{{{{{{\rm{t}}}}}}}=10\,{{{{{\rm{pA}}}}}}\); Fig. 1b: \({V}_{{{{{{\rm{sample}}}}}}}=-0.8\,{{{{{\rm{V}}}}}}\), \({I}_{{{{{{\rm{t}}}}}}}=100\,{{{{{\rm{pA}}}}}}\); Fig. 2:\(\,{V}_{{{{{{\rm{sample}}}}}}}=-0.15\,{{{{{\rm{V}}}}}}\), \({I}_{{{{{{\rm{t}}}}}}}=150\,{{{{{\rm{pA}}}}}}\). The crystal structures in Fig. 1 were created using VESTA44 and topography data was visualized using Gwyddion45.

To detect the gap size and doping levels (Fig. 2), we use an algorithm, which consists of the following steps: 1) Spatial averaging of the raw data with a 5 × 5 px2 (0.98 × 0.98 nm2) window using Gaussian weighting; 2) In each spectrum: Subtraction of a flat background corresponding to the minimum value of the spectrum. 3) Detection of \({E}_{{{{{{\rm{CB}}}}}}}\) and \({E}_{{{{{{\rm{VB}}}}}}}\) as the voltages at which the data crosses the dI/dV threshold value of 0.05 (arb. units). 4) Another round of spatial averaging as described in step 1) to mitigate outliers of the band edge detection.

We have tested different averaging window sizes around 1×1 nm2 and found little influence on the quantitative results of our analysis. We avoid gliding average smoothing of the individual curves due to possible boundary artefacts, which could affect the detected gap sizes, as the conduction band edges are in proximity to the upper limit of the spectra. A similar algorithm has been used in Ref. 46.

The thermal broadening of the tunnelling spectra (\(5{k}_{{{{{{\rm{B}}}}}}}T\approx 2\,{{{{{\rm{meV}}}}}}\)) becomes comparable to our experimentally observed gaps (2Δ) only for a tiny fraction (\( < 1 \%\)) of the data (see Fig. 2g). When thermal broadening becomes significant, our detection algorithm will tend to underestimate the gap. However, we expect this effect to be small compared to the dI/dV noise.

Sample preparation

MBST was grown by a flux method, resulting in single crystals of dimension 3 × 4 mm and a thickness of 0.1–1 mm24. The stoichiometry of the crystals was determined using energy-dispersive x-ray analysis, which did not show any compositional variation as function of position on the crystal or between crystals from the same batch. The crystals are cleaved in the UHV chamber at room temperature and pressures of \(\sim {10}^{-10}\,{{{{{\rm{mbar}}}}}}\) just before the introduction to the STM stage for characterization. Measurements on multiple cleaves and positions on the sample show consistent crystal quality (Supplementary Note 4). Photoemission and transport measurements of samples of similar composition have shown a transition temperature of \({T}_{{{{{{\rm{N}}}}}}} \sim 24\,{{{{{\rm{K}}}}}}\) and non-closing of the surface state gap through the magnetic transition24,32.