Introduction

The XY model by Berezinskii1,2, Kosterlitz, and Thouless3,4 reveals phase transitions in a set of 2D spins with nearest-neighbor interactions, in which the thermally excited topological defects such as vortex and antivortex bound together below the transition temperature and dissociate above. Since these vortex-antivortex pairs are effectively ±1 charged with logarithmical interactions in spatial separation, an elegant description to this model draws on the 2D Coulomb gas consisting of particles with positive or negative charges5,6. Such a paradigm can be transcribed into the scenario of magnetic domains and anti-domains in planar magnets7,8,9,10,11. A platform for testing refers to a quantum anomalous Hall insulator (QAHI)12,13,14 film, in which the perpendicular magnetization breaks the time-reversal symmetry and results in a quantized Hall conductance with vanishing longitudinal resistance. Here the Chern numbers are ±1, whose sign depends on the magnetization direction. This also elucidates the chirality of the topological edge state, i.e. chiral and antichiral (dashed circular arrows in Fig. 1), analogous to the vorticity in the XY model, i.e. clockwise and counterclockwise. Due to the pinning of domain walls by defects and impurities, existences of spin fluctuations15, and superparamagnetic dynamics, it was discovered that domains usually coexist with a certain amount of antidomains (or vice versa) in spite of a strong magnetic field16,17,18,19. Such a magnetic structure seems to be contrary to the observations using the macroscopic magnetometry which only captures the total magnetization. This is ascribed to the existence of the superparamagnetic state that has been widely observed in QAHIs using microscopic-magnetometry imaging16,17,18,19. In this state, the magnetization reversal occurs through a series of random events during which isolated nanoscale (a few tens of nanometers) islands reverse their out-of-plane magnetic moments, giving rise to small magnetic domains (and vice versa). The spatial distribution of these domains is random and isolated, producing nearly noninteracting magnetic moments that constitute the superparamagnet17. Therefore, these isolated domains in QAHI film resemble to the free vortex in the XY model. Based on this, while the amount of domain/antidomain and their interactions can be controlled by external parameters like the applied direct current, magnetic field, and temperature (Fig. 1), it is expected that a BKT-like transition will be driven by the binding and unbinding of domains-antidomains pairs11. However, such a system is certainly not an exact XY model in 2D as this transition is induced by controlling the perpendicular magnetization similar to a 2D Heisenberg or a 2D Ising magnet, which represents the main order parameter. Consider that the chiral edge state in QAHI is formed as a consequence of domain formation, it may act as a vestigial order for this transition. Since the edge state is confined within the XY-plane, it can alone be approximated by a 2D-XY model. From this respect, the QAHI film might act as a platform of a 2D disordered electronic system with random magnetic fluxes and broken time-reversal invariance20,21,22,23,24,25,26. This model, in which the static magnetic field is randomly distributed with zero mean, has attracted much interest in association with the gauge field theory of high-Tc superconductivity and the composite-fermion theory for the half-filled Landau level20,21,25. However, the conclusion of this model has been argued without apparent consensus.

Fig. 1: Schematics of the BKT-like transition driven by the binding and unbinding of domain-antidomain pairs in a QAHI film.
figure 1

In the BKT-like phase (blue panel), domains pair up (black arcs) with antidomains but these pairs dissociate (red panel) when getting out of the this phase by controlling parameters like the applied direct current, magnetic field, and temperature. The broken time-reversal symmetry results in chiral and antichiral edge states around the boundaries of domains and antidomains (clockwise and counterclockwise dashed circular arrows). Different from the standard 2D-XY model, there may be some unbalanced domains besides the pairs in the BKT-like phase.

Full size image

To address the above content, BKT-like transitions have been explored in a QAHI film of 6 nanometers thick so that the QAH effect and a trivial insulator at the coercivity coexist. The latter may originate from the surface hybridization14,27,28,29,30 or/and the formation of the spin-Chern insulator11, which is signified by a zero plateau in the Hall conductance (\({\sigma }_{{xy}}\)~0) and a small longitudinal conductance (\({\sigma }_{{xx}}\)~0.33\({e}^{2}/h\)) in Fig. 2a. This can be better visualized in the \({\sigma }_{{xy}}\)–\({\sigma }_{{xx}}\) plot in the inset where a double-semicircle curve reveals the energy competition. Following the analysis on the vortex interaction in 2D superconductors31,32, the domain-antidomain pairing energy \(U\left({{{{{\bf{r}}}}}}\right)\) with an applied current \(J\) can be written as

$$U\left({{{{{\bf{r}}}}}}\right)=2{\sigma }_{{{{{{\rm{d}}}}}}}-{{{{{{\bf{F}}}}}}}_{{{{{{\rm{L}}}}}}}\cdot {{{{{\bf{r}}}}}}=2{\sigma }_{d}+{q}^{2}\left[{{\ln}}\left(r/{d}_{0}\right)-2{mvr}/{{\hslash }}\right]$$
(1)

where \({\sigma}_{{{{{{\rm{d}}}}}}}\) is the energy to form a single domain that generally includes the exchange, anisotropic, and magnetostatic energies; \({d}_{0}\) is the size of a domain; by analogy with the 2D Coulomb gas, \(q=\sqrt{\pi n{\hslash }^{2}/2m}\) is the effective domain charge where \(n\) is the 2D electron density; \(v\) is the velocity. The pairing energy is modified by the Lorentz force \({{{{{{\bf{F}}}}}}}_{{{{{{\rm{L}}}}}}}\) owing to \(J\) and magnetic field, which results in a saddle point for domains orientated such that the vector connecting them is at a right angle to the direction of \(J\). The critical distance to the saddle point is

$${r}_{{{{{{\rm{c}}}}}}}={{\hslash }}/2{mv}={{\hslash }}{ne}/2{mJ}$$
(2)

with the energy \(U({r}_{{{{{{\rm{c}}}}}}})=2{\sigma }_{{{{{{\rm{d}}}}}}}+{q}^{2}[{{{{{\rm{ln}}}}}}(r/{d}_{0})-1]\approx 2{\sigma }_{{{{{{\rm{d}}}}}}}+{q}^{2}{{{{{\rm{ln}}}}}}(r/{d}_{0})\) if \(r\gg {d}_{0}\). When \(J\) is well below the critical current density \({J}_{0}\) for the breakdown of the QAHI, i.e. \({J}\ll {J}_{0}\), \(U({r}_{{{{{{\rm{c}}}}}}})\) is reduced to \(2{\sigma }_{{{{{{\rm{d}}}}}}}-{q}^{2}{{{{{\rm{ln}}}}}}(J/{J}_{0})\). The mechanism of this breakdown may associate to the flippings from domains to antidomains or vice versa so that a conductive path arises along the domain wall33 which finally connects one edge with the other due to the applied strong electric field34,35,36. The outcome of this causes a nonzero longitudinal resistance \({R}_{{xx}}\) and hence the deviation from the quantized Hall conductance. By assuming a classical escape over the saddle point, the rate to producing free domains will be

$$\Gamma \sim {{\exp }}\left[-U\left({r}_{{{{{{\rm{c}}}}}}}\right)/{k}_{{{{{{\rm{B}}}}}}}T\right] \sim {\left({r}_{c}/{d}_{0}\right)}^{-{q}^{2}/{k}_{{{{{{\rm{B}}}}}}}T} \sim {\left(J/{J}_{0}\right)}^{{q}^{2}/{k}_{{{{{{\rm{B}}}}}}}T}$$
(3)

where \({k}_{{{{{{\rm{B}}}}}}}\) is Boltzmann’s constant and \(T\) is the temperature. In the steady state, the density of free domains \(N\) will be proportional to \(\sqrt{\Gamma }\). Therefore \({R}_{{xx}}\) associates with \(N\) via37

$${R}_{{xx}}=2\pi {d}_{0}^{2}{R}_{0}N \sim {\left(J/{J}_{0}\right)}^{{q}^{2}/2{k}_{{{{{{\rm{B}}}}}}}T}$$
(4)

where \({R}_{0}\) imitates the “normal state” resistance of the superconductor after the breakdown of the QAHI35,38. The resulted voltage is then of the form

$$V \sim {I}^{\alpha }$$
(5)

where the exponent is \(\alpha =1+{q}^{2}/2{k}_{{{{{{\rm{B}}}}}}}T=1+\pi n{\hslash }^{2}/4m{k}_{{{{{{\rm{B}}}}}}}T\). By applying the “universal jump condition”39 for 2D superconductors and magnets in the BKT model

$${q}^{2}\left({T}_{{{{{{\rm{BKT}}}}}}}\right)=\pi n\left({T}_{{{{{{\rm{BKT}}}}}}}\right){{{\hslash }}}^{2}/2m=4{k}_{{{{{{\rm{B}}}}}}}{T}_{{{{{{\rm{BKT}}}}}}}$$
(6)

at the transition temperature \({T}_{{{{{{\rm{BKT}}}}}}}\), the critical exponent will be

$$\alpha \left({T}_{{{{{{\rm{BKT}}}}}}}\right)=1+{q}^{2}\left({T}_{{{{{{\rm{BKT}}}}}}}\right)/2{k}_{{{{{{\rm{B}}}}}}}{T}_{{{{{{\rm{BKT}}}}}}}=3$$
(7)

which is a constant signifying the BKT transition32,40,41. The probe of \(\alpha\) is therefore regarded as one of important benchmarks for the BKT-like transition in the QAHI film.

Fig. 2: Magnetoelectric transport measurements of a QAHI film with fits.
figure 2

a \({\sigma }_{{xx}}\) and \({\sigma }_{{xy}}\) as functions of \({B}_{z}\). Inset: \({\sigma }_{{xy}}\)–\({\sigma }_{{xx}}\) plot shows a double semi-circle feature. b Log-log plots of \(V \sim I\) characteristics under different \({B}_{z}\) at 20 mK. The arrows indicate the evolution directions when sweeping from 0.5 T to −0.3 T. c Log-log plots of \(V \sim I\) characteristics under different \({B}_{z}\) with fits of \(V \sim c{I}^{3}\) (\(c\) as a fitting parameter). d Linear plots of \({R}_{{xx}}\) and \({R}_{{xy}}\) as functions of \(I\) under 0.5 T (top) and −0.03 T (bottom). The dashed lines indicate the quantized values (\(h/{e}^{2}\) and 0), arrows denote the critical current for breakdown, and light grey areas are the current ranges showing \(\alpha =3\). e Log-log plots of \(V \sim I\) characteristics under 0.3 T at different temperatures. The black solid lines are fits to the curves of 20, 100, and 150 mK using \(V \sim c{I}^{3}\) (\(c\) as a fitting parameter). These fitting lines almost coincide to each other and cannot be differentiated clearly. No such a fit works for the curve at 240 mK over the measured \(I\)-range.

Full size image

We experimentally observed the BKT-like transition in the QAHI film with broken time-reversal symmetry. The electronic order parameter of quantized chiral/antichiral edge states around the magnetic domains/antidomains plays a crucial role in mapping the QAHI to a complex 2D-XY system that exhibits the BKT-like transition. The bindings and unbindings of these magnetic domain-antidomain pairs are manipulated by the applied dc current, perpendicular magnetic field, and temperature, the combination of which forms a line of critical points, in contrast to the single characteristic temperature of the conventional BKT transition. An exotic insulator-to-metal transition is observed via the control to the multidomain network no matter the transport is dominated by the gapless metallic edge states. Our results suggest that the QAHI film with random domains acts as a versatile platform in understanding the localization-delocalization problem.

Results

Exponent extraction

The primary probe to the topological transition in the QAHI focuses on the current-voltage characteristics that can easily extract \(\alpha\) as functions of \(T\), perpendicular magnetic field \({B}_{z}\), and applied current \(I\). Figure 2b summarizes the \(V(I)\) curves in a log-log scale obtained by applying various \({B}_{z}\) at 20 mK. The resulted resistance in the QAH state is at least four orders of magnitude smaller than the “normal state” resistance after the breakdown. When sweeping from 0.5 to 0.1 T within the QAH state, \(V(I)\) curves (blue) move towards the high-current regime (blue arrow). Below 0.1 T, the resistance gradually increases and \(V(I)\) curves (red) move towards the high-voltage regime (red arrow), during which the film approaches to the trivial insulator phase. Here, 0.1 T is regarded as a critical field that initiates antidomain nucleations since such a small \({B}_{z}\) cannot fully maintain the single-domain magnetization for the QAHI film which is due to the combined effect of both the weak perpendicular magnetic anisotropy and perturbation from \(I\). From −0.11 T to −0.3 T, the film re-enters the QAH state, while \(V(I)\) curves (black) again move towards high-current regime (black arrow). All \(V(I)\) curves in Fig. 2b exhibit nonlinear features while the \(V \sim {I}^{3}\) characteristic is only found when \({B}_{z}\ge -0.03{{{{{\rm{T}}}}}}\) and \(I\) ~ 10−6 A. Typical examples are shown in the log-log plots in Fig. 2c, in which the black lines fit to parts of these \(V(I)\) curves with \(\alpha ({B}_{z})=3\) at 20 mK. Note that this power-law dependency does not cover a large range, which is widely found in finite-size BKT systems and can be understood in terms of domain stability (see Supplementary Note 1 and Supplementary Figs. 1 and 2). This implies the competing roles between \(I\) and \({B}_{z}\) in stabilizing the domains, which forms certain windows for BKT-like transitions. In linear plots of \({R}_{{xy}}(I)\) and \({R}_{{xx}}(I)\) under different \({B}_{z}\) shown in Fig. 2d and Supplementary Fig. 3, the current intervals showing \(\alpha =3\) (grey areas) are well above the critical currents for breakdown (denoted by black and red arrows), which indicates that the BKT-like transition occurs after the breakdown of the QAH state. After breakdown, \({R}_{{xy}}\) deviates from the quantized value \(h/{e}^{2}\) while \({R}_{{xx}}\) becomes nonzero as required by the increasing density of free domains \(N\), which demonstrate the appearance of antidomains driven by the large currents. Thus, the breakdown of the QAH state provides sufficient antidomains for pairings in the BKT-like transition.

To explore the role of thermal fluctuation, we plot \(V(I)\) curves measured at \({B}_{z}=0.3{{{{{\rm{T}}}}}}\) at different temperatures in Fig. 2e. There exist current ranges around 10−6 A that can extract \(\alpha \left(T\right)=3\) at \(T\) ≤150 mK via linear fits to these log-log plots. Note that these fits (black lines) almost coincide to each other, which implies that the energy gaps for BKT-like transitions overwhelm the thermal fluctuations within this temperature range. At \(T\) > 150 mK no trace of \(\alpha (T)=3\) is detected throughout the entire current range probably because the pairing energy \(U\left({{{{{\bf{r}}}}}}\right)\) is overcome by the thermal fluctuation at the elevated temperature.

Phase diagram

Distinct from the conventional BKT-like transition that is marked by a single characteristic temperature \({T}_{{{{{{\rm{BKT}}}}}}}\), the BKT-like transition in the QAHI strongly depends on the dynamical stabilities of domains15 that are perturbed by \(I\), \({B}_{z}\), and \(T\), which combine to form a line of critical points. This could be seen by color-plotting \(\alpha\) as functions of both \(I\) and \(T\) at given \({B}_{z}\) as shown in Fig. 3a–f, where the \(\alpha =3\) contour (white) signifies this critical line. This contour expands towards larger current —higher temperature ranges as \({B}_{z}\) is decreased from 0.5 T to 0.1 T [Fig. 3a–c], followed by shrinking when decreasing to −0.03 T [Fig. 3d–f]. This threshold of \({B}_{z}=0.1{{{{{\rm{T}}}}}}\) is consistent with observations in Fig. 2a, b. In this case, \({F}_{{{{{{\rm{L}}}}}}}\) decreases with the decrease of \({B}_{z}\), which in turn increases the pairing energy \(U\left({{{{{\bf{r}}}}}}\right)\) according to \(U\left({{{{{\bf{r}}}}}}\right)=2{\sigma }_{{{{{{\rm{d}}}}}}}-{{{{{{\bf{F}}}}}}}_{{{{{{\rm{L}}}}}}}\cdot {{{{{\bf{r}}}}}}\). To lower \(U\left({{{{{\bf{r}}}}}}\right)\) and reach a stable BKT-like phase, higher critical current \({I}_{{{{{{\rm{BKT}}}}}}}\) and temperature \({T}_{{{{{{\rm{BKT}}}}}}}\) are favored to compensate the energetic variation. Such a process is illustrated as Path A of the contour [black arrow in Fig. 3a]. However, further increase of \(I\) will strongly perturb the system and suppress the pairings. Again, to lower the total energy, \({T}_{{{{{{\rm{BKT}}}}}}}\) decreases as shown by the sharp downturn in Path B. By sweeping from 0.1 T to −0.03 T [Fig. 3c–f], both \({I}_{{{{{{\rm{BKT}}}}}}}\) and \({T}_{{{{{{\rm{BKT}}}}}}}\) decrease. Meanwhile the multidomain structure starts to dominate and the domain fluctuation is enhanced, which is compensated by reducing \(I\) and lowering \(T\). The roles played by \(I\) and \({B}_{z}\) to the BKT-like transitions can be further revealed by color-plotting \(\alpha\) as functions of both these two parameters at given temperatures as shown in Fig. 3g–k. Likewise, in Path C of Fig. 3g a lower critical field \({B}_{{{{{{\rm{BKT}}}}}}}\) requires a higher \({I}_{{{{{{\rm{BKT}}}}}}}\) owing to the increase of \(U\left({{{{{\bf{r}}}}}}\right)\) at the given temperature when the film stays in the QAH state (\({B}_{z} \, > \, 0.1{{{{{\rm{T}}}}}}\)). A vanishing \({B}_{Z}\) (−0.03 T–0.1 T) gives rise to strong domain fluctuations which only requires a low \(I\) for pairing as shown in Path D. By warming from 20 mK to 150 mK [Fig. 3g–j], the contour shrinks and finally \(\alpha\) stays below 3 at temperatures above [see \(T\) = 240 mK in Fig. 3k]. Therefore, a strong thermal fluctuation can overwhelm \(U\left({{{{{\bf{r}}}}}}\right)\) and suppress the formation of the BKT-like phase, at which neither the \({I}_{{{{{{\rm{BKT}}}}}}}\) nor \({B}_{{{{{{\rm{BKT}}}}}}}\) is found. These critical behaviors are summarized in Fig. 3l, in which the maximums of both the critical current \({I}_{{{{{{\rm{BKT}}}}}}}^{{{\max }}}\) and critical temperature \({T}_{{{{{{\rm{BKT}}}}}}}^{{{\max }}}\) are plotted as functions of \({B}_{z}\). Both curves peak around 0.1 T, again showing the critical \({B}_{z}\) consistent with the above discussions. Hence, the BKT-like transition in the QAHI film is closely associated with the specific domain texture. In terms of the domain stability, this dependence not only shows the similar roles played by \(I\) and \(T\) but also highlights their competitions against \({B}_{z}\).

Fig. 3: Evolutions of BKT-like transitions parameterized by the applied current, magnetic field, and temperature.
figure 3

a–f Color-plots of \(\alpha\) as functions of \(I\) and \(T\) under given \({B}_{z}\). g–k Color-plots of \(\alpha\) as functions of \(I\) and \({B}_{z}\) under given \(T\). The white contours show the situations with \({\alpha}\) = 3, the critical windows for the BKT-like transitions. The black arrows A, B in a and C, D in g indicate the evolutions by varying \(I\) and \({B}_{z}\) respectively. l The maxima of the critical current (\({I}_{{{{{{\rm{BKT}}}}}}}^{{{\max }}}\)) and temperature (\({T}_{{{{{{\rm{BKT}}}}}}}^{{{\max }}}\)) for BKT-like transitions as a function of \({B}_{z}\), which are extracted by projecting \(\alpha\) contours in a-f onto the \(I\)- and \(T\)-axes, respectively.

Full size image

Exotic transition

Strikingly, the QAHI film can undergo an exotic “insulator”-to-“metal” transition in the presence of random magnetic domains. These random domains originate in the random magnetizations from two aspects: (i) the random distribution of magnetic dopants, and (ii) the random pinnings by defects and/or impurities. In the QAH state, e.g. under 0.5 T in Fig. 4a, the transport is dominated by the metallic edge states, in which \({R}_{{xx}}\) decreases to ~10−1 kΩ at 14 mK upon cooling. However, this characteristic only holds for \(I\) ≤1 μA [blue curve in Fig. 4a]. When increasing to 1.4–2.0 μA (blue and orange curves), \({R}_{{xx}}\) remains similar metallic behaviors but its values at 14 mK increase by one order of magnitude to ~100 kΩ, and further enhanced by another order of magnitude to ~101 kΩ when increasing to ~4.4 μA (red curve). In fine-scale plots of these curves in Fig. 4b, one can find that along with the increase of \(I\), \({R}_{{xx}}(T)\) curves indeed exhibit upturns upon cooling below ~50 mK, showing an insulator-like behavior. To probe its origin, we traced \({R}_{{xx}}(T)\) curves with different \(I\) and in Fig. 4c–h. Under −0.03 T–0.5 T, all \({R}_{{xx}}(T)\) curves with small \(I\) ~ 0.1 μA (blue) show similar metal-like behavior and stay at ~10−1 kΩ at 14 mK. At the medium range of 1 μA ~3 μA, upturns of \({R}_{{xx}}(T)\) curves are suppressed [Fig. 4d] and gradually taken over by downturns [Fig. 4f, h], exhibiting an exotic “insulator”-to-“metal” transition. When reaching to the large range of ~4.4 μA, this metal-like behavior can be clearly identified by sharp drops near the lowest temperatures in Fig. 4f, h, even though the overall values of \({R}_{{xx}}\) remain around 101 kΩ at all \({B}_{z}\) ranges. The dependences of this transition on both \(I\) and \({B}_{z}\) are quantitatively summarized in Fig. 4i, in which the ratio of change in \({R}_{{xx}}\) is defined as \(\triangle {R}_{{xx}}=\left[{R}_{{xx}}\left(14\, {{{{{\rm{mK}}}}}}\right)-{R}_{{xx}}\left(40\, {{{{{\rm{mK}}}}}}\right)\right]/{R}_{{xx}}\left(40\, {{{{{\rm{mK}}}}}}\right)\). In this regard, a positive \(\triangle {R}_{{xx}}\) implies an insulator-like behavior while a negative one is metal-like. Critically, the medium-\(I\) curve (green) drives the film from the metal-like under \({B}_{z}\) < 0.5 T to insulator-like under 0.5 T. Further increases in \(I\) (orange and red) consistently unveil a critical \({B}_{z}\) close to 0.1 T [indicated by the black arrow in Fig. 4i], below which the film is metal-like but becomes insulator-like above. As the QAHI film contains random magnetic dopants, the resistance upturn upon cooling may originates in the Kondo effect. To check this possible origin, we plot \({R}_{{xx}}-{R}_{{xx}}\left(14\, {{{{{\rm{mK}}}}}}\right)\) vs. \({{\log }}(T)\) under different \({B}_{z}\) in Fig. 4j since the Kondo effect can be suppressed by an external magnetic field42,43. It’s clear that the upturn is not always suppressed by \({B}_{z}\), e.g. the upturns at 0.5 and 0.2 T are stronger than that of 0.1 T. Moreover, the curves under −0.03 T–0.05 T show no trace of upturn. These observations convince us that the transition does not have a Kondo origin.

Fig. 4: Exotic “insulator”-to-“metal” transitions manipulated by the applied current and magnetic field.
figure 4

a–h Longitudinal resistances (\({R}_{{xx}}\)) as functions of \(T\) obtained by using different \(I\) and \({B}_{z}\). Note that b, d, f, and h are fine-scale plots of a, c, e, and g with respective \({R}_{{xx}}\) scale (multiple vertical axes). i The ratio of change in \({R}_{{xx}}\) defined as \(\triangle {R}_{{xx}}=\left[{R}_{{xx}}\left(14\,{{{{{\rm{mK}}}}}}\right)-{R}_{{xx}}\left(40\,{{{{{\rm{mK}}}}}}\right)\right]/{R}_{{xx}}\left(40\,{{{{{\rm{mK}}}}}}\right)\) vs. \({B}_{z}\), where \(\triangle {R}_{{xx}}\) are extracted from a–h with the same color code. The light yellow and blue regimes stand for the metal-like and insulator-like phases respectively, and the black arrow indicates the critical \({B}_{z}\). (j) The modulated longitudinal resistances \({R}_{{xx}}\!-\!{R}_{{xx}}\,\left(14\,{{{{{\rm{mK}}}}}}\right)\) vs. \({{\log}}(T)\) plot by applying ~4.4 μA dc current under different \({B}_{z}\).

Full size image

In a device from another batch of QAHI wafer, we found similar observations supporting the BKT-like transitions and “insulator”-to-“metal” transitions (see Supplementary Note 2 and Supplementary Figs. 4–6).

Discussion

Anderson localization as a possible origin to the observed transition is examined in Supplementary Information Note 3. In the QAHI, the \({R}_{{xx}}({B}_{z})\) curves at different \(T\) (Supplementary Fig. 7) and \({R}_{{xx}}\left(I\right)\) curves under different \({B}_{z}\) and \(T\) (Supplementary Fig. 8) do not show any crossing point with a critical value of \({B}_{z}\) or \(I\) within the \({B}_{z}\)- and \(I\)-ranges that show the BKT-like transition. Importantly, these data do not exhibit the Anderson-type metal-to-insulator transition (Supplementary Fig. 9) and, thus, cannot be described using the one-parameter scaling relation. Therefore, Anderson localization is excluded from the origin of the observed transition.

All known BKT systems are classified to the 2D-XY universality class, which requires the spins in 2D magnets confined in the XY-plane. However, it’s likely that the spins in the QAHI have three degrees of freedom, similar to a 2D Heisenberg magnetic system (Fig. 1). This issue roots in the complex multicomponent order parameter in the observed transition. In the QAHI the chiral edge state couples to the domain magnetization, which results in a complex interaction between the electronic order parameter and magnetic order parameter. The present transport results demonstrate that the electronic order, which can be thought as a vestigial order in the multicomponent order parameter44,45, plays a crucial role in this BKT-like transition. At the point of magnetization reversals driven by the combined effect from \({B}_{z}\), \(I\), and \(T\), the amplitude of the magnetic order parameter passes through zero, leaving a random flux as a perturbation so that the electronic order takes the leading role. Since this electronic order parameter, i.e. the chiral edge state, is confined to the XY-plane, when decoupled from the temporarily suppressed magnetic order, they can be mapped to a 2D-XY system similar to that in Ref. 9. Therefore, by considering the contribution from the electronic order parameter, the QAHI can still be approximately mapped to a complex 2D-XY system and exhibits the BKT-like transition.

In the 2D-XY model the pairing of vortex and antivortex is balanced and the system exhibits a zero net vorticity. In the QAHI, the net chirality which dominates the overall topology of the QAH state is nonzero and unbalanced as a result of the net magnetization. However, theoretically the BKT-like transition could still take place in a critical region between the multidomain and single domain11. This actually corresponds to our case that certain amount of domains are flipped to antidomains or vice versa by the combined effect of \({B}_{z}\), \(I\), and \(T\). Thus, such a perturbed QAHI could still undergo a BKT-like transition although the pairs are unbalanced and net chirality is nonzero. This behavior could be understood that the observed BKT-like transition may be mainly executed by the local domain flipping near the edges followed by domain-antidomain pairing, while the chiral edge state due to the overall net chirality only acts as a probe to this local flipping and induces a background in the BKT-like transition. As long as this local flipping does not strongly percolate to the opposite edge and the gap is not fully closed, the overall net chirality remains almost intact. However, this flipping could still result in a percolative network near the edge, which contains sufficient balanced pairs for the BKT-like transition. This network allows for tunneling from the domain to antidomain or vice versa28 and gives rise to a finite longitudinal resistance \({R}_{{xx}}\), which is the origin of the “normal state” resistance as derived above. Such a process can be evidenced by Fig. 2d and Supplementary Fig. 3 that, during the BKT-like transition (grey regions), \({R}_{{xx}}\) increases from 0 to about 0.1\(h/{e}^{2}\), whereas \({R}_{{xy}}\) only slightly reduces from \(h/{e}^{2}\) to 0.99~0.98\(h/{e}^{2}\). Therefore, the observed BKT-like transition in the QAHI is mainly reflected by the local balanced domain-antidomain pairings superimposed onto a background of the chiral edge state.

Our findings shed lights on understanding the localization-delocalization problems9,10,11,22,23,24,25,46,47 and metal-to-insulator transitions10, and the complex domain-induced percolation. The present study may act as an experimental reference to further study the random flux model. The analysis in terms of the BKT-like transition may also be applied to other layered magnets to determine the 2D limit of the magnetic order.

Methods

I. Films growths and devices fabrications

The single crystalline Cr-doped (BixSb1-x)2Te3 films were epitaxially grown on semi-insulating GaAs (111)B substrates in a molecular beam epitaxy system. The substrates were annealed to 580 °C to remove the native oxide under Te/Se rich atmosphere. During the growth, the substrates were maintained around 200 °C with 6 N purity Bi, Sb, Te, and Cr co-evaporated from Knudsen cells. The epitaxial growth was in situ monitored by the reflection high-energy electron diffraction to optimize the growth conditions and achieve the target thicknesses. The films were patterned into Hall bars (2 mm × 1 mm) by a stencil mask using reactive ion etching technique.

II. Magneto-electrical measurements

The magneto-electrical transport measurements were performed in a dilution refrigerator equipped with a 14 T superconducting magnet. The AC (19 Hz) / DC current was applied across a 1-MΩ reference resistor and the sample via the NI PXIe-6739 analog output while the drain current, longitudinal, and Hall voltages were measured via the NI PXIe-4499 sound and vibration module.