## Abstract

The Berezinskii-Kosterlitz-Thouless (BKT) mechanism, building upon proliferation of topological defects in 2D systems, is the first example of phase transition beyond the Landau-Ginzburg paradigm of symmetry breaking. Such a topological phase transition has long been sought yet undiscovered directly in magnetic materials. Here, we pin down two transitions that bound a BKT phase in an ideal 2D frustrated magnet TmMgGaO_{4}, via nuclear magnetic resonance under in-plane magnetic fields, which do not disturb the low-energy electronic states and allow BKT fluctuations to be detected sensitively. Moreover, by applying out-of-plane fields, we find a critical scaling behavior of the magnetic susceptibility expected for the BKT transition. The experimental findings can be explained by quantum Monte Carlo simulations applied on an accurate triangular-lattice Ising model of the compound which hosts a BKT phase. These results provide a concrete example for the BKT phase and offer an ideal platform for future investigations on the BKT physics in magnetic materials.

## Introduction

Topology plays an increasingly important role in understanding different phases and phase transitions in correlated quantum matters and materials. One prominent example is the Berezinskii–Kosterlitz–Thouless (BKT) mechanism in two-dimensional (2D) systems^{1,2,3,4,5}, which is associated with the binding and unbinding of topological defects. The BKT transition cannot be characterized by conventional order parameters and constitutes the earliest example of phase transition beyond the Landau–Ginzburg paradigm of spontaneous symmetry breaking. Historically, the BKT mechanism was introduced in the *X**Y* spin model and long predicted to occur in magnetic thin films^{1,4}. In experiments, signatures of the BKT transition have been observed in superfluid helium films^{6}, as well as in 2D superconducting films^{7,8} and arrays^{9}. Regarding the original proposal in layered *X**Y*-type magnets, despite intensive efforts^{10,11,12,13,14,15}, direct and unambiguous observation of the BKT transition is still lacking. One major obstacle is the three-dimensional (3D) couplings in the magnets, although weak, will inevitably enhance the confining potential of vortices^{15}, leading to 3D ordering that masks the BKT transition. Therefore, it is of fundamental interest to find and identify BKT materials that could overcome the obstacle and study the topology-related low-energy dynamics.

Recently, a layered frustrated rare-earth antiferromagnet TmMgGaO_{4}^{16,17,18} was reported to ideally realize the triangular-lattice quantum Ising (TLI) model^{19}. The relatively large interlayer distance of 8.3774 Å along the *c* axis gives rise to excellent two dimensionality^{17} and no sign of conventional 3D phase transition was observed in either specific heat or magnetic susceptibility measurements. Nevertheless, it was reported from neutron scattering that TmMgGaO_{4} ordered below ~1 K into an antiferromagnetic phase with a sixfold symmetry breaking^{16,18}, which closely resembles the ground state of the TLI model originated from an order-by-disorder mechanism^{20,21}. At higher temperatures, the effective *X**Y* degrees of freedom emerge and the BKT mechanism is expected to come into play^{21}.

In TmMgGaO_{4}, each Tm^{3+}, with a 4*f*^{12} electron configuration and a spin–orbit moment *J* = 6, forms a non-Kramers doublet due to the crystal-electric-field splitting. The doublet is well separated from the rest levels by about 400 K^{16} and can thus be regarded as an effective spin-1/2. There is further a fine energy splitting within the doublet, induced by the local trigonal crystal field^{17}, acting as an intrinsic “transverse field” applied on the effective spin. From the magnetization measurements^{16,17,18}, one observes that Tm^{3+} ions contribute highly anisotropic Ising-type moments with *J*_{z} = ±6 along the *c* axis, resulting in an effective out-of-plane *g*-factor ~ 13.2^{16,19}. On the contrary, the effective in-plane *g*-factor and dipolar moment in the *a**b* plane are negligible.

Facilitated by this feature in TmMgGaO_{4}, in this work we employed nuclear magnetic resonance (NMR), a sensitive low-energy probe, to detect the BKT phase. We applied a moderate in-plane field of 3 T, which is adequate to collect the ^{69}Ga NMR signals and, at the same time, hardly disturbs the low-energy electronic states of the material. This is important, as in the TLI model that is believed to accurately model TmMgGaO_{4}^{19}, the BKT phase can be fragile against out-of-plane fields^{22,23,24}, thus posing a challenge to NMR measurements. Taking advantage of the fact that in-plane moment in TmMgGaO_{4} is mostly multipolar^{16,19}, our NMR experiments with in-plane fields, which merely couple to the nuclear spins, can clearly identify the BKT phase in the material.

As shown in Fig. 1, from our NMR measurements of the spin-lattice relaxation rate 1/*T*_{1}, we identify *T*_{U} ≃ 1.9 K and *T*_{L} ≃ 0.9 K, which represent the upper- and lower-BKT transitions, where a critical BKT phase resides at zero magnetic field in between the high-*T* paramagnetic and low-*T* antiferromagnetic phases. This finding is further substantiated by our scaling analysis of the measured susceptibility data near *T*_{L}, as well as the simulated NMR and susceptibility data using large-scale quantum Monte Carlo (QMC) calculations.

## Results

### NMR probe of the BKT phase

The obtained NMR spectra with an in-plane magnetic field *μ*_{0}*H* = 3 T are shown in Fig. 2a at representative temperatures. To better resolve the magnetic transition, the hyperfine shifts ^{69}*K*_{n} of the spectra were analyzed and plotted in Fig. 2b as a function of temperature. Upon cooling, ^{69}*K*_{n} peaks at about 0.8–0.95 K and then starts to drop at lower temperatures. Therefore, the ordering temperature is determined to be *T*_{L} ≃ 0.9 K, consistent with neutron scattering experiments^{16,18}. In addition, both the second moments (width of the NMR spectra) and the third moments (asymmetry of the spectra) of the spectra change dramatically below ~ 2 K, suggesting the onset of local hyperfine fields enhanced by the static or quasi-static magnetic ordering (Supplementary Fig. 4). These two characteristic temperatures signal the two-step melting of magnetic order through two BKT transitions, suggesting an intermediate floating BKT phase in the system. We suspect that there is some inhomogeneity of the local hyperfine fields, which is very likely caused by the quenched disorder from Mg/Ga site mixing^{16}, although no significant influence on the electronic and more importantly the magnetic properties is seen (see more detailed discussions in Supplementary Note 4).

The spin-lattice relaxation rate 1/*T*_{1} provides a highly sensitive detection of low-energy spin fluctuations^{25,26,27,28,29}, and thus the BKT transition. In Fig. 2c, we show the 1/^{69}*T*_{1} obtained under in-plane fields *μ*_{0}*H* = 3 T and 1 T, which reflects intrinsic spin fluctuations with zero out-of-plane field. At 3 T, 1/^{69}*T*_{1} first decreases upon cooling from 10 K then suddenly increases below *T*_{U} ≈ 1.9 K, indicating the onset of strong low-energy spin fluctuations. The data at 1 T show similar behaviors. Below *T*_{L} ≃ 0.9 K, 1/^{69}*T*_{1} drops sharply, consistent with the onset of the magnetic ordering as also inferred from the hyperfine shift. Here, 1/^{69}*T*_{1} is dominated by the gapped long wavelength excitations in the ordered state. At the magnetic phase transition, a peaked feature in 1/*T*_{1} develops, caused by the gapless low-energy spin fluctuations with diverging correlation length. Remarkably, at temperatures between *T*_{U} ≃ 1.9 K and *T*_{L} ≃ 0.9 K, 1/^{69}*T*_{1} exhibits a plateau-like structure, indicating the emergence of a highly fluctuating phase with diverging spin correlations yet no true long-range order, which is the hallmark of a BKT phase^{1,2,3,4,5}. Therefore, it is for the first time that such a phase is unambiguously observed in a magnetic crystalline material.

Our unbiased QMC simulations on the TLI model of the material (see “Methods”), with accurate coupling parameters determined in ref. ^{19}, quantitatively justifies the existence of the BKT phase between *T*_{L} and *T*_{U}. We computed 1/*T*_{1} and compared with the experiment below. Figure 2d shows the calculated 1/*T*_{1} by including fluctuations from all momentum points in the Brillouin zone (cf. Supplementary Fig. 2) and compare to that from *K*′ (around the *K* point at the corner of Brillouin zone). The former shows a decrease upon cooling below 4 K and then an upturn above \({T}_{{\rm{U}}}^{* }\simeq 2\) K, followed by a rapid decrease below \({T}_{{\rm{L}}}^{* }\simeq 1\) K. These behaviors are in excellent agreement with the measured 1/^{69}*T*_{1}. The latter reflects gapless excitations of the *X**Y* degrees of freedom emergent in the BKT phase, where the calculated 1/*T*_{1} from *K*′ exhibits an anomalous increase down to \({T}_{{\rm{U}}}^{* }\), below which the increment slows down. The contribution to 1/*T*_{1} near the *K* point reaches a maximum at \({T}_{{\rm{L}}}^{* }\) and drops rapidly below it. The absence of critical spin fluctuations at momentum away from the K point suggests that the measured 1/^{69}*T*_{1} below 2 K is mainly contributed by excitations around the *K* point (see Supplementary Note 3).

Overall, the quasi-plateau behaviors in the QMC results and the two characteristic temperature scales are in full consistency with the NMR measurements. This constitutes both strong support for the accurate quantum many-body modeling of the material TmMgGaO_{4} and also solid proof of the BKT phase therein detected by NMR. Nevertheless, we note that there are still subtle differences between the experimental and numerical data. Needless to say, the real material is always more complicated than our theoretical minimal model. For example, influences from higher crystal-electric-field levels above the non-Kramers doublet, the interlayer couplings not included in our model calculations, and the lack of knowledge on the precise local hyperfine coupling constant, etc., may explain the difference remaining between Fig. 2c, d.

### Universal magnetic susceptibility scaling

Magnetic susceptibility *χ* measurements were also performed to strengthen the finding of the BKT phase. In Fig. 3a, we show the overall temperature dependence of *χ* at different out-of-plane fields. For *T* ≳ 2 K, *χ* increases monotonically upon cooling and barely changes with fields. However, for *T* ≲ 2 K, approximately the upper BKT transition *T*_{U} as obtained from the 1/^{69}*T*_{1} measurements, *χ* increases as the field decreases, suggesting the onset of peculiar magnetic correlations. With further cooling, the susceptibility gets flattened with temperature. The magnetization *M*(*H*) was further measured at selected temperatures (data shown in Supplementary Fig. 6), and for the sake of clarity, the differential susceptibility *d**M*/*d**H* is plotted in Fig. 3b. At around 2.5 T, a pronounced peak can be observed at low temperature, indicating the existence of a quantum phase transition, and the phase at lower fields should be a magnetically ordered phase, although its precise nature remains to be uncovered. Besides the high-field feature, for temperatures at 0.8 K and above, a kinked feature is clearly resolved on each *d**M*/*d**H* curve at low fields, whereas at 0.4 K, the low-field kink disappears, which posts a question of whether there is a quantum transition or merely a crossover from the zero-field AFM phase to the finite-field ordered phase under the dome in Fig. 1. The temperature and field values indicated by the down arrows in Fig. 3 are denoted as \(T^{\prime}\) and *T*″ in the phase diagram (Fig. 1).

Field-theoretical analysis of the TLI model^{23,24} has predicted that upon applying a small out-of-plane field, the differential susceptibility *d**M*/*d**H* shall exhibit a divergent power-law behavior as *d**M*/*d**H* ~ *H*^{−α} in proximity to the BKT phase. At *T*_{L}, *α* has the value of 2/3, which corresponds to a critical exponent *η* = 1/9 at the lower-BKT transition and is originated from the sixfold symmetry breaking^{23}. The exponent *α* gradually decreases as temperature increases, and above an intermediate temperature between *T*_{L} and *T*_{U}, *α* = 0 due to non-universal contributions. This is exactly what we observe in Fig. 3c. We fit the *d**M*/*d**H* with the power-law function at different temperatures, with the fitting regime chosen between 0.6–0.9 T. At 0.8 K, *α* is very close to the expected value of 2/3 (and thus *η* = 1/9) at *T*_{L}, which constitutes a remarkable fingerprint evidence for the BKT transition. At lower temperatures such as 0.4 K, the exponent is also close to 2/3, because the susceptibility saturates with temperature, as shown in Fig. 3a. At high temperatures, *α* drops rapidly to a small value 0.12 at 2.1 K and becomes effectively zero at 3.0 K.

Therefore, the susceptibility scaling gives the lower-BKT transition at about 0.8 K and upper transition probably between 2.1 and 3 K, in good agreement with the *T*_{L} and *T*_{U} estimated from NMR. These results are also fully consistent with our QMC calculations on the susceptibility shown in Fig. 3d. At *T*_{L} or lower, *α* is 2/3, then decreases to a very small exponent 0.086 at 2.67 K, and above 3 K, becomes zero within numerical uncertainty. Such a power-law behavior in *d**M*/*d**H* again signifies the finite-temperature window of the BKT phase with diverging magnetic correlations, which gives rise to the universal power-law scaling of magnetic susceptibility.

## Discussion

We believe the findings in this work are of various fundamental values. Since the original proposal of a BKT phase in magnetic films^{3,4,5}, which also triggered the currently thriving research field of topology in quantum materials, tremendous efforts have been devoted to finding the BKT phase in magnetic crystalline materials, yet hindered by the obstacle outlined in the Introduction. Here, benefiting from NMR as a sensitive low-energy probe, and the nearly zero planar gyromagnetic factor in a TLI magnet TmMgGaO_{4}, we are able to reveal two BKT transitions and a critical BKT phase with an emergent *X**Y* symmetry. Together with the power-law behavior of the susceptibility and excellent agreement between our QMC simulation and experiment data, we unambiguously identify the long-sought BKT phase in a magnetic crystalline material.

Many intriguing questions are stimulated, based on the phase diagram in Fig. 1 obtained here. First, what is the nature of the ordered phase under finite fields, are there further exotic phases and transitions in the phase diagram, and will there be a field-induced quantum phase transition at the high-field side of the dome—these are all of great interests to be addressed in future studies. Second, it should be noted that the dynamical properties obtained by QMC calculations in Fig. 2 are computed on a large, while finite-size, 36 × 36 lattice, which already produces 1/*T*_{1} data in very good agreement with the experimental measurements. Such a great agreement is surprising, given the possible existence of randomness from Ma/Ga site mixing in the material TmMgGaO_{4}^{16}, and also the lattice disorder revealed by the large high-temperature second moment of the NMR spectra (Supplementary Note 4). Although the random distribution in intrinsic transverse fields and spin couplings does not seem to alter the low-temperature spin-ordered phase and the sharp spin excitation line shapes^{18,19}, its intriguing effects on the finite-temperature phase diagram of TLI and also the compound TmMgGaO_{4} call for further studies.

Third, in the study of BKT transition in superfluid systems, it has been observed experimentally and understood theoretically that additional dissipations also appear above the transition temperature due to fluctuations of vortices^{6}. Hence, the plateau of 1/*T*_{1} we observe may also cover regions slightly above the upper BKT transition temperature. We leave this subtlety to future numerical and experimental efforts. Lastly, in general terms, whether there are other rare-earth magnetic materials in the same family of TmMgGaO_{4} that, acquire similar 2D competing magnetic interactions from highly anisotropic gyromagnetic factor and unique triangular-lattice structures, and also exhibit the BKT physics, is quite intriguing and calls for future investigations. All these directions are ready to be explored from here.

## Methods

### Crystal growth and susceptibility measurements

Single crystals were grown by the optical-floating-zone method with an image furnace (IR Image Furnace G2, Quantum Design). The natural cleavage surface of the crystals is the *a**b* plane, which allows us to align the field orientation within an error of 2^{∘}. The dc susceptibility was measured in a PPMS VSM (Quantum Design) for temperatures above 2 K and in a He-3 MPMS (Quantum Design) for temperatures ranging from 0.4 to 2 K.

### NMR measurements

The ^{69}Ga (*I* = 3/2, *γ* = 10.219 MHz/T) NMR spectra were collected with the standard spin-echo sequence, with frequency sweep by a 50 kHz step using a topping tuning circuit. The NMR hyperfine shift was obtained by calculating the change of the first moment of the spectra. The spin-lattice relaxation rate 1/^{69}*T*_{1} was measured by the inversion-recovery method, where a *π*/2 pulse was used as the inversion pulse. The NMR data from 1.8 K and above were measured in a variable temperature insert, and the data from 1.8 K and below were measured in a dilution refrigerator. The weak NMR signal at low fields and the rapid decrease of ^{69}*T*_{2} upon cooling (Supplementary Note 5) prevented us to measure the 1/^{69}*T*_{1} for in-plane fields <3 T, with temperature below 1.8 K. Whereas for in-plane fields of 4 T and higher, the sample could not be held in position because of the large anisotropy in the *g*-factor and unavoidable sample misalignment (≲2^{∘}). At *T* = 1.2 K, we did not find any change of 1/^{69}*T*_{1} with two different radio frequency excitation levels (14 mT and 24 mT), and with different frequencies across the NMR line, within the error bar.

### Triangular-lattice Ising model

At zero field, the intralayer couplings in TmMgGaO_{4} can be described by the TLI Hamiltonian,

where *J*_{1} and *J*_{2} are the superexchange interactions among Tm^{3+}, 〈*i*, *j*〉 and 〈〈*i*, *j*〉〉 refer to the nearest neighbors and the next-nearest neighbors, respectively, and Δ is the energy splitting within the non-Kramers doublet imposed by the crystal field. We have shown that the parameter set *J*_{1} = 0.99 meV, *J*_{2}/*J*_{1} ≈ 0.05 and Δ/*J*_{1} ≈ 0.54 reproduces the experimental results of the transition temperatures and the inelastic neutron scattering spectra^{19}.

In the TLI model [Eq. (1)], we can define a complex field *ψ* as a combination of the Ising (*Z*) components \({m}_{{\rm{A}},{\rm{B}},{\rm{C}}}^{z}\) on three sublattices, i.e.,

Notably, *ψ* = ∣*ψ*∣e^{iθ} is a complex order parameter that represents the emergent *X**Y* degree of freedom relevant to the BKT physics in the TLI model.

### QMC calculations

QMC simulations were performed in the path integral in the \({S}_{i,\tau }^{z}\) basis with discretization in space and time. The lattice of *L* × *L* × *L*_{τ}, where *L* = 36 and *L*_{τ} = *β*/Δ*τ* with Δ*τ* = 0.05 and *β* = 1/*T*, were simulated with both local and Wolff-cluster update schemes^{30,31}. The 1/*T*_{1} results were obtained by first computing the dynamical spin–spin correlation function \(\langle {S}_{i}^{z}(\tau ){S}_{j}^{z}(0)\rangle\) and then acquiring its real-frequency dependence *S*(**q**, *ω*) from the stochastic analytic continuation^{32}. We then determined the 1/*T*_{1} either by summing the contributions close to momentum *K* or over the entire Brillouin zone, as discussed in the Fig. 2d of the main text,

where *A*_{hf}(**q**) is the hyperfine coupling form factor (see Supplementary Note 2). Similar analyses have been successfully applied to the QMC computation of NMR 1/*T*_{1} for the spin-1/2 and spin-1 chains^{33,34}.

## Data availability

The data that support the findings of this study are available from the corresponding authors upon reasonable request.

## Code availability

All numerical codes in this paper are available upon request to the corresponding authors.

## References

- 1.
Berezinskii, V. L. Destruction of long-range order in one-dimensional and two-dimensional systems having a continuous symmetry group I. classical systems.

*JETP***32**, 493 (1971). - 2.
Berezinskii, V. L. Destruction of long-range order in one-dimensional and two-dimensional systems possessing a continuous symmetry group II. quantum systems.

*JETP***34**, 610 (1972). - 3.
Kosterlitz, J. M. & Thouless, D. J. Long range order and metastability in two dimensional solids and superfluids. (application of dislocation theory).

*J. Phys. C Solid State Phys.***5**, L124–L126 (1972). - 4.
Kosterlitz, J. M. & Thouless, D. J. Ordering, metastability and phase transitions in two-dimensional systems.

*J. Phys. C Solid State Phys.***6**, 1181–1203 (1973). - 5.
Kosterlitz, J. M. The critical properties of the two-dimensional xy model.

*J. Phys. C Solid State Phys.***7**, 1046–1060 (1974). - 6.
Bishop, D. J. & Reppy, J. D. Study of the superfluid transition in two-dimensional

^{4}He films.*Phys. Rev. Lett.***40**, 1727–1730 (1978). - 7.
Hebard, A. F. & Fiory, A. T. Evidence for the Kosterlitz-Thouless transition in thin superconducting aluminum films.

*Phys. Rev. Lett.***44**, 291–294 (1980). - 8.
Epstein, K., Goldman, A. M. & Kadin, A. M. Vortex-antivortex pair dissociation in two-dimensional superconductors.

*Phys. Rev. Lett.***47**, 534–537 (1981). - 9.
Resnick, D. J., Garland, J. C., Boyd, J. T., Shoemaker, S. & Newrock, R. S. Kosterlitz-Thouless transition in proximity-coupled superconducting arrays.

*Phys. Rev. Lett.***47**, 1542–1545 (1981). - 10.
Heinrich, M., Krug von Nidda, H.-A., Loidl, A., Rogado, N. & Cava, R. J. Potential signature of a Kosterlitz-Thouless transition in BaNi

_{2}V_{2}O_{8}.*Phys. Rev. Lett.***91**, 137601 (2003). - 11.
Cuccoli, A., Roscilde, T., Vaia, R. & Verrucchi, P. Detection of

*X**Y*behavior in weakly anisotropic quantum antiferromagnets on the square lattice.*Phys. Rev. Lett.***90**, 167205 (2003). - 12.
Wawrzyńska, E. et al. Charge disproportionation and collinear magnetic order in the frustrated triangular antiferromagnet AgNiO

_{2}.*Phys. Rev. B***77**, 094439 (2008). - 13.
Wheeler, E. M. et al. Spin dynamics of the frustrated easy-axis triangular antiferromagnet 2

*H*-AgNiO_{2}explored by inelastic neutron scattering.*Phys. Rev. B***79**, 104421 (2009). - 14.
Tutsch, U. et al. Evidence of a field-induced Berezinskii–Kosterlitz–Thouless scenario in a two-dimensional spin–dimer system.

*Nat. Commun.***5**, 5169 (2014). - 15.
Kumar, R. et al. Structural, thermodynamic, and local probe investigations of the honeycomb material Ag

_{3}LiMn_{2}O_{6}.*Phys. Rev. B***99**, 144429 (2019). - 16.
Li, Y. et al. Partial up-up-down order with the continuously distributed order parameter in the triangular antiferromagnet TmMgGaO

_{4}.*Phys. Rev. X***10**, 011007 (2020). - 17.
Cevallos, F. A., Stolze, K., Kong, T. & Cava, R. J. Anisotropic magnetic properties of the triangular plane lattice material TmMgGaO

_{4}.*Mater. Res. Bull.***105**, 154–158 (2018). - 18.
Shen, Y. et al. Intertwined dipolar and multipolar order in the triangular-lattice magnet TmMgGaO

_{4}.*Nat. Commun.***10**, 4530 (2019). - 19.
Li, H. et al. Kosterlitz-Thouless melting of magnetic order in the triangular quantum Ising material TmMgGaO

_{4}.*Nat. Commun.***11**, 1111 (2020). - 20.
Moessner, R. & Sondhi, S. L. Ising models of quantum frustration.

*Phys. Rev. B***63**, 224401 (2001). - 21.
Isakov, S. V. & Moessner, R. Interplay of quantum and thermal fluctuations in a frustrated magnet.

*Phys. Rev. B***68**, 104409 (2003). - 22.
Liu, C., Huang, C.-J. & Chen, G. Intrinsic quantum Ising model with intertwined multipolarness on a triangular lattice magnet TmMgGaO

_{4}.*Phys. Rev. Research***2**, 043013 (2020). - 23.
Damle, K. Melting of three-sublattice order in easy-axis antiferromagnets on triangular and Kagome lattices.

*Phys. Rev. Lett.***115**, 127204 (2015). - 24.
Biswas, S. & Damle, K. Singular ferromagnetic susceptibility of the transverse-field Ising antiferromagnet on the triangular lattice.

*Phys. Rev. B***97**, 085114 (2018). - 25.
Julien, M.-H. et al.

^{63}Cu NMR evidence for enhanced antiferromagnetic correlations around Zn impurities in YBa_{2}Cu_{3}O_{6.7}.*Phys. Rev. Lett.***84**, 3422–3425 (2000). - 26.
Kitagawa, K., Katayama, N., Ohgushi, K., Yoshida, M. & Takigawa, M. Commensurate itinerant antiferromagnetism in BaFe

_{2}As_{2}:^{75}As-NMR studies on a self-flux grown single crystal.*J. Phys. Soc. Jpn.***77**, 114709 (2008). - 27.
Koutroulakis, G. et al. Quantum phase diagram of the

*S*= 1/2 triangular-lattice antiferromagnet Ba_{3}CoSb_{2}O_{9}.*Phys. Rev. B***91**, 024410 (2015). - 28.
Jeong, M. et al. Magnetic-order crossover in coupled spin ladders.

*Phys. Rev. Lett.***118**, 167206 (2017). - 29.
Moriya, T. & Ueda, K. Antiferromagnetic spin fluctuation and superconductivity.

*Rep. Prog. Phys.***66**, 1299–1341 (2003). - 30.
Wang, Y.-C., Qi, Y., Chen, S. & Meng, Z. Y. Caution on emergent continuous symmetry: a Monte Carlo investigation of the transverse-field frustrated Ising model on the triangular and honeycomb lattices.

*Phys. Rev. B***96**, 115160 (2017). - 31.
Jiang, W., Pan, G. & Meng, Z. Y. Solving quantum rotor model with different Monte Carlo techniques. Preprint at https://arxiv.org/abs/1912.08229 (2019).

- 32.
Sandvik, A. W. Constrained sampling method for analytic continuation.

*Phys. Rev. E***94**, 063308 (2016). - 33.
Sandvik, A. W. NMR relaxation rates for the spin-1/2 Heisenberg chain.

*Phys. Rev. B***52**, R9831–R9834 (1995). - 34.
Capponi, S., Dupont, M., Sandvik, A. W. & Sengupta, P. NMR relaxation in the spin-1 Heisenberg chain.

*Phys. Rev. B***100**, 094411 (2019).

## Acknowledgements

We thank Changle Liu, Rong Yu, Nvsen Ma, and Anders Sandvik for stimulating discussions. We acknowledge the supports from the National Key Projects for Research and Development of China through Grant numbers 2016YFA0300502 and 2016YFA0300504, the National Natural Science Foundation of China through Grant numbers 11574359, 11674370, 11822405, 11674157, 11974036, 11834014, 11874115, and 51872328, RGC of Hong Kong SAR China through Grant number 17303019, Natural Science Foundation of Jiangsu Province with Grant number BK20180006, Fundamental Research Funds for the Central Universities with Grant number 020414380117, and the Research Funds of Renmin University of China. We thank the Center for Quantum Simulation Sciences in the Institute of Physics, Chinese Academy of Sciences, the Computational Initiative at the Faculty of Science and the Information Technology Services at the University of Hong Kong, the Platform for Data-Driven Computational Materials Discovery at the Songshan Lake Materials Laboratory, Guangdong, China, and the Tianhe-I, Tianhe-II, and Tianhe-III prototype platforms at the National Supercomputer Centers in Tianjin and Guangzhou for their technical support and generous allocation of CPU time.

## Author information

### Affiliations

### Contributions

W.Q.Y. and J.S.W. designed the experiments, with proposals from Y.Q., W.L., and Z.Y.M. Z.M. grew and characterized the single crystals and performed susceptibility measurements and analysis, with help from Y.Y.S.G., Z.T.H., Z.H., and H.L. Z.H., C.S.M., and Y.C. performed NMR measurements and analysis. Y.D.L. and H.L. carried out the large-scale quantum many-body calculations, with the guidance from Y.Q., W.L., and Z.Y.M. W.Q.Y., J.S.W., W.L., Z.Y.M., and Y.Q. wrote the manuscript with comments from all coauthors.

### Corresponding authors

## Ethics declarations

### Competing interests

The authors declare no competing interests.

## Additional information

**Peer review information** *Nature Communications* thanks the anonymous reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Supplementary information

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Hu, Z., Ma, Z., Liao, YD. *et al.* Evidence of the Berezinskii-Kosterlitz-Thouless phase in a frustrated magnet.
*Nat Commun* **11, **5631 (2020). https://doi.org/10.1038/s41467-020-19380-x

Received:

Accepted:

Published:

## Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.