## Abstract

Frustrated magnets hold the promise of material realizations of exotic phases of quantum matter, but direct comparisons of unbiased model calculations with experimental measurements remain very challenging. Here we design and implement a protocol of employing many-body computation methodologies for accurate model calculations—of both equilibrium and dynamical properties—for a frustrated rare-earth magnet TmMgGaO_{4} (TMGO), which explains the corresponding experimental findings. Our results confirm TMGO is an ideal realization of triangular-lattice Ising model with an intrinsic transverse field. The magnetic order of TMGO is predicted to melt through two successive Kosterlitz–Thouless (KT) phase transitions, with a floating KT phase in between. The dynamical spectra calculated suggest remnant images of a vanishing magnetic stripe order that represent vortex–antivortex pairs, resembling rotons in a superfluid helium film. TMGO therefore constitutes a rare quantum magnet for realizing KT physics, and we further propose experimental detection of its intriguing properties.

## Introduction

Kosterlitz–Thouless (KT) physics bestows interesting mechanism of phase transition upon two-dimension (2D) interacting system with a continuous symmetry. Although such symmetry is not allowed to break spontaneously at any finite temperature^{1}, phase transition can still take place from the high-temperature disordered phase to a KT phase with quasi-long-range order, which has a topological root in the binding of the vortex and antivortex pair^{2,3}. Experimentally, the KT transition has been observed in thin helium films^{4} and ultracold 2D Bose gases^{5,6}, etc. Two distinct types of elementary excitations, i.e., phonons and rotons, play essential roles in the related superfluid phenomena^{7,8}, and they are important for understanding liquid helium thermodynamics^{9}. Besides interacting bosons in liquid and gas, there are also theoretical proposals of KT transitions in solid-state magnetic systems such as the 2D classical XY^{2,3} and the frustrated quantum Ising models^{10}. However, to date, the material realization of the KT transition in 2D magnets has rarely been reported.

In the mean time, the search of exotic quantum magnetic states in the triangular-lattice spin systems—the motif of frustrated magnets—has attracted great attention over the decades. Experimentally, the triangular-lattice quantum magnets have been synthesized only very recently, including compounds Ba_{3}CoSb_{2}O_{9}^{11,12,13,14}, Ba_{8}CoNb_{6}O_{24}^{15,16}, and a rare-earth oxide YbMgGaO_{4}—which has been suggested as a quantum spin liquid candidate^{17,18,19,20}—whereas an alternative scenario of glassy and disorder-induced state has also been proposed recently^{21,22,23}. On the other hand, an Ising-type triangular antiferromagnet TmMgGaO_{4} (TMGO, with Yb^{3+} replaced by another rare-earth ion Tm^{3+})^{24,25,26}, as shown in Fig. 1a and explained in details in this study, is the successful material realization of a quantum magnet with strong Ising anisotropy.

In this work, we construct the microscopic model of TMGO and employ two state-of-the-art quantum many-body simulation approaches: the exponential tensor renormalization group (XTRG)^{27} and quantum Monte Carlo (QMC) equipped with stochastic analytic continuation (QMC-SAC)^{28,29,30,31,32}, to calculate both the thermodynamic and dynamic properties. By scanning various parameters and fit our simulation results to the existing experimental data^{24,25,26}, we find TMGO realizes a triangular-lattice transverse-field Ising model and determine accurately its model parameters. Based on this, we conclude that TMGO should host the celebrated KT phase and further predict several prominent features to be observed in TMGO, inspired by the experimental measurements for detecting KT physics in a superfluid thin film^{4}. It is worthwhile to point out that our calculation of quantum fluctuations goes beyond the linear spin-wave approximation in ref. ^{26} and puts the system in the clock-ordered (later melted through KT transitions), rather than disordered, regime. Therefore, our methods and results do not only explain the experimental findings but, more importantly, establish a protocol for acquiring equilibrium and dynamic experiments of strongly correlated quantum materials, such as TMGO, in an unbiased manner.

## Results

### Microscopic spin model

Due to strong spin–orbit coupling and crystal electric field splitting, TMGO can be described as an effective spin-1/2 model with strong easy-axis anisotropy, i.e., a triangular-lattice Ising model (TLI), as shown in Fig. 1a. First-principle calculations of the TMGO material are performed based on the density functional theory (DFT)^{33,34,35,36} where we see a large easy-axis anisotropy of room-temperature energy scale. By comparing the DFT energies of antiferromagnetic spin configuration with a ferromagnetic one, one finds the former has a lower energy and the coupling strength can be estimated as a few tenths of meV. Density distribution of 4*f* electrons in TMGO can also be obtained, where it is observed that the 4*f* electrons of Tm^{3+} are coupled via superexchange mediated by 2*p* electrons of O^{2−} within the triangular-lattice plane (Supplementary Note 1).

In ref. ^{25}, the authors took the lowest two levels \(\left|{\Phi }_{\pm }\right\rangle\) of Tm^{3+} as non-Kramers doublet and construct a classical TLI with both nearest-neighbor (NN) and next-nearest-neighbor (NNN) interactions to account for the absence of zero-point entropy observed in experiments. Substantial randomness was also introduced to explain the smooth magnetization curves even at a very low temperature. Later on, inelastic neutron scattering (INS) results of TMGO reveal a clear magnon band^{26}, suggesting the influence of coupling randomness should be modest in TMGO and an adequate modeling of the material shall include in the Hamiltonian non-commuting terms with quantum fluctuations. As the Kramers theorem is absent in Tm^{3+} system with total angular momentum *J* = 6, a small level splitting Δ between the quasi-doublet \(\left|{\Phi }_{\!\pm }\!\right\rangle\) is involved, as shown in Fig. 1a. Therefore, a quantum TLI model was proposed^{26}, with spin-1/2 Hamiltonian

where 〈,〉\(\left(\langle \langle ,\rangle \rangle \right)\) stands for NN (NNN) couplings *J*_{1} (*J*_{2}), Δ the energy splitting between \(\left|{\Phi }_{\!\pm }\!\right\rangle\) (i.e., the intrinsic transverse field), and *h* is the external magnetic field. *g*_{∥} = 2*J**g*_{J} constitutes the effective spin-1/2 *g* factor, with *g*_{J} the Landé factor.

The phase diagram of quantum TLI has been studied intensively with analytic and numeric methods in the past^{10,37,38,39}, and is schematically shown in Fig. 1b. We indicate the TMGO model parameter with the vertical arrow (the determination of parameters is given below). From high to low temperatures, the system first goes through an upper KT transition at *T*_{h} from the paramagnetic phase to a KT phase with power-law (algebraic) spin correlations. At a lower temperature *T*_{l}, the system enters the clock phase with a true long-range order depicted in Fig. 1a. This three-sublattice clock order breaks the discrete lattice point group and the *Z*_{2} spin symmetries, giving rise to a low but finite transition temperature *T*_{l}.

Increasing the next-nearest-neighbor coupling *J*_{2}, say, at *J*_{2}/*J*_{1} = 0.2, we find the static magnetic structure factor develops a stripe order^{40} with structure factor peak at M point (see, e.g., Fig. 4a) of the Brillouin zone (BZ) (see Supplementary Notes 2 and 3). This magnetic stripe order, as shown in Fig. 1c, has been observed previously in TLI material AgNiO_{2}, where *J*_{2} coupling is relatively strong (~0.15*J*_{1}, along with other interactions) and the exotic KT physics is absent there^{41}. In TMGO, however, the clock order wins over the stripe order as *J*_{2} ∕*J*_{1} ≃ 0.05 is relatively small in this material. Nevertheless, as will be shown below, a ghost of the stripe order—the M rotonlike modes—remains in the spin spectrum^{26}, which turns out to be related to a vortex–antivortex pair excitation in the topological language (see Discussion section).

### Thermodynamics and parameter fittings

The model parameters in Eq. (1) can be accurately determined through fitting the available experimental data of TMGO^{24,25,26}, from which we find *J*_{1} = 0.99 meV, *J*_{2} = 0.05*J*_{1}, Δ = 0.54*J*_{1} and *g*_{∥} = 13.212. We present in Fig. 2 the calculated thermodynamic quantities and their experimental counterparts^{25,26}, where excellent agreements are seen. In Fig. 2a, at *T* > 30 K, the magnetic entropy *S*_{m} approaches \(R\mathrm{ln}\,2\), corresponding to the high-temperature paramagnetic phase with effective spin-1/2. As temperature decreases, *S*_{m} gradually releases throughout the intermediate-temperature regime and approaches zero below \({T}_{{\rm{l}}}^{* }\simeq 1\) K, as the long-range clock order develops. In Fig. 2b, the very good agreement in magnetic-specific heat *C*_{m} extends from high *T* (~30 K) all the way down to low temperature *T* ~ 1 K.

In the fittings of the magnetic entropy and specific heat curves in Fig. 2a, b, we rescale the *T*-axes (in the unit of *J*_{1}) to lay the model calculations on top of the experimental measurements, and in this way we find the optimized *J*_{1} = 0.99 meV. In both plots, the *y*-axes scaling ratios are associated with the ideal gas constant *R* = 8.313 J mol^{−1} K^{−1} and thus fixed. In Fig. 2c, we fit the magnetic susceptibility \(\chi (h)=\frac{M(h)}{h}\), with *M*(*h*) the uniform magnetization (per Tm^{3+}), under external fields of a small value *h* = 1 kOe and a larger one 10 kOe. As shown in Fig. 2c, by setting the effective Landé factor *g*_{∥} = 13.212, we fit both susceptibility curves very well. This completes the model parameters in the Hamiltonian Eq. (1) (see more fitting details in Supplementary Note 4) and note this parameter set also leads to accurate entropy results at 5 kOe when put in a direct comparison with the experimental data in Fig. 2a.

With the parameters *J*_{1}, *J*_{2}, Δ, and *g*_{∥} determined from the above fittings, we can compute the magnetization curve *M*(*h*) and compare it directly with several independently measured experimental curves in Fig. 2d. It is noteworthy that there exists a turning point at about 1/3 magnetization, under a magnetic field around 20–25 kOe. It becomes clearer as *T* decreases further down to 40 mK (see the inset of Fig. 2d), suggesting the existence of field-induced quantum-phase transition in TMGO. This sharp change of behaviors can also be witnessed in the specific heat curves under various magnetic fields (see Supplementary Note 6), which can also be understood very well within the set of parameters obtained above.

Lastly, we briefly discuss the scalings in the uniform susceptibility *χ* in Figs. 2c and 3, where susceptibility curves under more external fields are computed theoretically in the latter plot as a complementary. These *χ* data reflect the two-step establishment of magnetic order as *T* lowers. As pointed out in refs. ^{42,43}, a universal scaling *χ*(*h*) = *h*^{−α} appears for small fields *h* with \(\alpha =\frac{4 \, - \, 18\eta (T)}{4\, - \, 9\eta (T)}\) in the KT regime, where \(\eta (T)\in [\frac{1}{9},\frac{2}{9}]\) is the anomalous dimension exponent of the emergent XY-order parameter varying with temperature. For *T* = *T*_{l}, \(\eta =\frac{1}{9}\) and *χ*(*h*) ~ *h*^{−2/3}, which diverges as *h* approaches zero, whereas above some higher temperature \(T^* _h\), \(\eta =\frac{2}{9}\) and *χ*(*h*) remains a constant vs. *h*. Therefore, at small external field, the increase of *χ* at intermediate *T* reflects the decrease of *η*(*T*) vs. *T* and such enhancement becomes less prominent for a relatively larger field, say, *h* = 10 kOe. This salient difference indeed can be noticed in the experimental and our numerical curves in Fig. 2c. Moreover, in Fig. 3 we plot in the inset susceptibility curves under various magnetic fields *h* between 6.5 kOe and 9 kOe, where the power-law scaling is shown explicitly and the anomalous exponent *η* can be extracted therein.

### Spin spectra and magnetic structure factors

Spin frustration can lead to strong renormalization effects, which in turn gives rise to interesting spectrum in dynamics. Here we employ the QMC-SAC approach^{28,29,30,31,32} to compute the spin spectra *S*(**q**, *ω*) from *S*^{z} spin correlations, at various temperatures (see Methods). The obtained spectra, with model parameters determined from equilibrium data fittings, are plotted in Fig. 4 and are compared directly with the INS results^{26}. Figure 4a depicts the spin spectrum inside the clock phase, at a low temperature *T* = 0.5 K. As the clock phase is of discrete symmetry breaking, the *S*(** q**,

*ω*= 0) at K point signals the Bragg peak of the clock order and there exists a small gap ~0.1 meV between the

*ω*= 0 and finite

*ω*spectra, consistent with the INS result. The rotonlike modes are also clearly present in the QMC-SAC results, with an energy gap about 0.4 meV, in quantitative agreement with that in ref.

^{26}.

Figure 4b is the QMC-SAC spectrum calculated according to the parameters (*Δ/**J*_{1} ≃ 1.36, *J*_{2}/*J*_{1} ≃ 0.046) given in ref. ^{26}. As mentioned in the introduction, we find, via spin structure factor calculations, that such set of parameters actually put the model in the disordered paramagnetic phase with Δ > Δ_{c} ~ 0.8*J*_{1}. It is possible that the fitting scheme adopted in ref. ^{26} is based on mean-field treatment and cannot capture the quantum fluctuations inherent to the quantum TLI model and the material TMGO. This is a clear sign that the unbiased quantum many-body calculation scheme in our work is the adequate approach to explain the experimental results.

We continue with the correct parameter set and rise the temperature to *T* = 1.45 K in Fig. 4c. It is interesting to see that the dispersion still resembles that in the clock phase of Fig. 4a but with a vanishing gap at the K point and softened M roton modes. To show it more clearly, we plot the intensity at M in Fig. 4d, where the roton gap gets reduced as *T* increases, with substantially broadened linewidths. As M rotonlike excitations can be related to vortex-pair excitation (see Discussions), this softening of M roton is consistent with the scenario of vortex proliferations near the upper KT transition. Such remarkable spectra constitute a nontrivial prediction to be confirmed in future INS experiments.

Besides, the static magnetic structure factor \(S({\bf{q}})=\sum _{i,j}\) e\(^{i{\bf{q}}\cdot ({{\bf{r}}}_{i}-{{\bf{r}}}_{j})}\langle {S}_{i}^{z}\ {S}_{j}^{z}\rangle\) are also simulated, where **r**_{i} and **r**_{j} run throughout the lattice. Figure 5a shows the temperature dependence of *S*_{K} and *S*_{M}, where one observes an enhancement of *S*_{M} at intermediate temperature, signifying its closeness towards the stripe order. At \(T \, < \, {T}_{{\rm{l}}}^{* }\), the enhancement of *S*_{M} vanishes and instead the *S*_{K} intensity becomes fully dominant. Figure 5b–d show the *S*(**q**) results at low (*T* = 0.57 K), intermediate (*T* = 2.2 K), and high temperaure (*T* = 4.5 K). In the clock phase, *S*(**q**) evidently peaks at the K point, the ordering wavevector of the three-sublattice clock phase, whereas in the intermediate temperature regime, notably there exists a “ghost” peak at the M point, manifesting the existence of short-range stripe order selected by thermal fluctuations. These interesting features are gone at higher *T* = 4.5 K, where strong fluctuations considerably weaken the structure factor peaks.

## Discussion

As postulated by Landau, the phonon–roton spectrum play an essential role in understanding low-*T* thermodynamics and critical velocities of superfluidity, etc, in liquid helium, where the rotons are believed to be related to local vorticity of the fluid^{7}. Roton constitutes a minima in the spectrum at finite momentum and energy, and has been regarded as a quantum analog of hydrodynamic vortex ring, as coined “the ghost of vanished vortex ring” by Onsager^{44}. On the other hand, as derived from a trial wavefunction introduced by Feynman^{8}, the roton excitation has energy *ϵ*_{q} ~ *ℏ***q**^{2}/[2*m* *S*(**q**)], with *m* is the helium atom mass and *S*(**q**) the structure factor peak, and it is therefore also associated with an incipient crystalization Bragg peak competing with superfluidity^{45}. Rotonlike excitations are also found in thin helium films^{46} and frustrated triangular Heisenberg (TLH) magnets. In the latter case, M point rotonlike modes were predicted theoretically^{47,48,49,50} and confirmed in recent experiments^{13,14}, whose nature is under ongoing investigations^{51,52}. Notably, it has been proposed that the M rotons in TLH can be softened by further enhancing spin frustration (and thus quantum fluctuations)^{52} or thermal fluctuations^{53}, which melts the long-range or incipient semi-classical 120° order, driving the system into liquid-like spin states.

In the frustrated magnet TMGO, as mentioned earlier in Fig. 4, there exists rotonlike modes with large density of states, which becomes softened even at low temperatures, melting the clock order, and strongly influences thermodynamics of the system. Similar to that in liquid helium, roton in TMGO has also a topological origin: we demonstrate below that the rotonlike modes represent bound states of topological vortex excitations, via a pseudo-spin mapping of the spin stripe order. As shown in Fig. 1c, the stripe order constitutes a proximate competing order to the clock state. Although it eventually gets perished in TMGO at low *T*, the stripe order leaves a “ghost image” in the excitation spectrum, i.e., rotonlike dip along the Γ-M-Γ path in Fig. 4a. Correspondingly, there exists an incipient *S*_{M} peak in the static structure factor at intermediate *T* (Fig. 5c), i.e., *ϵ*_{q} ~ 1/*S*(**q**), similar to rotons in the superfluid helium discussed above^{8,45}.

On top of the spin stripe order, the M rotonlike excitation can be related to a locally bounded vortex pair, some form of “rotational motion” happening in TMGO. We perform a pseudo-spin mapping

where *ψ* = ∣*ψ*∣e^{iθ} is the complex order parameter^{37}, i.e., the pseudo-spin. As shown in Fig. 1c, *ψ* is located in the center of each triangle, with emergent XY degree of freedom *θ*. In Eq. (2), \({m}_{\gamma }^{z}=\pm \! 1/2\) represents the spin-up(-down) of corresponding spin *S*^{z} components at *γ*-sublattice (*γ* = A, B, C). This mapping helps establishing a Landau–Ginzburg theory of TLI^{10,37} and the clock order shown in Fig. 1a corresponds to a ferromagnetic order of pseudo spins.

As shown in Fig. 1c, we create a vortex pair by applying *S*^{x} operator on two adjacent sites to simultaneously flip their spin orientations. We note that any closed loop enclosing only the vortex defect (red dot in Fig. 1c) leads to a winding number 1 (modulo 2*π*), whereas those around the antivortex (black dot) to −1. Zero winding number can be counted when a pair of defects (or no defects at all) are enclosed by the loop. Moreover, one can further move the vortex on the triangular lattice, such as in a “tight-binding” model, by flipping spins on further neighboring sites, which naturally leads to a quadratic-type low-energy dispersion near M point along Γ-M-Γ (see Supplementary Note 7). As the ghost peak in Fig. 5c only suggests a short-range stripe correlation, the vortex pair can thus move only within a small cluster with incipient stripe order, i.e., they are bounded. The vortex pair only unbinds at the upper KT transition *T*_{h} where vortices are proliferated^{2,3}, as seen by the “softening” of M rotonlike mode in dynamical spectrum in Fig. 4.

To conclude, in this work, we have established a protocol of understanding and explaining experiments of frustrated magnets in an unbiased manner, with XTRG and QMC-SAC machinery. The thermodynamic and dynamic results of TMGO are captured to great accuracy, thus allowing comprehensive studies of KT physics therein. At intermediate temperature, the KT phase of TMGO realizes a magnetic analog of 2D superfluid phase with several intriguing properties: (i) there emerges a spin XY symmetry and correspondingly complex order parameter *ψ*, which bears quasi-long-range correlation and phase coherence; (ii) the finite-*T* spin spectrum contains the long-wavelength magnon and competing gapped rotonlike modes near the BZ boundary with energy signifying the binding of vortex–antivortex pair, which plays a key role in determining finite-*T* phase diagram of the system; (iii) the quasi-long-range XY order melts and the TMGO becomes paramagnetic as *T* is above *T*_{h}, driven by the proliferation of vortex excitations, in analogy to the superfluid transition in a helium thin film^{4,54}. We note that, different from the liquid helium, TMGO has two-temperature scales that outline the intermediate-temperature KT phase. Similar separation of scales has been seen in other quantum magnetic materials, by temperature^{53,55} or spatial dimensions^{56}, but the KT phase is the first time to be seen. Nevertheless, the \({T}_{{\rm{l}}}^{* },{T}_{{\rm{h}}}^{* }\) in the present work constitute tentative estimated of two KT transition temperatures roughly from thermodynamics, which still needs to be precisely determined both numerically and experimentally in the future.

The extraction of anomalous exponents *η*(*T*) constitutes another interesting future study. The exponent *η*(*T*) of the KT phase appears in the algebraic correlations in \(\langle {S}_{i}^{z}{S}_{j}^{z}\rangle \sim | {r}_{i}-{r}_{j}{| }^{-\eta (T)}\) and seems rather indirect to measure in solid-state experiements. Nevertheless, as discussed above in Fig. 3, *η*(*T*) can be determined from the uniform susceptibility *χ*, a routine magnetic quantity in experiments. Moreover, it would be interesting to check several distinct predictions of KT physics in this 2D magnetic material. One renowned phenomenon is the universal jump in superfluid density at the KT transition^{57}, as observed experimentally in helium film^{4}. Through calculations of the *q*-clock model (*q* = 5, 6), people have revealed universal jumps in the spin stiffness at both upper and lower KT transitions^{58}, which certainly is an interesting prediction to check in TMGO. Dynamically, nuclear magnetic resonance measurements of relaxation time 1/*T*_{1} can be conducted, which probe signals of low-energy magnetic dissipations at the KT transition where vortices proliferate. Besides, non-equilibrium thermodynamics such as thermal transport, would also be very worthwhile to explore in the TMGO magnet.

## Methods

### Quantum many-body computations

In this work, we combined two many-body numerical approaches: QMC^{37} and XTRG, the latter method is recently introduced based on matrix product operators (MPOs) and logarithmic temperature scales^{27}. XTRG is employed to simulate the TLI down to temperatures *T* < 0.5 K on YC *W* × *L* geometries up to width *W* = 9 with various lengths up to *L* = 12. Both dynamical and equilibrium properties are simulated, with the purpose of fitting the experimental data and obtaining the right parameters, as well as to make predictions for experiments. The QMC is performed in the space-time lattice of *L* × *L* × *L*_{τ}, where *L* = 36 and *L*_{τ} = *β*/Δ*τ* with Δ*τ* = 0.05 and *β* ≡ 1/*T*. The space-time configuration is written in the \({S}_{i,\tau }^{z}\) basis with both local and Wolff-cluster updates to overcome the long autocorrelation time. As the QMC method is standard, we will only introduce the SAC scheme below and leave the QMC itself to the Supplementary Note 8.

### Exponential thermal tensor network method

For the calculations of equilibrium properties, we start from a high-*T* density matrix \(\hat{\rho }(\Delta \tau )={e}^{-\Delta \tau H}\), whose MPO representation can be obtained conveniently and accurately (up to machine precision), at a small Δ*τ* ~ 10^{−3~−4}. One way to obtain such an accurate MPO representation is to exploit the series-expansion thermal tensor network technique^{59} via the expansion

Given the \(\hat{\rho }(\Delta \tau )\) representation, traditionally one evolves \(\hat{\rho }(\beta )\) linear in *β* to reach various lower temperatures, i.e., *β* = *L*_{τ}Δ*τ* increases by a small value Δ*τ* after each step by multiplying \(\hat{\rho }(\Delta \tau )\) to the density matrix^{60}. However, this linear scheme is not optimal in certain aspects, and encounters challenges in generalization to 2D. Instead, recent study shows that the block entanglement entropy of MPO is bound by \({S}_{E}\le a\ \mathrm{ln}\,\beta +{\rm{const.}}\) at a conformal critical point, with *a* an universal coefficient proportional to the central charge^{27}. This suggests an exponential procedure of performing cooling procedure. Based on this idea, we have developed the XTRG method, which turns out to be highly efficient in simulating both one-dimensional (1D) critical quantum chains and various 2D lattice systems^{27,53,61}.

In XTRG, we cool down the system by multiplying the thermal state by itself, i.e., *ρ*_{0} ≡ *ρ*(Δ*τ*), *ρ*_{1} ≡ *ρ*_{0} ⋅ *ρ*_{0} = *ρ*(2Δ*τ*), thus *ρ*_{n} ≡ *ρ*_{n−1} ⋅ *ρ*_{n−1} = *ρ*(2^{n}Δ*τ*), and reach the low-*T* thermal states exponentially fast. Efficient compression of MPO bonds is then required to maintain the cooling procedure, where a truncation scheme optimizing the free energy and in the mean time maintaining the thermal entanglement, is involved. One advantage of XTRG is the convenience and high efficiency to deal with long-range interactions after the quasi-1D mapping. For the TLI model with NN (*J*_{1}) and NNN (*J*_{2}) interactions considered in this work, we map the 2D lattice into a quasi-1D geometry following a snake-like path. The Hamiltonian thus contains “long-range” interactions and has an efficient MPO representation with geometric bond dimension *D*_{H} = 2*W* + 2, with *W* the width of the lattice. In XTRG calculations, the computational costs scale with power *O*(*D*^{4}), with *D* the retained bond dimension in MPO, which is chosen as large as 500–600 in the present study, assuring accurate thermodynamical results down to sub-Kelvin regime.

### QMC-SAC approach

We exploit the path integral QMC^{37}, equipped with SAC approach, to compute the dynamical properties. The time displaced correlated function, defined as *G*(*τ*) = 〈*S*^{z}(*τ*)*S*^{z}(0)〉, for a set of imaginary times *τ*_{i}(*i* = 0, 1, ⋯ , *L*_{τ}) with statistical errors can be obtained from QMC simulations. By SAC method^{28,29,31,32}, the corresponding real-frequency spectral function *S*(*ω*) can be obtained via \(S(\tau )={\int }_{-\infty }^{\infty }d\omega S(\omega )K(\tau ,\omega )\), where the kernel *K*(*τ*, *ω*) depends on the type of the spectral function, i.e., fermionic or bosonic, finite or zero temperature. The spectra at positive and negative frequencies obey the relation of *S*(−*ω*) = *e*^{−βω}*S*(*ω*) and we are restricted at the positive frequencies and the kernel can therefore be written as \(K(\tau ,\omega )=\frac{1}{\pi }({e}^{-\tau \omega }+{e}^{-(\beta -\tau )\omega })\). To work with a spectral function that is itself normalized to unity, we further modify the kernel and the spectral function, and arrive at the transformation between the imaginary time Green’s function *G*(**q**, *τ*) and real-frequency spectral function *B*(**q**, *ω*)

where *B*(**q**, *ω*) = *S*(**q**, *ω*)(1 + *e*^{−βω}).

In the practical calculation, we parametrize the *B*(**q**, *ω*) with a large number of equal-amplitude *δ*-functions sampled at locations in a frequency continuum as \(B(\omega )=\sum _{i=0}^{{N}_{\omega }-1}{a}_{i}\delta (\omega -{\omega }_{i})\). Then the relationship between Green’s function obtained from Eq. (4) and from QMC can be described by the goodness of fit *χ*^{2}, i.e., \({\chi }^{2}=\sum _{i=1}^{{N}_{\tau }}\sum _{j=1}^{{N}_{\tau }}({G}_{i}-{\overline{G}}_{i}){C}_{ij}^{-1}({G}_{j}-{\overline{G}}_{j})\), where \({\overline{G}}_{i}\) is the average of QMC measurement and *C*_{ij} is covariance matrix \({C}_{ij}=\frac{1}{{N}_{B}({N}_{B} \,- \, 1)}\sum _{b=1}^{{N}_{B}}({G}_{i}^{b}-{\overline{G}}_{i})({G}_{j}^{b}-{\overline{G}}_{j})\), with *N*_{B} the number of bins. Then we update the series of *δ*-functions in a Metropolis process, from (*a*_{i}, *ω*_{i}) to \(({a}_{i}^{\prime},{\omega }_{i}^{\prime})\), to get a more probable configuration of *B*(**q**, *ω*). The weight for a given spectrum follows the Boltzmann distribution \(P(B)\propto \exp (-{\chi }^{2}/2\Theta )\), with Θ a fictitious temperature chosen in an optimal way so as to give a statistically sound mean *χ*^{2}-value, while still staying in the regime of significant fluctuations of the sampled spectra so that a smooth averaged spectral function is obtained. The resulting spectra will be collected as an ensemble average of the Metropolis process within the configurational space of {*a*_{i}, *ω*_{i}}, as detailed in refs. ^{28,29,31,32}.

## Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

## Code availability

All numerical codes in this paper are available upon request to the authors.

## References

- 1.
Mermin, N. D. & Wagner, H. Absence of ferromagnetism or antiferromagnetism in one- or two-dimensional isotropic Heisenberg models.

*Phys. Rev. Lett.***17**, 1133–1136 (1966). - 2.
Kosterlitz, J. M. & Thouless, D. J. Ordering, metastability and phase transitions in two-dimensional systems.

*J. Phys. C: Solid State Phys.***6**, 1181–1203 (1973). - 3.
Kosterlitz, J. M. The critical properties of the two-dimensional xy model.

*J. Phys. C: Solid State Phys.***7**, 1046–1060 (1974). - 4.
Bishop, D. J. & Reppy, J. D. Study of the superfluid transition in two-dimensional

^{4}He films.*Phys. Rev. Lett.***40**, 1727–1730 (1978). - 5.
Hadzibabic, Z., Krüger, P., Cheneau, M., Battelier, B. & Dalibard, J. Berezinskii-Kosterlitz-Thouless crossover in a trapped atomic gas.

*Nature***441**, 1118–1121 (2006). - 6.
Desbuquois, R. et al. Superfluid behaviour of a two-dimensional Bose gas.

*Nature Phys.***8**, 645–648 (2012). - 7.
Landau, L. On the theory of superfluidity.

*Phys. Rev.***75**, 884–885 (1949). - 8.
Feynman, R. P.

*Progress in Low Temperature Physics: Chapter II, Application of Quantum Mechanics to Liquid Helium*(ed Gorter, C. G.) Vol. 1 (North Holland, 1955). - 9.
Bendt, P. J., Cowan, R. D. & Yarnell, J. L. Excitations in liquid helium: thermodynamic calculations.

*Phys. Rev.***113**, 1386–1395 (1959). - 10.
Isakov, S. V. & Moessner, R. Interplay of quantum and thermal fluctuations in a frustrated magnet.

*Phys. Rev. B***68**, 104409 (2003). - 11.
Zhou, H. D. et al. Successive phase transitions and extended spin-excitation continuum in the \(S\, {\boldsymbol{=}}\,\frac{1}{2}\) triangular-lattice antiferromagnet Ba

_{3}CoSb_{2}O_{9}.*Phys. Rev. Lett.***109**, 267206 (2012). - 12.
Susuki, T. et al. Magnetization process and collective excitations in the

*S*= 1/2 triangular-lattice Heisenberg antiferromagnet Ba_{3}CoSb_{2}O_{9}.*Phys. Rev. Lett.***110**, 267201 (2013). - 13.
Ma, J. et al. Static and dynamical properties of the spin-1/2 equilateral triangular-lattice antiferromagnet Ba

_{3}CoSb_{2}O_{9}.*Phys. Rev. Lett.***116**, 087201 (2016). - 14.
Ito, S. et al. Structure of the magnetic excitations in the spin-1/2 triangular-lattice Heisenberg antiferromagnet Ba

_{3}CoSb_{2}O_{9}.*Nat. Commun.***8**, 235 (2017). - 15.
Rawl, R. et al. Ba

_{8}CoNb_{6}O_{24}: a spin-\(\frac{1}{2}\) triangular-lattice Heisenberg antiferromagnet in the two-dimensional limit.*Phys. Rev. B***95**, 060412(R) (2017). - 16.
Cui, Y. et al. Mermin-Wagner physics, (H,T) phase diagram, and candidate quantum spin-liquid phase in the spin-\(\frac{1}{2}\) triangular-lattice antiferromagnet Ba

_{8}CoNb_{6}O_{24}.*Phys. Rev. Mater.***2**, 044403 (2018). - 17.
Li, Y. et al. Rare-earth triangular lattice spin liquid: a single-crystal study of YbMgGaO

_{4}.*Phys. Rev. Lett.***115**, 167203 (2015). - 18.
Li, Y. et al. Muon spin relaxation evidence for the U(1) quantum spin-liquid ground state in the triangular antiferromagnet YbMgGaO

_{4}.*Phys. Rev. Lett.***117**, 097201 (2016). - 19.
Shen, Y. et al. Evidence for a spinon Fermi surface in a triangular-lattice quantum-spin-liquid candidate.

*Nature***540**, 559–562 (2016). - 20.
Li, Y. et al. Rearrangement of uncorrelated valence bonds evidenced by low-energy spin excitations in YbMgGaO

_{4}.*Phys. Rev. Lett.***122**, 137201 (2019). - 21.
Zhu, Z., Maksimov, P. A., White, S. R. & Chernyshev, A. L. Disorder-induced mimicry of a spin liquid in YbMgGaO

_{4}.*Phys. Rev. Lett.***119**, 157201 (2017). - 22.
Kimchi, I., Nahum, A. & Senthil, T. Valence bonds in random quantum magnets: Theory and application to YbMgGaO

_{4}.*Phys. Rev. X***8**, 031028 (2018). - 23.
Ma, Z. et al. Spin-glass ground state in a triangular-lattice compound YbZnGaO

_{4}.*Phys. Rev. Lett.***120**, 087201 (2018). - 24.
Cevallos, F. A., Stolze, K., Kong, T. & Cava, R. J. Anisotropic magnetic properties of the triangular plane lattice material TmMgGaO

_{4}.*Mater. Res. Bull.***105**, 154–158 (2018). - 25.
Li, Y. et al. Partial up-up-down order with the continuously distributed order parameter in the triangular antiferromagnet TmMgGaO

_{4}.*Phys. Rev. X***10**, 011007 (2020). - 26.
Shen, Y. et al. Intertwined dipolar and multipolar order in the triangular-lattice magnet TmMgGaO

_{4}.*Nat. Commun.***10**, 4530 (2019). - 27.
Chen, B.-B., Chen, L., Chen, Z., Li, W. & Weichselbaum, A. Exponential thermal tensor network approach for quantum lattice models.

*Phys. Rev. X***8**, 031082 (2018). - 28.
Sandvik, A. W. Constrained sampling method for analytic continuation.

*Phys. Rev. E***94**, 063308 (2016). - 29.
Qin, Y. Q., Normand, B., Sandvik, A. W. & Meng, Z. Y. Amplitude mode in three-dimensional dimerized antiferromagnets.

*Phys. Rev. Lett.***118**, 147207 (2017). - 30.
Shao, H., Qin, Y. Q., Capponi, S., Chesi, S., Meng, Z. Y. & Sandvik, A. W. Nearly deconfined spinon excitations in the square-lattice spin-1∕2 Heisenberg antiferromagnet.

*Phys. Rev. X***7**, 041072 (2017). - 31.
Huang, C.-J., Deng, Y., Wan, Y. & Meng, Z. Y. Dynamics of topological excitations in a model quantum spin ice.

*Phys. Rev. Lett.***120**, 167202 (2018). - 32.
Sun, G.-Y. et al. Dynamical signature of symmetry fractionalization in frustrated magnets.

*Phys. Rev. Lett.***121**, 077201 (2018). - 33.
Kresse, G. & Furthmüller, J. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.

*Phys. Rev. B***54**, 11169–11186 (1996). - 34.
Kresse, G. & Joubert, D. From ultrasoft pseudopotentials to the projector augmented-wave method.

*Phys. Rev. B***59**, 1758–1775 (1999). - 35.
Giannozzi, P. et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials.

*J. Phys. Condens. Matter***21**, 395502 (2009). - 36.
Giannozzi, P. et al. Advanced capabilities for materials modelling with Quantum ESPRESSO.

*J. Phys. Condens. Matter***29**, 465901 (2017). - 37.
Wang, Y.-C., Qi, Y., Chen, S. & Meng, Z. Y. Caution on emergent continuous symmetry: a Monte Carlo investigation of the transverse-field frustrated Ising model on the triangular and honeycomb lattices.

*Phys. Rev. B***96**, 115160 (2017). - 38.
Powalski, M., Coester, K., Moessner, R. & Schmidt, K. P. Disorder by disorder and flat bands in the kagome transverse field Ising model.

*Phys. Rev. B***87**, 054404 (2013). - 39.
Fey, S., Kapfer, S. C. & Schmidt, K. P. Quantum criticality of two-dimensional quantum magnets with long-range interactions.

*Phys. Rev. Lett.***122**, 017203 (2019). - 40.
Metcalf, B. D. Ground state spin orderings of the triangular ising model with the nearest and next nearest neighbor interaction.

*Phys. Lett. A***46**, 325–326 (1974). - 41.
Wheeler, E. M. et al. Spin dynamics of the frustrated easy-axis triangular antiferromagnet 2

*H*-AgNiO_{2}explored by inelastic neutron scattering.*Phys. Rev. B***79**, 104421 (2009). - 42.
Damle, K. Melting of three-sublattice order in easy-axis antiferromagnets on triangular and kagome lattices.

*Phys. Rev. Lett.***115**, 127204 (2015). - 43.
Biswas, S. & Damle, K. Singular ferromagnetic susceptibility of the transverse-field Ising antiferromagnet on the triangular lattice.

*Phys. Rev. B***97**, 085114 (2018). - 44.
Donnelly, R. J. In

*Quantum Statistical Mechanics in the Natural Sciences: A Volume Dedicated to Lars Onsager on the Occasion of his Seventieth Birthday*(eds. Mintz, S. L. & Widmayer, S. M.) p. 359–402 (Springer US, Boston, MA, 1974). - 45.
Nozières, P. Is the roton in superfluid He

^{4}the ghost of a Bragg spot?*J. Low Temp. Phys.***137**, 45–67 (2004). - 46.
Carneiro, K., Ellenson, W. D., Passell, L., McTague, J. P. & Taub, H. Neutron-scattering study of the structure of adsorbed helium monolayers and of the excitation spectrum of few-atomic-layer superfluid films.

*Phys. Rev. Lett.***37**, 1695–1698 (1976). - 47.
Starykh, O. A., Chubukov, A. V. & Abanov, A. G. Flat spin-wave dispersion in a triangular antiferromagnet.

*Phys. Rev. B***74**, 180403(R) (2006). - 48.
Chernyshev, A. L. & Zhitomirsky, M. E. Magnon decay in noncollinear quantum antiferromagnets.

*Phys. Rev. Lett.***97**, 207202 (2006). - 49.
Zheng, W., Fjærestad, J. O., Singh, R. R. P., McKenzie, R. H. & Coldea, R. Anomalous excitation spectra of frustrated quantum antiferromagnets.

*Phys. Rev. Lett.***96**, 057201 (2006). - 50.
Zheng, W., Fjærestad, J. O., Singh, R. R. P., McKenzie, R. H. & Coldea, R. Excitation spectra of the spin-\(\frac{1}{2}\) triangular-lattice Heisenberg antiferromagnet.

*Phys. Rev. B***74**, 224420 (2006). - 51.
Verresen, R., Moessner, R. & Pollmann, F. Avoided quasiparticle decay from strong quantum interactions.

*Nat. Phys.***15**, 750–753 (2019). - 52.
Ferrari, F. & Becca, F. Dynamical structure factor of the

*J*_{1}−*J*_{2}Heisenberg model on the triangular lattice: magnons, spinons, and gauge fields.*Phys. Rev. X***9**, 031026 (2019). - 53.
Chen, L. et al. Two-temperature scales in the triangular lattice Heisenberg antiferromagnet.

*Phys. Rev. B***99**, 140404(R) (2019). - 54.
Minnhagen, P. The two-dimensional Coulomb gas, vortex unbinding, and superfluid-superconducting films.

*Rev. Mod. Phys.***59**, 1001–1066 (1987). - 55.
Banerjee, A. et al. Proximate kitaev quantum spin liquid behaviour in a honeycomb magnet.

*Nat. Mater.***15**, 733–740 (2016). - 56.
Lake, B., Tennant, D. A., Frost, C. D. & Nagler, S. E. Quantum criticality and universal scaling of a quantum antiferromagnet.

*Nat. Mater.***4**, 329–334 (2005). - 57.
Nelson, D. R. & Kosterlitz, J. M. Universal jump in the superfluid density of two-dimensional superfluids.

*Phys. Rev. Lett.***39**, 1201–1205 (1977). - 58.
Chatelain, C. DMRG study of the Berezinskii–Kosterlitz–Thouless transitions of the 2D five-state clock model.

*J. Stat. Mech*. P11022 (2014). - 59.
Chen, B.-B., Liu, Y.-J., Chen, Z. & Li, W. Series-expansion thermal tensor network approach for quantum lattice models.

*Phys. Rev. B***95**, 161104(R) (2017). - 60.
Li, W. et al. Linearized tensor renormalization group algorithm for the calculation of thermodynamic properties of quantum lattice models.

*Phys. Rev. Lett.***106**, 127202 (2011). - 61.
Li, H. et al. Thermal tensor renormalization group simulations of square-lattice quantum spin models.

*Phys. Rev. B***100**, 045110 (2019).

## Acknowledgements

We are indebted to Ziyu Chen, Wentao Jin, Andreas Weichselbaum, Yao Shen, and Yuesheng Li for stimulating discussions. This work was supported by the Ministry of Science and Technology of China through the National Key Research and Development Program (Grant Number 2016YFA0300502), the Strategic Priority Research Program of the Chinese Academy of Sciences (Grant Number XDB28000000), the National Science Foundation of China (Grant Numbers 11421092, 11574359, 11674370, 11974036, 11874115, and 11834014), and Research Grants Council of Hong Kong Special Administrative Region of China through 17303019 and the Aspen Center for Physics, which is supported by National Science Foundation grant PHY-1607611. B.-B.C. was supported by the German Research foundation, DFG WE4819/3-1. W.L. was supported by the Fundamental Research Funds for the Central Universities. We thank the Center for Quantum Simulation Sciences at Institute of Physics, Chinese Academy of Sciences, the Computational Initiative at the Faculty of Science at the University of Hong Kong, the Tianhe-1A platform at the National Supercomputer Center in Tianjin, and Tianhe-2 platform at the National Supercomputer Center in Guangzhou for their technical support and generous allocation of CPU time.

## Author information

### Affiliations

### Contributions

H.L. and Y.D.L. contributed equally to this work. W.L., Y.Q., and Z.Y.M. initiated the work. H.L. and B.B.C. performed the calculations of thermodynamics. X.T.Z. and X.L.S. performed first-principle calculations of the band structure and Y.D.L. performed the dynamical property simulations. All authors contributed to the analysis of the results. Z.Y.M. and W.L. supervised the project.

### Corresponding authors

Correspondence to Yang Qi or Zi Yang Meng or Wei Li.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

## Additional information

**Peer review information** *Nature Communications* thanks Arnab Banerjee, Frédéric Mila and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Supplementary information

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Li, H., Liao, Y.D., Chen, B. *et al.* Kosterlitz-Thouless melting of magnetic order in the triangular quantum Ising material TmMgGaO_{4}.
*Nat Commun* **11, **1111 (2020). https://doi.org/10.1038/s41467-020-14907-8

Received:

Accepted:

Published:

## Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.