Frustration is a key concept to understand various emergent phenomena in modern many-body physics1,2. When different interactions among particles strongly compete with each other, e.g., for a geometric reason, the system is “frustrated” in determining the true ground state. The study on the interplay of the frustration and strong quantum fluctuations has been one of the core challenges of quantum many-body physics, presenting many open problems in connection with nontrivial magnetic states including quantum spin liquids3 and as a challenge for numerical techniques to handle highly entangled ground states4,5. Quantum simulation with the use of ultracold atomic gases in optical lattices6,7,8 has been discussed as a promising approach to make a critical breakthrough in this research area. However, there still remain many challenges that have to be overcome in realizing and controlling frustrated quantum systems with cold atoms, whereas many theoretical proposals have been made9,10,11,12,13,14.

One straightforward idea for creating frustration is the use of two-component Fermi gases with (pseudo)spins σ = 15,16,17,18,19,20,21,22,23 loaded into a nonbipartite (e.g., triangular24 and kagome25) optical lattice. The second-order hopping process provides antiferromagnetic superexchange interactions between the (pseudo)spins σ26,27, which result in a frustrated situation, because complete staggered spin configuration is not allowed by the lattice geometry. Although long-range magnetic correlation over a distance comparable to the system size has recently been observed in a square optical lattice21, a further technical breakthrough is required to realize far lower temperatures to study frustrated quantum magnetism. Another interesting idea is a fast shaking of optical lattice, which can effectively invert the sign of the hopping integral from the natural one28,29. For ultracold Bose gases with sign-inverted hopping, the relative local phase of Bose–Einstein condensates (BECs) tends to be π on neighboring sites, analogous to antiferromagnetic spin coupling, which induces geometric frustration in nonbipartite lattices. A frustrated classical XY model has been successfully simulated with this technique30. However, it is challenging to reach a quantum regime of low temperature and density, because the lattice shaking can be a source of heating.

Recently, it has become realistic to create a well-controlled system at negative absolute temperatures31 in laboratory. A state in thermal equilibrium is usually described by a statistical ensemble in which the lower-energy states are more occupied than higher-energy ones, obeying the probability proportional to the Boltzmann factor with temperature T ≥ 0. However, if the system has an upper energy bound, the opposite distribution with the largest occupation of the highest energy could also manage to be prepared. Such a state is characterized by a negative absolute temperature T ≤ 0. In a pioneer work, Braun et al.32 have created a thermodynamically stable negative-temperature state of Bose gases in a square optical lattice by achieving the maximum interaction and potential energies in the regime of negligible kinetic energy. There the absolute temperature remains so low that the quantum phase transition from the Mott insulator (MI) to the superfluid (SF) has been observed.

Here we propose and examine a realistic route to create frustrated Bose gases in a quantum regime using the combination of the phase-imprinting techniques33,34,35 and the statistics of negative absolute temperatures. Our proposal is based on the fact that a negative-temperature state of a system \(\hat{{\mathcal{H}}}\) at T < 0 realizes the corresponding equilibrium state of the sign-inverted Hamiltonian \(- {\hat{{\mathcal{H}}}}\) at T > 0. Using this, one can achieve the same effect as inverting the sign of hopping, instead, by inverting the other factors, namely interatomic interaction and trap potential, and then preparing a negative-temperature state. To this end, we propose a phase-imprinting scheme combined with sudden sign inversion of the interaction and potential, which causes much less heating compared with the lattice shaking. Supposing Bose gases in a triangular lattice, we simulate the dynamics along the protocol within the time-dependent Gutzwiller approach (TDGA) to demonstrate that quantum phases of the frustrated Bose–Hubbard model, including chiral SF (CSF), could indeed be realized. Moreover, considering more general hoppings with spatial anisotropy, in which the hopping amplitude in one direction can be different from those in the other two directions, we give more quantitative analysis on the quantum phase transition between frustrated CSF and MI by means of the cluster mean-field plus scaling (CMF + S) method36,37,38 with a two-dimensional (2D) density matrix renormalization group (DMRG) solver14. This enables us to discuss the interplay of frustration and quantum fluctuations, which is a critical factor for producing various exotic frustrated states. The theoretical predictions provide a solid guidepost for future experiments to confirm that the interplay effects are properly captured in the quantum simulator. We set  = 1, except in the figures.


The Bose–Hubbard model on triangular lattice

A system of Bose gases in a deep optical lattice is described by the Bose–Hubbard model:

$$\hat{{\mathcal{H}}}=-\sum _{i,j}{J}_{ij}{\hat{b}}_{i}^{\dagger }{\hat{b}}_{j}+\frac{U}{2}\sum _{i}{\hat{n}}_{i}({\hat{n}}_{i}-1)+\frac{V}{{a}^{2}}\sum _{i}| {{\bf{r}}}_{i}{| }^{2}{\hat{n}}_{i}$$

with hopping integral Jij (for i ≠ j), chemical potential Jii ≡ μ, onsite interaction U, harmonic trap potential Vri2a2, and lattice constant a. Here we consider a triangular optical lattice with spatially anisotropic hopping of “isosceles type,” which is parameterized by Jij = J1 for nearest-neighbor (NN) sites (ij) in the a1 = (a, 0) direction and Jij = J2 in the \({{\bf{a}}}_{2}=(-a/2,\sqrt{3}a/2)\) and \({{\bf{a}}}_{3}=(-a/2,-\sqrt{3}a/2)\) directions (see Fig. 1). The spatial anisotropy can be created by tuning the intensity of one of the three lasers to be different from the others (see Methods). The two extreme limits, J1/J2 = 0 and J1/J2 1, are reduced to square lattice and one-dimensional (1D) chain, respectively.

Fig. 1: Triangular optical lattice with anisotropic hoppings.
figure 1

a Typical potential landscape of isosceles triangular optical lattice created by three laser beams in the case where one of the three lasers (in the a1 = (a, 0) direction with a being the lattice constant) has a larger intensity than the other two (in the a2 \(=(-a/2,\sqrt{3}a/2)\) and a3 \(=(-a/2,-\sqrt{3}a/2)\) directions). b Schematic figures of the modeling with Bose–Hubbard parameters: the hoppings J1 in the a1 direction and J2 in the a2 and a3 directions, and the onsite interparticle interaction U. The size of the arrows schematically shows the difference of the hopping amplitudes depending on the potential height between the neighboring lattice sites in each direction.

Gutzwiller analysis

We first discuss the ground state for V = 0 within the site-decoupling Gutzwiller approach (GA)39,40,41 to get a basic insight into the problem. The ground state in the weak-coupling regime (UJij) is well described within the GA under the assumption of the BEC order \(\langle {b}_{i}\rangle \equiv {\psi }_{i}=\overline{\psi }{e}^{i{\bf{q}}\cdot {{\bf{r}}}_{i}+\varphi }\) with momentum q and global phase φ. The kinetic energy of Eq. (1) is given by \(-{\sum }_{i,j\ne i}{J}_{ij}\langle {\hat{b}}_{i}^{\dagger }{\hat{b}}_{j}\rangle \approx {\varepsilon }_{{\bf{q}}}{\overline{\psi }}^{2}M\). Here, \({\varepsilon }_{{\bf{q}}}\equiv -2({J}_{1}\cos {\bf{q}}\cdot {{\bf{a}}}_{1}+{J}_{2}\cos {\bf{q}}\cdot {{\bf{a}}}_{2}+{J}_{2}\cos {\bf{q}}\cdot {{\bf{a}}}_{3})\) and M is the number of lattice sites. For natural-sign hoppings J1J2 > 0, the minimum kinetic energy is obtained at q = 0, leading to a uniform SF state. The maximum of the kinetic energy is achieved at q = ±Q with

$${\bf{Q}}=\left\{\begin{array}{lll}(2\pi /a,0)\equiv {{\bf{Q}}}_{{\rm{M}}}&{\rm{for}}&0\ \le \ {J}_{1}/{J}_{2}\ \le \ 0.5,\\ (2\arccos [\frac{-{J}_{2}}{2{J}_{1}}]/a,0)&{\rm{for}}&{J}_{1}/{J}_{2}\,> \, 0.5.\end{array}\right.$$

Therefore, if sign-inverted hopping (J1J2 < 0) is prepared, a frustrated CSF state with finite BEC momentum q = ±Q is realized. The choice of q = Q or  −Q represents the degeneracy with respect to the chirality of vortex in unit triangles. Hereafter, we suppose q = Q to be spontaneously selected. In the equilateral case (J1 = J2 ≡ J), the momentum is Q = (4π/3a, 0) ≡ QK. Therefore, the CSF state forms a “three-color” arrangement of the local phase (\({\rm{Arg}}[{\psi }_{i}]=0\), 2π/3, and 4π/3 within a global phase shift) as shown in Fig. 2a. This can be understood as a compromise solution for the frustration in the bond energy minimization. For generic J1/J2 > 0.5, the phase factor changes spatially with incommensurate pitch vector Q. In the parameter range 0 ≤ J1/J2 ≤ 0.5 and the 1D limit J1/J2 1, a “two-color” (0 and π) pattern is formed with no chiral degeneracy. The J1/J2 dependence of Q is analogous to the pitch vector of spin spiral states in spatially anisotropic triangular antiferromagnets42,43,44,45.

Fig. 2: Predictions from the Gutzwiller approach.
figure 2

a The J1/J2 dependence of the local phase pattern in the frustrated case J1J2 < 0 is illustrated. The colors indicate the sublattice structure in the local phase and the arrows depict the chirality. b, c The order parameter \(\overline{\psi }\) as a function of U/J2 (being the ratio of the interaction and the hopping in the a2 and a3 directions) at the filling factor ρ = 1 for b J1J2 > 0 and c J1J2 <  0. The value of \(\overline{\psi }\) becomes zero at the transition from the superfluid (SF) or chiral superfluid (CSF) to Mott insulator (MI) phase.

For large, repulsive interaction U, lattice bosons undergo a quantum phase transition to the MI state when the filling factor \(\rho \equiv {M}^{-1}{\sum }_{i}\langle {\hat{n}}_{i}\rangle\) is an integer46. Performing the GA decoupling \({\hat{b}}_{i}^{\dagger }{\hat{b}}_{j}\approx {\psi }_{j}{\hat{b}}_{i}^{\dagger }+{\psi }_{i}^{* }{\hat{b}}_{j}-{\psi }_{i}^{* }{\psi }_{j}\) in Eq. (1), we calculate the order parameter \(\overline{\psi }\) for ρ = 1 in the unfrustrated (J1J2 > 0: q = 0) and frustrated (J1J2 < 0: q = Q) ground states as a function of U/J2 (see Methods). Figure 2b, c show the GA results for the SF-to-MI and CSF-to-MI quantum phase transitions, respectively. At J1 = J2  = J, the critical point at which \(\overline{\psi }\) vanishes is given as \({U}_{{\rm{c}}}^{({\rm{GA}})}/| J| =17.5\) for the frustrated case, which is a half of \({U}_{{\rm{c}}}^{({\rm{GA}})}/J=35.0\) for the unfrustrated case. The strong reduction is attributed by the fact that the CSF state is less stable due to the frustrated local-phase arrangement in which the NN bonds are not fully satisfied in minimizing the local kinetic energy. For general values of the anisotropy J1/J2 and ρ, the critical point is given by \({U}_{{\rm{c}}}^{({\rm{GA}})}=-{\varepsilon }_{{\bf{q}}}{(\sqrt{\rho }+\sqrt{\rho +1})}^{2}\) with q = 0 (q = Q) for the unfrustrated (frustrated) case. The ratio εQ/ε0 equals to 1 only at J1/J2 = 0 or J1/J2 → , indicating that the reduction effect due to frustration exists even in the “two-color” region of 0 < J1/J2 ≤ 0.5.

It is noted that the curves of \(\overline{\psi }\) for the unfrustrated system in Fig. 2b exactly overlap those for the frustrated system in Fig. 2c, respectively, for each J1/J2, by changing the scale of the interaction U by the factor εQ0 (specifically, the factor 1/2 when J1 = J2). This is the case for general values of J1/J2 and ρ. Indeed, the ground-state properties, including the density and order parameter, for the unfrustrated system with certain JijU > 0 and those for the frustrated system with negative hopping −Jij and repulsion εQ/ε0U are identical within the GA, except for the spatial phase distribution \({\rm{Arg}}[{\psi }_{i}]={\bf{Q}}\cdot {{\bf{r}}}_{i}\). This is because the scale of the kinetic energy for the frustrated system is reduced by the factor εQ  ∕ ε0 due to the frustrated phase configuration (see Methods).

Negative absolute temperature

To experimentally create the frustrated quantum states, it is required to prepare sign-inverted hoppings Jij < 0 with avoiding serious heating of the system. Below, we explain the details of the protocol through the use of negative-temperature statistics. Let us suppose an initial state in which N particles are distributed in a triangular lattice and a harmonic potential, which realizes the SF ground state of the standard Bose–Hubbard model (1) with Jij > 0 and UV > 0. See a typical example in Fig. 3a obtained within the GA41 for N = 1400, J1 = J2 = J = 0.08U0, U = U0, and V = 0.001U0 (with U0 > 0 being the energy unit). The phase \({\rm{Arg}}[{\psi }_{i}]\) is uniform in the unfrustrated SF state.

Fig. 3: Dynamical stability of the negative-temperature chiral superfluid (CSF) state within the time-dependent Gutzwiller approach.
figure 3

a Profiles of density ni (solid lines) and order parameter ψi (dashed lines) in the harmonic trap along the cut of \({y}_{i}=a/\sqrt{3}\) (with a being the lattice constant) in the initial state. b The three operations to make frustrated negative-temperature states (for spatially isotropic hoppings J1 = J2 = J). The single-particle spectrum εq in units of J is plotted as a function of the wave vector q in units of πa−1. The illustrations depict a tilting of the optical lattice by energy difference δE for the phase imprinting, the inverting from replusive (U > 0) to attractive (−U/2 < 0) interaction, and the inverting from confinement (V > 0) to anti-confinement (−V/2 < 0) trap potential, respectively. c Negative-temperature CSF state after time 200 in units of the inverse of the initial interaction strength \({U}_{0}^{-1}\), which shows its stability for a sufficiently long time. The case without the phase imprinting operation (d) is shown in e for a comparison. f The evolution of the density fluctuation δn2 of the initial state with and without the phase imprinting.

First, one needs to introduce the spatial phase distribution \({\rm{Arg}}[{\psi }_{i}]={\bf{Q}}\cdot {{\bf{r}}}_{i}\) to create the frustrated CSF state. To this end, here we suggest the use of the phase-imprinting techniques33,34,35. When a single-particle energy difference δE is introduced between two sites, the relative phase on the two sites starts to develop with \(\exp [i\delta Et]\) in time t. In a region deep inside the SF phase, the kinetic energy (εq) reaches the maximum when the BEC momentum takes q = Q given in Eq. (2). One can transfer q from 0 to Q by introducing a temporary linear gradient potential \({\hat{V}}^{{\rm{ext}}}=\delta E{\sum }_{i}({x}_{i}/a){\hat{n}}_{i}\) for appropriate time δt = Qxa/δE (see Fig. 3b). Such a temporary potential can be created, e.g., by a magnetic field gradient or by an extra 1D optical lattice satisfying Q = QM or Q = QK (see Methods). One has to perform the phase-imprinting operation in a much shorter time than the time scale of the hopping (δt Jij−1), to affect only the local phases. Besides, the local energy difference δE has to be set to a large enough value compared with U and V so that one can safely avoid the influence of the inhomogeneity of the density profile (see Methods). As such well-controlled phase imprinting has been successfully made in previous experiments34,35, we will assume in the theoretical simulations presented below that a perfect phase imprinting can be achieved. The created CSF state with the “forced” phase distribution \({e}^{i{\bf{Q}}\cdot {{\bf{r}}}_{i}}\) should of course be dynamically unstable, as it has the maximum kinetic energy.

By changing U and V to be attractive (U < 0) and anti-trapping (V < 0), we make the interaction and potential energies also reach their maximum, to realize a stable negative-temperature ground state (at T ≈ −0) for Jij > 0 and UV < 0. The created negative-temperature state should be equivalent to the ground state (at T ≈ +0) of the frustrated system with Jij < 0 and UV > 0, as the physics of the two systems obey the same factor \(\exp [-\hat{{\mathcal{H}}}/{k}_{{\rm{B}}}T]\). In ref. 32 for square (unfrustrated) lattice, this has been performed simply with U → −U and V → −V by using the Feshbach resonance47 and blue-detuned lasers. In the present case with frustration, one has to pay special attention to the change of the kinetic energy scale after the sign inversion of U and V. Specifically, the initial state with the density (ni) and order-parameter (ψi) distributions shown in Fig. 3a after the phase imprinting \({\rm{Arg}}[{\psi }_{i}]={\bf{Q}}\cdot {{\bf{r}}}_{i}\) is expected to correspond to the ground state of the frustrated Hamiltonian with the interactions rescaled by the factor εQ/ε0, as explained above in the homogeneous case. Therefore, the values of U and V have to be changed as U → −εQ/ε0U and V → −εQ/ε0V (e.g., U → −U/2 and V → −V/2 for J1 = J2), to adjust the energy scale in consideration of the reduction of the kinetic energy due to the frustration.

TDGA simulation

In the framework of the TDGA48,49, we simulate the time evolution of the initial state in Fig. 3a after suddenly performing the three operations shown in Fig. 3b. In the simulation, we implement the phase imprinting on the initial state by the operation \({\sum }_{n}\; {f}_{i}^{(n)}\left|n\right\rangle \to {\sum }_{n}{e}^{in{{\bf{Q}}}_{{\rm{K}}}\cdot {{\bf{r}}}_{i}}{f}_{i}^{(n)}\left|n\right\rangle\) on the local wave function at every site i. Here, \({f}_{i}^{(n)}\) is the coefficient of the Fock basis \(\left|n\right\rangle\). In addition, we perform the sudden changes of U = U0 → −U0/2 and V = 0.001U0 → −0.0005U0. After those three operations are performed at t = 0, we calculate the time evolution of the state fixing the value of U/J for \(0<t\ \le \ 200{U}_{0}^{-1}\) to see the stability of the created negative-temperature CSF state.

As shown in Fig. 3c, the created negative-temperature state is predicted to be indeed dynamically stable for a long time \(0<t\ \le \ 200{U}_{0}^{-1}\). As a reference for the comparison, we also simulate the case with the same settings but without the phase-imprinting operation (Fig. 3d). In this case, as shown in Fig. 3e, the state collapses immediately within \(t\;\lesssim\; 2{U}_{0}^{-1}\) due to the dynamical instability; the order parameter rapidly decreases, although the density profile is kept. The difference between the two cases (with and without the phase imprinting) can be clearly seen in the time evolution of the density fluctuation \(\delta {n}^{2}\equiv \overline{\langle {\hat{n}}_{i}^{2}\rangle -{\langle {\hat{n}}_{i}\rangle }^{2}}\) averaged over the center sites within ri ≤ 10a. As shown in Fig. 3f, the value of δn2 rapidly increases and then exhibits an irregular oscillation in the case without the phase imprinting, whereas it is almost constant in the case that the negative-temperature state is properly created by simultaneously achieving the maximum kinetic, interaction, and potential energies.

The CSF-to-MI phase transition

For the stable negative-temperature CSF state shown in Fig. 3c, we slowly increase the value of U/J for \(t\, > \, 200{U}_{0}^{-1}\) to observe the CSF-to-MI transition. In experiments, the tuning of U/J is performed by controlling the height of the optical lattice46. Here we assume that U/J increases simply through J decreasing linearly with J = 0.08U0 − 0.0001(tU0 − 200)U0 for \(t\, > \, 200{U}_{0}^{-1}\). As shown in Fig. 4a, when U/J increases, the MI plateau is gradually formed and ψi in the trap center decreases towards zero. Until the transition point, the three-color phase profile in the CSF state is properly kept within a global phase shift (Fig. 4b, c).

Fig. 4: Quantum phase transition from chiral superfluid (CSF) to Mott insulator (MI) within the time-dependent Gutzwiller approach.
figure 4

a Time evolution of the profiles of density ni (solid lines) and order parameter ψi (dashed lines) in the harmonic trap along the cut of \({y}_{i}=a/\sqrt{3}\) (with a being the lattice constant) for the negative-temperature CSF state when U/J increases for \(t\,> \, 200{U}_{0}^{-1}\). b, c Color plots for the local phase near the center of the trap at b \(t=100{U}_{0}^{-1}\) and c \(700{U}_{0}^{-1}\). d, e Time evolution of the density fluctuation δn2(t)∕δn2(0) for d a negative-temperature CSF state and e an unfrustrated superfluid (SF) state. The red solid and blue dashed lines represent the cases of harmonic and box-shaped trap potentials, respectively. The time schedule for increasing the ratio of the interaction and the hopping, U/J, is plotted together. The time t is measured in units of the inverse of the initial interaction strength \({U}_{0}^{-1}\).

To show the transition process more clearly, we plot in Fig. 4d the time evolution of the scaled value of the density fluctuation δn2(t) ∕ δn2(0). There we see that δn2(t) ∕ δn2(0) decreases with U/J and almost vanishes at \(| U/J| \approx | {U}_{{\rm{c}}}^{({\rm{GA}})}/J| =17.5\), which is consistent with the GA prediction of the critical point for the frustrated system. In the case of a box-shaped trap potential50,51, which is modeled by V = 0 and the open boundary at ri = 36a, the transition is more sharply observed (the blue dashed line in Fig. 4d for N = 4692). We also plot in Fig. 4e the unfrustrated case of the standard SF-MI transition as a reference, which shows a sharp difference from the frustrated case in the values of U/J of the transition region.

At the end of this section, let us briefly mention the validity and limitation of the TDGA method with respect to the calculations presented above. The mean-field analysis with the site-decoupling approximation reproduces the exact wave function in the weak-coupling (U ≈ 0) and strong-coupling (U → ) limits, and is thus expected to reasonably interpolate the two limits in two or higher dimensions6,39,52. In fact, the TDGA simulation has been often used to describe the dynamics of Bose–Hubbard systems53,54,55,56, which should be fairly reliable, at least, in the absence of strong quantum correlations, e.g., in the deep-SF regime or in three dimensions (3D)57. It is therefore expected that the above TDGA simulation gave a proper description for the dynamical stability of the negative-temperature CSF state after the quench in the deep-SF regime. On the other hand, the site-decoupling treatment could fail to capture some quantitative features in the vicinity of the CSF-MI transition, including the location of the critical point and the values of critical exponents, as the intersite quantum correlations become important.

Quantitative analysis by CMF + S with DMRG

We provide more quantitative estimation of the CSF-MI critical point for V = 0 beyond the site-decoupling approximations (GA and TDGA) to predict the interplay effect of frustration and quantum fluctuations as a guideline necessary for experiments. The quantum effects on frustrated Bose gases in 2D lattices have been poorly studied due to the lack of effective computational methods. Here we generalize and apply the numerical CMF + S method with 2D DMRG solver, which recently established in studies on frustrated quantum spins14, to the present system. We consider the NC-site cluster Hamiltonian on a triangular-shaped cluster (see the illustrations in Fig. 5a) under the mean-field boundary condition. We work in the twisted frame, \({\tilde{b}}_{i}\equiv {e}^{-i{\bf{Q}}\cdot {{\bf{r}}}_{i}}{\hat{b}}_{i}\), with optimizing Q = (Qx, 0) (see Methods). The CMF + S method permits the systematic inclusion of quantum intersite correlations by increasing NC, which connects between the GA (NC = 1) and exactly quantum (NC → ) results.

Fig. 5: Quantitative analysis on the quantum critical point.
figure 5

a The critical points Uc /J2 between the chiral superfluid (CSF) or superfluid (SF) state and the Mott insulator (MI) as functions of the hopping anisotropy J1 /J2. The values obtained by the cluster mean-field plus scaling (CMF + S) analysis are compared with those obtained by the Gutzwiller approach (GA). In the CMF + S results, the error bars estimated from the variation in the linear fittings for different pairs of the NC = 10, 15, 21 data are smaller than the symbol size. The bar on the vertical axis marks the quantum Monte-Carlo result on the square lattice66 as reference. b The relative difference between the GA and CMF + S results for the critical point. c The momentum of the Bose–Einstein condensates, Q = (Qx, 0), in units of πa−1 (with a being the lattice constant) at the critical interaction strength U = Uc.

Figure 5a summarizes the NC = 10, 15, 21 data and the CMF + S (NC → ) result for the ρ = 1 CSF-MI (SF-MI) critical point in the frustrated (unfrustrated) case with J1J2 < 0 (J1J2 > 0). The value of Uc/J2 for the frustrated case exhibits a nonmonotonic behavior with a dip around J1/J2 ≈ 0.8, whereas the unfrustrated one simply increases as the total hopping J1 + 2J2 increases. This is indeed the frustration effect, which destabilizes the CSF state. Besides, the value of Uc is strongly reduced from the GA prediction \({U}_{{\rm{c}}}^{({\rm{GA}})}\) due to the inclusion of the intersite quantum correlations. It should be noted that the relative difference \(({U}_{{\rm{c}}}^{({\rm{GA}})}-{U}_{{\rm{c}}})/{U}_{{\rm{c}}}^{({\rm{GA}})}\) between the GA and CMF + S values is much larger for the frustrated case (~40–50%) than the unfrustrated case (~20%) as shown in Fig. 5b. This indicates that the quantum effects are strongly enhanced by the interplay with frustration. Figure 5c shows that the BEC momentum Qx in the CSF state is little affected by the inclusion of quantum correlations. The slight variance of Qx from 4π/3 at J2/J1 = 1 is thought to be due to the finite cluster-size effect (see Methods).

Detection method

The BEC momentum Q in the CSF state and the transition to the MI can be simply detected by time-of-flight (TOF) images of momentum distributions46. A more precise determination of the critical point can be made by extracting the condensate fraction from the TOF images58, by observing the critical velocity using a moving optical lattice57, or by measuring the density fluctuation δn2 using the quantum-gas microscope18,19,20,21,22,23. The frustrated CSF-MI transition is easily distinguishable from the standard SF-MI transition thanks to the sharp difference in the value of Uc/J2.


We made a proposal and provided the necessary theoretical analysis for analog quantum simulation of frustrated quantum magnetism by using ultracold Bose gases in triangular optical lattices. We proposed an experimental protocol to create a frustrated quantum state at negative absolute temperature by performing a phase imprinting together with sudden inversion of the interatomic interaction and the trap potential. Simulating the time evolution, we demonstrated that a dynamically stable SF state with chiral symmetry breaking was indeed realized and underwent the quantum phase transition to the MI as the hopping amplitude decreased. Moreover, we performed state-of-the-art numerical calculations on the quantum critical point as a function of the spatial hopping anisotropy, which predicted a significant interplay of frustration and quantum fluctuations. The quantum dynamics in the presence of such enhanced fluctuations near the criticality is out of reach with the currently available numerical methods and thus gives a strong motivation for future experimental quantum simulations as an interesting and important subject to be investigated.

A connection of the present synthetic system to real materials of frustrated antiferromagnets can be obtained by using the approximate mapping from the bosonic to spin-1 operators59: \({\hat{b}}_{i}\to \sqrt{\frac{\rho }{2}}{\hat{S}}_{i}^{-},\,{\hat{n}}_{i}\to {\hat{S}}_{i}^{z}+\rho\), which is valid in the vicinity of the transition between the CSF and MI phases at integer fillings ρ. The Hamiltonian (1) is mapped onto the spin-1 XY model

$${\hat{{\mathcal{H}}}}_{{\rm{spin}}}=-\sum _{i,j}{J}_{ij}^{{\rm{XY}}}({\hat{S}}_{i}^{x}{\hat{S}}_{j}^{x}+{\hat{S}}_{i}^{y}{\hat{S}}_{j}^{y})+D\sum _{i}{({\hat{S}}_{i}^{z})}^{2}$$

with the XY spin exchange \({J}_{ij}^{{\rm{XY}}}=-\rho {J}_{ij}\) and the single-ion anisotropy D = U/2. Therefore, the physics discussed in the present study is deeply related to the pressure-induced phase transition from the “120” magnetic order (corresponding to CSF) to nonmagnetic state (MI) in spin-1 easy-plane triangular antiferromagnets such as CsFeCl360,61, which has recently attracted considerable attention in the connection with the novel excitations near quantum criticality62. From the standpoint of fundamental statistical physics, the CSF-to-MI transition should be associated with the spontaneous U(1)\(\times {{\mathbb{Z}}}_{2}\) symmetry breaking with respect to the global phase and the chirality determined by q = Q or  −Q, whose quantum critical phenomena and universality class have not yet been established. The present bosonic system of synthetic antiferromagnets is advantageous for exploring exotic quantum critical phenomena in low dimensions, whereas the real materials have strong 3D couplings between the triangular layers60,61.

Moreover, it has been expected that adding long-range interatomic interaction to the present system may give rise to an exotic chiral MI state63 in between possible separate U(1) and \({{\mathbb{Z}}}_{2}\) symmetry breakings. This is essentially equivalent to the so-called “chiral liquid” expected in a spin-1 frustrated magnet64. Besides, our protocol for direct creation of a frustrated quantum state is advantageous for preparing the phases that are not neighboring to the MI phase, such as quantum spin liquids expected for ρ = half integers3,11. Thus, the present study gives a crucial guidepost for cold-atom quantum simulations of those exotic quantum frustrated physics.


Triangular optical lattice with anisotropic hoppings

A triangular optical lattice can be created by superposing three laser beams that intersect in the xy plane with wave vectors k1 = kL(1, 0), k2 = \({k}_{{\rm{L}}}(-1/2,-\sqrt{3}/2)\), and k3 = \({k}_{{\rm{L}}}(-1/2,\sqrt{3}/2)\), and equal frequency ωL24. All beams are linearly polarized orthogonal to the plane and each has field strength Ei (i = 1, 2, 3). The total electric field is given by

$${\bf{E}}_{\rm{{tot}}}({\bf{r}},t)=\sum _{i=1}^{3}{E}_{i}\cos ({{\bf{k}}}_{i}\cdot {\bf{r}}-{\omega }_{{\rm{L}}}t+{\phi }_{i}){{\bf{e}}}_{z}.$$

The dipole potential generated by the electric field is proportional to its squared amplitude,

$$V({\bf{r}}) \propto \, {\left|{{\bf{E}}}_{{\rm{tot}}}({\bf{r}},t)\right|}^{2}\\ = \, \frac{{E}_{1}^{2}+{E}_{2}^{2}+{E}_{3}^{2}}{2}+{E}_{2}{E}_{3}\cos ({{\bf{b}}}_{2}\cdot {\bf{r}}-{\phi }_{23})\\ \hskip 10pt\,+{E}_{1}{E}_{3}\cos (({{\bf{b}}}_{1}-{{\bf{b}}}_{2})\cdot {\bf{r}}+{\phi }_{13})\\ \, \hskip 10pt+{E}_{1}{E}_{2}\cos ({{\bf{b}}}_{1}\cdot {\bf{r}}+{\phi }_{12})+A(t),$$

where b1 = k1 − k2, b2 = k3 − k2, ϕij = ϕi − ϕj, and A(t) represents the terms dependent on time t. As the frequency of light is quite large, only the time-averaged value of Etot2 can affect atoms. All the terms in A(t) oscillate at frequency 2ωL and thus can be dropped. Finally, we obtain a periodic dipole potential

$$V({\bf{r}})= \, -{V}_{1}\cos ({{\bf{b}}}_{2}\cdot {\bf{r}}-{\phi }_{23})\\ \, -{V}_{2}\cos (({{\bf{b}}}_{1}-{{\bf{b}}}_{2})\cdot {\bf{r}}+{\phi }_{13})\\ \, -{V}_{3}\cos ({{\bf{b}}}_{1}\cdot {\bf{r}}+{\phi }_{12})+\,{\rm{const.}},$$

where V1, V2, and V3 are proportional to E2E3, E1E3, and E1E2, respectively. A variation of the phases ϕi of the laser beams yield only a global shift of the lattice in position. The primitive lattice vectors a1 and a2 are given so that ai bj = 2πδij. We define a3 = − a1 − a2 for convenience. For red-detuned lasers with Vi > 0, the potential minima form a geometrically equilateral triangular lattice in lattice constant: a = ai = 4π/3kL = 2λL ∕ 3 with λL being the laser wavelength. The spatial anisotropy in the hopping amplitudes can be introduced by the difference in V1, V2, and V3 through tuning the laser intensities E1, E2, and E3. For example, the set of the field strengths with the relation of E1 = 1.6E2 = 1.6E3 (1.6V1 = V2 = V3) yields the potential landscape shown in Fig. 1a, which gives anisotropy of “isosceles type” in the nearest-neighbor hopping amplitudes.

Tilting triangular optical lattice with an additional 1D optical lattice

To perform the phase-imprinting process in the protocol proposed here, one has to introduce a single-particle energy difference δE between adjacent two sites. This could be directly implemented by a magnetic field gradient, which introduces an extra linear gradient potential. Here, let us also provide another way to perform the phase-imprinting process for preparing the commensurate Q = QM ≡ (2π/a, 0) (two-color) and Q = QK ≡ (4π/3a, 0) (three-color) states by the use of an additional 1D optical lattice. We suppose that a potential of 1D optical lattice is created with additional laser beams in the a1 direction:

$${V}_{1{\rm{D}}}(x)={V}^{\prime}{\cos }^{2}({k}_{{\rm{L}}}^{\prime}x+{\phi }^{\prime})$$

with amplitude \({V}^{\prime}\), wave vector \({k}_{{\rm{L}}}^{\prime}=2\pi /{\lambda }_{{\rm{L}}}^{\prime}\), and phase \({\phi }^{\prime}\). Here, let us set ϕi = 0 (i = 1, 2, 3) for the triangular optical lattice without loss of generality.

Let us first consider the range of 0 ≤ J1/J2 0.5 (V1V2 = V3), in which the configuration of the local phase factor is expected to form the two-sublattice (say A and B) structure with the pitch vector Q = QM as illustrated in the left panel of Fig. 2a. For creating the local phase configuration by the phase imprinting, it is required to introduce a temporary single-particle energy difference δE only between the two-sublattice groups of sites for time δt = 2π/δE. This can be achieved by using an additional 1D optical lattice of magic wavelength defined by \({\lambda }_{{\rm{L}}}^{\prime}=4{\lambda }_{{\rm{L}}}/3\) and phase shift \({\phi }^{\prime}=0\) or π/2 (mod π). Figure 6a, b show an example of the total potential V(r) + V1D(x) with the parameters \(2{V}_{1}={V}_{2}={V}_{3}={V}^{\prime}\) and \({\phi }^{\prime}=0\). It is noteworthy that the two options in \({\phi }^{\prime}\) correspond to the exchange of A and B.

Fig. 6: Periodically tilted triangular optical lattice.
figure 6

a Potential landscapes of triangular optical lattice without and with an additional 1D periodic potential (whose amplutude and phase are \({V}^{\prime}\) and \({\phi }^{\prime}\), respectively) for \(2{V}_{1}={V}_{2}={V}_{3}={V}^{\prime}\) and \({\phi }^{\prime}=0\). The amplutudes of the three standing waves forming the triangular optical lattice are denoted by V1, V2, and V3. The cuts along r = r(1, 0) and \({\bf{r}}=r(1/2,\sqrt{3}/2)\) are shown in b. The spatial coordinate r = (xy) and its norm r are measured in units of the lattice constant a. c, d Same as a and b, respectively, for \({V}_{1}={V}_{2}={V}_{3}={V}^{\prime}\) and \({\phi }^{\prime}=\pi /12\).

In a similar way, using a 1D periodic potential with \({\lambda }_{{\rm{L}}}^{\prime}=2{\lambda }_{{\rm{L}}}\), one can also implement a temporary energy difference δE between the three-sublattice groups of sites, say A, B, and C. The additional lasers are shined for time δt = 4π/3δE to imprint the three-color phase configuration illustrated in the middle panel of Fig. 2a to the initial SF state. Figure 6c, d show an example of the total potential V(r) + V1D(x) with the parameters \({V}_{1}={V}_{2}={V}_{3}={V}^{\prime}\) and \({\phi }^{\prime}=\pi /12\). It is noteworthy that the phase shift \({\phi }^{\prime}\) has six options, (2n − 1)π/12(n = 1, 2, , 6), reflecting the possible permutation of A, B, and C.

The GA analysis for finite-momentum BEC states

Within the site-decoupling mean-field approximation, known as the GA, the effective local Hamiltonian at site i is given by

$${\hat{{\mathcal{H}}}}_{i}^{{\rm{GA}}}= \, -\sum _{j\ne i}{J}_{ij}\left({\psi }_{j}{\hat{b}}_{i}^{\dagger }+{\psi }_{j}^{* }{\hat{b}}_{i}-{\psi }_{i}^{* }{\psi }_{j}\right)-\mu {\hat{n}}_{i}\\ \, +\frac{U}{2}{\hat{n}}_{i}({\hat{n}}_{i}-1)+\frac{V}{{a}^{2}}| {{\bf{r}}}_{i}{| }^{2}{\hat{n}}_{i},$$

as the result of the decoupling \({\hat{b}}_{i}^{\dagger }{\hat{b}}_{j}\approx {\psi }_{j}{\hat{b}}_{i}^{\dagger }+{\psi }_{i}^{* }{\hat{b}}_{j}-{\psi }_{i}^{* }{\psi }_{j}\) in the original Hamiltonian (1). The results displayed in Fig. 2b are calculated under the assumption of finite-momentum BEC, \({\psi }_{i}=\overline{\psi }{e}^{i{\bf{q}}\cdot {{\bf{r}}}_{i}+\varphi }\), for V = 0. In this case, one has only to consider a certain single site i, e.g., the site at the origin ri = 0, and φ = 0 without loss of generality. Equation (8) becomes

$${\hat{{\mathcal{H}}}}_{i}^{{\rm{GA}}}={\varepsilon }_{{\bf{q}}}\overline{\psi }\left({\hat{b}}_{i}^{\dagger }+{\hat{b}}_{i}-\overline{\psi }\right)-\mu {\hat{n}}_{i}+\frac{U}{2}{\hat{n}}_{i}({\hat{n}}_{i}-1),$$

for the origin site i. The minimization of the kinetic energy gives q = 0 for J1J2 > 0 and q = Q given in Eq. (2) for J1J2 < 0. The Hamiltonian \({\hat{{\mathcal{H}}}}_{i}^{{\rm{GA}}}\) can be easily diagonalized on the Fock state basis for the local wave function, \(\left|{\Psi }_{i}\right\rangle \equiv {\sum }_{n = 0}^{{n}_{\max }}{f}_{i}^{(n)}\left|n\right\rangle\), in which the maximum one-site occupation number \({n}_{\max }\) must be sufficiently large (we take \({n}_{\max }=10\)). The order parameter is obtained from \(\overline{\psi }={\sum }_{n}\sqrt{n}{f}_{i}^{(n-1)* }{f}_{i}^{(n)}\) with eigenvector \({f}_{i}^{(n)}\) in a self-consistent way.

It is worth noting that the effective one-body Hamiltonians \({\hat{{\mathcal{H}}}}_{i}^{{\rm{GA}}}\) in the unfrustrated and frustrated cases differ only in εq (ε0 or εQ). Therefore, if the values of all the other terms are multiplied by εQ/ε0, the results of the GA calculations, such as the value of ψi, for the unfrustrated system become exactly the same as those for the frustrated system, except for the chiral phase distribution \({e}^{i{\bf{Q}}\cdot {{\bf{r}}}_{i}}\) and the overall energy scale (which is also multiplied by εQ/ε0).

In the presence of the trap potential V ≠ 0, the mean field ψi can no longer be assumed to have spatially uniform amplitude. Therefore, one has to deal with Eq. (8) on the entire lattice sites, each of which is connected to the six neighboring sites through the mean fields \(\{{\psi }_{i\pm {{\bf{a}}}_{n}};n=1,2,3\}\). To prepare the initial state shown in Fig. 3a, we solve the set of self-consistent equations \({\psi }_{i}={\sum }_{n}\sqrt{n}{f}_{i}^{(n-1)* }{f}_{i}^{(n)}\) for all sites within a cutoff length lc from the trap center. We take lc = 36a. The chemical potential μ is determined from the global number equation \({\sum }_{i}\langle {\hat{n}}_{i}\rangle =N\), which must be solved simultaneously with the equations for ψi. To efficiently achieve the convergence in the self-consistent calculations, we take the uniform solution for ψi obtained in the absence of the trap potential as the initial condition and employ the Newton–Raphson method.

The TDGA simulation in a trap potential

The TDGA equation is given by

$$i\frac{\partial }{\partial t}\left|{\Psi }_{i}(t)\right\rangle ={\hat{{\mathcal{H}}}}_{i}^{{\rm{GA}}}\left|{\Psi }_{i}(t)\right\rangle .$$

From Eq. (8) with \(\left|{\Psi }_{i}(t)\right\rangle ={\sum }_{n}{f}_{i}^{(n)}(t)\left|n\right\rangle\), Eq. (10) becomes

$$i{\dot{f}}_{i}^{(n)}(t)= -\sum _{j\ne i}{J}_{ij}\left(\sqrt{n}{\psi }_{j}(t){f}_{i}^{(n-1)}(t)\right.\\ \, \left.+\,\sqrt{n+1}{\psi }_{j}^{* }(t){f}_{i}^{(n+1)}(t)-{\psi }_{i}^{* }(t){\psi }_{j}(t){f}_{i}^{(n)}(t)\right)\\ +\,\left(\frac{U}{2}(n-1)+\frac{V}{{a}^{2}}| {{\bf{r}}}_{i}{| }^{2}-\mu \right)n{f}_{i}^{(n)}(t)$$

with \({\psi }_{j}(t)=\langle {\Psi }_{j}(t)| {\hat{b}}_{j}| {\Psi }_{j}(t)\rangle ={\sum }_{n}\sqrt{n}{f}_{j}^{(n-1)* }{f}_{j}^{(n)}(t)\).

We numerically solve the set of equations of motion for \({f}_{i}^{(n)}(t)\) using the Crank–Nicolson method65. We first rewrite Eq. (11) in a matrix form:

$$i{\dot{{\bf{f}}}}_{i}(t)={{\bf{H}}}_{i}^{{\rm{GA}}}(\{{\psi }_{j}(t)\}){{\bf{f}}}_{i}(t),$$

where fi is the vector with the components \({f}_{i}^{(n)}\) (\(0\ \le \ n\ \le \ {n}_{\max }\)) and \({{\bf{H}}}_{i}^{{\rm{GA}}}\) is the \(({n}_{\max }+1)\times ({n}_{\max }+1)\) matrix form of the GA local Hamiltonian, which is time-dependent via the order parameters ψj(t) of neighboring sites of site i. The explicit components of \({{\bf{H}}}_{i}^{{\rm{GA}}}\) are easily obtained from Eq. (11). The value of fi after time evolution in a short time Δt is given by

$${{\bf{f}}}_{i}(t+\Delta t)=\frac{{\bf{I}}-i\Delta t{{\bf{H}}}_{i}^{{\rm{GA}}}(\{{\psi }_{j}(t)\})/2}{{\bf{I}}+i\Delta t{{\bf{H}}}_{i}^{{\rm{GA}}}(\{{\psi }_{j}(t)\})/2}{{\bf{f}}}_{i}(t),$$

where I is the identity matrix. Here, Δt must be sufficiently shorter than the time scale of the physics considered. The time evolution of the system is numerically calculated, step by step, with short time interval Δt. To calculate fi(t + Δt) at site i, the values of ψj(t) on its neighboring sites are required. Therefore, one has to calculate \({\psi }_{j}(t)={\sum }_{n}\sqrt{n}{f}_{j}^{(n-1)* }{f}_{j}^{(n)}(t)\) and update fi(t) according to Eq. (13) in parallel for all sites (within ri ≤ lc). The local particle number at time t can be calculated by \({n}_{i}(t)={\sum }_{n}n| {f}_{i}^{(n)}(t){| }^{2}\).

The phase-imprinting operation

In the section “TDGA simulation”, we assume that the phase imprinting can be theoretically implemented by the operation \({\sum }_{n}{f}_{i}^{(n)}\left|n\right\rangle \to {\sum }_{n}{e}^{in{{\bf{Q}}}_{{\rm{K}}}\cdot {{\bf{r}}}_{i}}{f}_{i}^{(n)}\left|n\right\rangle\) on the local wave functions. Here we provide a brief discussion on the experimental conditions for achieving such perfect phase imprinting.

Let us consider the same initial state as shown in Fig. 3a (with the parameters N =  1400, J1 = J2 = J = 0.08U0, U = U0, and V = 0.001U0) and its time evolution within the TDGA method in the presence of a temporary linear gradient potential \({\hat{V}}^{{\rm{ext}}}=\delta E{\sum }_{i}({x}_{i}/a){\hat{n}}_{i}\). As seen in Fig. 7a, the chiral structure in the local phase, \({\rm{Arg}}[{\psi }_{i}]={\bf{Q}}\cdot {{\bf{r}}}_{i}\), could be successfully imprinted by applying the temporary potential for δt = Qxa/δE (=4π/3δE for J1 = J2). Figure 7b indicates that the phase imprinting becomes almost perfect with no changes other than the local phase distribution when δE exceeds  ~ 10U0. The corresponding imprinting time δt 0.4U0−1 is much shorter than the typical time scale of the experiments on the SF(CSF)-MI transition (see Fig. 4).

Fig. 7: Phase imprinting.
figure 7

a Time evolution of the phase difference between the neighboring sites in the a1 direction, \(\delta \varphi \equiv \overline{({\rm{Arg}}[{\psi }_{i}]-{\rm{Arg}}[{\psi }_{i+{{\bf{a}}}_{1}}])}\), averaged over the center sites within ri ≤ 10a, in the presence of a temporary linear gradient potential. The colors correspond to different potential strengths δE/U0  = 1, 2, 5, 10, and 20, although all the curves are almost overlapped. b The density (solid lines) and order-parameter (dashed lines) profiles in the final state after applying a gradient potential for time δt.

The CMF + S analysis with 2D DMRG solver for bosons

In the CMF + S analysis, we consider the NC-site cluster Hamiltonian

$${\tilde{{\mathcal{H}}}}_{{\rm{C}}}= \, -\sum _{i,j\in {\rm{C}}}{J}_{ij}{e}^{i{\bf{q}}\cdot ({{\bf{r}}}_{j}-{{\bf{r}}}_{i})}{\tilde{b}}_{i}^{\dagger }{\tilde{b}}_{j}+\frac{U}{2}\sum _{i\in {\rm{C}}}{\hat{n}}_{i}({\hat{n}}_{i}-1)\\ \, -\overline{\psi }\sum _{i\in \partial {\rm{C}}}\left(\sum _{j\notin {\rm{C}}}{J}_{ij}{e}^{i{\bf{q}}\cdot ({{\bf{r}}}_{j}-{{\bf{r}}}_{i})}{\tilde{b}}_{i}^{\dagger }+{\rm{H}}.{\rm{C}}.\right)$$

on a triangular-shaped cluster of NC = 10, 15, 21 sites. The mean-field boundary condition on the cluster-edge sites ∂C is implemented by the third term and the twisted frame, \({\tilde{b}}_{i}\equiv {e}^{-i{\bf{q}}\cdot {{\bf{r}}}_{i}}{\hat{b}}_{i}\), is adopted. The cluster Hamiltonian (14) is treated with 2D DMRG solver. Here we take the maximum one-site occupation \({n}_{\max }=4\), which is confirmed to be sufficient for the discussion near the ρ = 1 SF-MI (CSF-MI) transition. For large-size clusters and especially for the large Hilbert space of bosons, the exact diagonalization is practically not realistic as a solver for the cluster problem. Therefore, we employ the DMRG on the equivalent 1D chain model with long-range hoppings and mean fields (see Fig. 8a). The DMRG calculation is performed in the standard way but with the mean-field terms in Eq. (14)14. The dimension of the truncated matrix product states kept in the present DMRG calculations is typically  ~103 to obtain numerically precise results. To solve

$$\overline{\psi }=\frac{1}{{N}_{{\rm{C}}}}\sum _{i\in {\rm{C}}}{\langle {\tilde{b}}_{i}\rangle }_{{\tilde{{\mathcal{H}}}}_{{\rm{C}}}(\overline{\psi })}$$

in a self-consistent way, we iteratively perform the DMRG calculations until convergence.

Fig. 8: Cluster mean-field calculations with 2D density matrix renormalization group.
figure 8

a Mapping of the 2D cluster problem with the mean-field boundary onto an equivalent 1D chain with long-range interactions and mean fields. b Typical behavior of the provisional critical point \({U}_{{\rm{c}}}^{* }/| {J}_{2}|\) as a function of the given momentum q = (qx, 0) in units of πa−1 for the hopping anisotropy J1J2 = 0.7 and the cluster size NC = 21.

It is noteworthy that when we put a real number \(\overline{\psi }\) as an input for \({\tilde{{\mathcal{H}}}}_{{\rm{C}}}(\overline{\psi })\) in the fixed global gauge, the output \({N}_{{\rm{C}}}^{-1}{\sum }_{i\in {\rm{C}}}{\langle {\tilde{b}}_{i}\rangle }_{{\tilde{{\mathcal{H}}}}_{{\rm{C}}}(\overline{\psi })}\) includes a small but finite imaginary component (4% for NC = 21). This is due to a finite-size effect; the order with uniform amplitude \(\overline{\psi }\) and spiral phase twist \(\exp [i{\bf{q}}\cdot {{\bf{r}}}_{i}]\) is not fully commensurate with the shape of the finite-size clusters with the mean-field boundary. We just ignore the small imaginary component in the calculations for each NC, as it decreases with NC and is expected to vanish at the limit of NC → .

The optimization of the spiral twist q is performed in the following way: For different values of q = (qx, 0), the “provisional” critical point \({U}_{{\rm{c}}}^{* }/| {J}_{2}|\) (at which \(\overline{\psi }={0}^{+}\)) is numerically determined (Fig. 8b). The maximum value of \({U}_{{\rm{c}}}^{* }({q}_{x})/| {J}_{2}|\) and the corresponding (qx, 0) were adopted as the CMF + S prediction of the critical point Uc/J2 and the BEC momentum Q at the critical point, respectively. The slight variance of Qx from 4π/3 at J1/J2 = 1 (see Fig. 5c) is thought to stem from the same finite cluster-size effect mentioned above.

The CMF + S curves in Fig. 5a are obtained from the size scaling of the phase boundaries for NC = 10, 15, 21. Figure 9 shows the extrapolation of the NC = 10, 15, 21 data to NC →  (λ → 1) for several values of J1/J2 with a linear function of the scaling parameter λ ≡ NB ∕ 3NC36,37. Here, NB is the number of NN bonds treated exactly in the cluster (NB = 18, 30, 45 for NC = 10, 15, 21, respectively). The error bars in Fig. 5a are estimated from the variation in the linear fittings for different pairs of the NC = 10, 15, 21 data.

Fig. 9: Scaling analysis in the cluster mean-field plus scaling analysis.
figure 9

Cluster-size scalings of the critical points for the transitions (a) between the superfluid and Mott insulator states in the unfrustrated (J1J2 > 0) case and b between the chiral superfluid and Mott insulator states in the frustrated (J1J2 < 0) case. The extrapolated (λ → 1) values are plotted as a function of the hopping anisotropy J1/J2 in Fig. 5a.