Introduction

Experiments of cold atoms in optical lattices allow addressing fundamental questions of condensed matter physics1,2. From a condensed matter perspective, those experiments accomplish two major achievements. First, they can serve as quantum simulators of systems that have a solid-state analogue with the crucial difference that one has a high degree of control and excellent tunability of most of the system parameters3. One may even go beyond traditional models and study parameter regimes that are not accessible in the solid state. On the other hand, cold-atom setups provide platforms for the realization of novel phases of matter and new phenomena that have never been observed in the solid state so far. Examples of the latter is the interaction driven Mott-superfluid quantum phase transition in the Bose-Hubbard model4 or SU(N) magnetism using alkaline-earth atoms5. Examples of the first case are investigations of Fermi-Hubbard models6,7,8,9,10 in search for d-wave superfluidity or the study of three-dimensional Rashba spin-orbit coupling11.

One of the most active areas of research in condensed matter physics at present are topological phases and, in particular, topological insulators12,13,14. These phases represent a new paradigm of quantum matter with a bulk gap but gapless surface states that are protected by symmetries such as time-reversal invariance. In real materials, the non-trivial band topology is usually caused by virtue of spin-orbit coupling. The realization of synthetic spin-orbit couplings for cold atoms15,16,17,18,19,20,21,22,23,24,25 thus marks the advent of topology to the field of ultra-cold gases.

The dimensionality of space is an important parameter of topological phases in their classification according to anti-unitary symmetries26. The crossover between different dimensions is in general non-trivial: simply stacking the two-dimensional (2D) topological insulator27,28,29,30 does not lead to its 3D counterpart, the strong topological insulator (STI)31,32,33. Instead, one obtains a 3D weak topological insulator (WTI)34.

The situation becomes even more interesting in the presence of sufficiently strong interactions, where the topological classification26 breaks down in some cases (see, e.g., Ref. 35) and new phases can appear. As the interaction strength in cold-atom systems can be tuned over a wide range, they provide an ideal testbed to study the interplay of non-trivial topology and strong interactions. One of the most exciting theoretical proposals for such exotic states of matter is the topological Mott insulator of Pesin and Balents36. It represents a spin liquid-like phase where the charge is stripped from the original electrons and frozen in a Mott insulating phase; the spinons (i.e., the electrons' spin degree of freedom), however, inherit the non-trivial band topology of the topological insulator and provide spin-only gapless surface states. The experimental observation of this state has so far remained elusive and even the theoretical understanding is rather limited36,37,38,39,40. Note that this notion of a topological Mott insulator, which will be used in this paper, crucially differs from that of the proposal in Ref. 41 (see Supplementary Discussion S1).

Here, we propose an experimental setup which represents an example of both achievements of cold atoms in optical lattices: first, a quantum simulator for known phases which can be tuned into each other in a unique way. Second, a realization of exotic weak and strong topological Mott insulating phases which have not yet been found in solids.

The model we propose is experimentally feasible and can be seen as a 3D generalization of an experimental setup for 2D topological insulators in optical lattices42,43,44,45,46. We show that the 2D system, which effectively consists of two time-reversed copies of massive Dirac Hamiltonians, can be tuned to a 3D weak or strong topological insulator just by varying a single hopping parameter. Since on-site Hubbard interactions are experimentally available, we analyze the interacting phase diagram within slave-rotor theory47,48,49. Interactions drive a transition into a Mott insulating state, where the topological nature of the system survives and emerges as fractionalized gapless spin-only surface modes. Our proposal for a cold atom experiment and its possible subsequent realization will help to establish a deeper understanding of this exotic phase.

Results

Dimensional crossover from 2D to 3D topological insulator

The starting point of our analysis is the time-reversal invariant Hofstadter model on the square lattice42,43,44

It describes fermions with spin-1/2 in the presence of synthetic gauge fields; the parameter α describes the flux per plaquette of an artificial magnetic field perpendicular to the 2D plane, which due to the Pauli matrix σz points in opposite directions for opposite spins. The spin mixing term γ induces spin flips if the particle moves along the x-axis and the λ-term describes a staggering of the optical lattice potential along the x-direction. All three terms can be experimentally realized42,43,44.

The model belongs to the symmetry class AII26 and its phase diagram in 2D hosts quantum spin Hall (QSH) and normal insulating (NI) as well as (semi-)metallic phases42,45,46. The QSH phase is characterized by an odd number of helical edge states per edge, while the NI features an even number (including zero). The topological invariant ν distinguishes between QSH (ν = 1) and NI (ν = 0) phase28. It is sufficient to consider half-filling and fixed values of α = 1/6 and γ = 1/4, as the phase diagram as a function of the staggered potential λ already contains both gapped phases as well as semi-metallic points. As shown in Fig. 1(a) the 2D system is a QSH insulator for |λ| < λc = 21/3t and becomes normal insulating for |λ| > λc42,45,46. The system is semi-metallic at λ = 0 hosting two doubly degenerate Dirac cones and at λ = λc with one doubly degenerate Dirac cone.

Figure 1
figure 1

Phase diagram and spectra of the lattice model.

(a) 2D–3D crossover phase diagram as a function of layer coupling tz and staggered lattice potential λ. (b) Bulk gap as a function of tz and λ. (c) Surface state spectrum (blue) of the isotropic 3D system tz = t, λ = λc at the x = 0 surface as a function of momenta ky and kz. Bulk states are shown as gray dots. We use open (periodic) boundary conditions along x (y, z). (d–e) One-dimensional cuts of the spectrum for fixed values of kz = 0 and kz = π. Gapless edge states at x = 0 (x = L) edge are shown in blue (red) yielding z0 = 1 and zπ = 0 and thus ν0 = z0 + zπ = 1.

Full size image

We are interested in studying the dimensional crossover from two to three dimensions by continuously turning on a hopping parameter tz in the third direction that couples the different 2D layers. The easiest interlayer coupling term that makes STI phases possible is of the form

It contains a synthetic gauge field that represents an artificial magnetic field along the y-direction, which points in opposite directions for spins aligned parallel and anti-parallel to the y-axis. Such a term can be most easily implemented in an all optical realization of the lattice potential43,44. Note that this term respects time-reversal symmetry and, hence, the resulting 3D Hamiltonian H2D + Hz still belongs to class AII.

As we have to deal with a 12-band model in 3D, some intuitive understanding is desirable. For this reason, we derive an effective theory valid in the vicinity of the point (tz, λ) = (0, λc) in the phase diagram of Fig. 1(a), where all the distinct phases meet: QSH and NI (when tz = 0) as well as STI, WTI and NI (when tz > 0). The doubly degenerate Dirac cone at this multi-critical point is formed out of four bands, the other eight bands are well-separated from the Fermi level.

Neglecting corrections quadratic in tz/t, (λ − λc)/t and kx, ky, quasi-degenerate perturbation theory50 yields the effective four-band Hamiltonian

where h(k) denotes the full 12-component Bloch-Hamiltonian associated with the terms in Eqs. (1) and (2), are the four zero-energy eigenfunctions of h(0) at the multi-critical point. Retaining only the relevant terms linear in kx and ky, Eq. (3) assumes the form

where the upper left 2 × 2-block is given by

with mass and δλ = λ − λc. The Pauli matrices τi act within the 2 × 2-blocks in Eq. (4) and a, b, c, d are positive constants. For convenience, we have chosen the basis functions such that time-reversal is given by , where the Pauli matrices si act between the different 2 × 2-blocks and denotes complex conjugation. The relation between the lower and the upper diagonal blocks in Eq. (4) is thus a consequence of time-reversal symmetry. In addition, this convention directly reveals the connection of the 2D system (tz = 0) to the Bernevig-Hughes-Zhang-model29, the paradigmatic model of a QSH state. Consequently, we immediately know that δλ = 0 marks the boundary between QSH and NI phases. As the regime δλ > 0 is adiabatically connected to the topologically trivial limit λ → ∞, the effective model reproduces the 2D phase diagram.

To continue with the 3D system, let us first emphasize that the effective Hamiltonian (4) is valid for the entire range −π < kz ≤ π, since tz (and not kz) has been taken as expansion parameter. In 3D, insulating phases are characterized by four invariants νi, with i = 0, 1, 2, 3, defined by invariants of 2D cuts of the 3D Brillouin zone31,32,33. To determine the strong invariant ν0 ≡ (z0 + zπ) mod 2, we need to calculate the invariants z0 and zπ associated with the time-reversal invariant planes kz = 0 and kz = π. As Eq. (4) again assumes the form of the Bernevig-Hughes-Zhang-model for fixed kz = 0 and kz = π, we can directly conclude that the two QSH invariants z0 and zπ differ when m(0)m(π) < 0. Therefore, the system is in the STI phase if and only if

Note that, despite the fact that the mass m(kz) has to vanish somewhere between kz = 0 and kz = π, the Hamiltonian is still fully gapped. This is a consequence of the block off-diagonal terms in Eq. (4) that are finite for kz ≠ 0, π. Interestingly, for λ = λc, this model allows for a transition into an STI, a phase that has no 2D analog, already for infinitesimal coupling tz.

Being adiabatically connected to tz = 0, the other two phases (|δλ| > 2.85 tz) can easily be identified from the knowledge about the two-dimensional system. For δλ > 2.85 tz, we find a NI, whereas, in case of δλ < −2.85 tz, the system resides in a WTI phase characterized by (ν0; ν1, ν2, ν3) = (0; 0, 0, 1)31,32,33.

Finally, the effective Hamiltonian also allows understanding why a spin- and position-independent hopping term along the z-direction cannot result in an STI phase. Such a term would simply lead to a contribution proportional to the identity matrix in the 12-band Bloch-Hamiltonian h(k) and thus to a term in the effective low-energy theory (3). Consequently, g in Eq. (5) would be independent of kz and hence z0 = zπ, excluding the appearance of an STI.

We verified our analysis of the effective model by numerically computing the invariant in all insulating phases of the full 12-band model using the approach of Ref. 51. The corresponding phase diagram is illustrated in Fig. 1(a) and is in perfect agreement with our previous discussion. The 2D gapped QSH (NI) phase turns into a WTI (NI), but most importantly the STI phase emerges from the 2D quantum critical point. Note that this is the only possibility for an STI to appear at infinitesimal coupling tz as this phase is not adiabatically connected to the NI and QSH.

An alternative way of realizing a dimensional crossover in topological systems consists of varying the thickness in the z-direction. As discussed in Refs. 52 and 53 in the solid state context, this leads to oscillatory properties of the effective 2D system. In contrast, in our crossover scheme, the system is fully 3D at every finite tz which is more feasible in a cold atom experiment. For that reason we calculate 3D topological invariants for any tz ≠ 0, which do not show oscillatory behavior.

The bulk gap, which closes at the phase transitions, is shown in Fig. 1(b). It reaches its maximal value of the order of t for isotropic hopping. Computing the spectrum with open (periodic) boundary conditions along the x- (y, z)-direction, we clearly observe in Fig. 1(c) (shown for tz = t) a single gapless Dirac-like surface state (in blue) localized at the x = 0 surface, which crosses the bulk gap (bulk bands in gray). The Dirac point is located inside the bulk bands (which is not unusual). The surface Fermi circle encloses an odd number of time-reversal invariant points reflecting the non-trivial strong invariant ν0 = 1. One-dimensional cuts of the spectrum for kz = 0 and kz = π are shown in Fig. 1(d) and (e), respectively. We see explicitly that m(0) and m(π) have opposite signs as the parity of the number of Kramers partners of surface states on a given boundary changes from odd (QSH) to even (NI) when kz is tuned from 0 to π.

Interaction effects and topological Mott phases

While non-interacting topological phases are characterized by the properties of their Bloch functions and can be classified according to symmetries, the question what occurs in the presence of interactions is of great current interest54,55,56,57. However, there are still many open questions concerning the fundamental classification of interacting states of matter and the explicit investigation of concrete interacting topological systems. So far, in explicit calculations, mainly the effect of Hubbard on-site interactions , where , has been considered in order to address interaction effects in topological band structures36,37,58,59. While this is very natural for solid state systems, the achievement of ultracold quantum gases and optical lattices allows considering interactions which cannot be realized in real materials, opening a path towards exotic physics and giving rise to an even richer phenomenology. As illustrated in Fig. 2(a), the key idea for effectively realizing an exotic interaction term is to encode the spin degree of freedom σ = ↑, ↓ spatially and use the internal atomic hyperfine states to represent the even/odd site information along x: μ = + for jx = 2n and μ = − for jx = 2n + 1 with integer n. We denote the fermionic operators of the new lattice by .

Figure 2
figure 2

Twisting the Hubbard interaction.

Part (a) shows the relabeling of the spin σ and spatial even/odd site μ degrees of freedom. To obtain an identical Hamiltonian H2D + Hz one must realize different hopping elements between nearest and next-nearest neighbor sites as illustrated in (b) for the x-direction. The required hoppings along the y- and z-direction are described in detail in Supplementary Discussion S2.

Full size image

As one can readily see in Fig. 2(a), a local Hubbard interaction realized on the new lattice, , where , reads, at half-filling, in terms of the occupation numbers of the original cjσ fermions as

Consequently, does not couple ↑-spin and ↓-spin on a given site but instead pairs of neighboring sites having the same spin orientation; the Hubbard term is twisted. As shown below, it will generate topological Mott insulating (TMI) phases36 if we keep the same non-interacting Hamiltonian H2D + Hz as before. To achieve this, one needs to experimentally implement different laser induced hopping elements for the fermions as illustrated in Fig. 2(b) for the x-direction. Only nearest neighbor and next-nearest neighbor terms along the three spatial directions are required.

The TMI is characterized by a fractionalization of the original atoms into an internal and a number degree of freedom. In the TMI phase the atoms are localized, but their internal degree of freedom remains deconfined and inherits the non-trivial band topology of the original fermions. This state has been proposed to occur in Ir-based pyrochlore materials36, but has so far never been experimentally observed.

We approach the TMI using slave-rotor theory47,48,49 which starts by writing the fermion operator as a product of number and internal degree of freedom . Here, θj denote phases conjugate to the total particle number (of djμ fermions) on site j and fjμ is a spinon fermion operator that carries the internal index. The system is then described by two coupled mean-field Hamiltonians: a 3D quantum XY rotor model, which captures the number degrees of freedom and a renormalized non-interacting spinon Hamiltonian (see Methods). As the strength of the quantum fluctuations in the rotor model is determined by the interaction U, the rotor undergoes a transition from a ferro- to a paramagnetic state as U is increased beyond a critical value Uc. This transition corresponds to the Mott transition. The spinons, on the other hand, are characterized by a bandstructure with renormalized parameters (set by the correlations between rotors), that can be topologically non-trivial and carry gapless surface excitations.

Analyzing the interaction term (7) within slave-rotor mean-field theory, we find that the interaction only renormalizes the hopping elements t, tz but leaves λ unchanged. Physically, this is due to the fact that is not sensitive to an imbalance between μ = + and μ = −. Furthermore, the interaction induces slight spatial anisotropies between the hopping elements as well as inhomogeneities in the six atomic unit cell. In Fig. 3(a) we show the interacting phase diagram as a function of U/t and tz/t in the region of the dimensional crossover for λ/t = 0.25. The hoppings of the djμ fermions on the new lattice have been chosen such that their Bloch Hamiltonian is unitarily equivalent to the Bloch Hamiltonian of the cjσ fermions and, consequently, the U = 0 line reproduces a cut of the phase diagram in Fig. 1(a). The non-interacting phases remain stable for small interaction , but the anisotropies lead to the emergence of a correlated semi-metallic (SM) phase at intermediate , where interactions induce a formation of Dirac cones.

Figure 3
figure 3

Properties of the interacting system according to slave-rotor theory.

(a) Interacting phase diagram as a function of interaction U and hopping tz for fixed λ/t = 0.25. Upon increasing U the phases found at U = 0 remain mostly intact with renormalized parameters. For an extended correlated semi-metallic (SM) phase appears at . At a critical value Uc/t (green line) the system enters a Mott insulating (MI) state with weak and strong topological MI (WTMI and STMI) as well as gapless MI (GMI) phases. The GMI phase exhibits a semimetallic spinon spectrum. Dashed lines are a guide to the eye showing that Mott phases persist for U > Uc. (b) Interacting phase diagram and (c) bulk gap at Uc as a function of staggering λ and hopping tz. The STMI phase occupies a large part of the phase diagram and features a significant bulk gap. (d–e) One-dimensional cut through the spinon bandstructure in the STMI phase for the isotropic system tz = t and λ/t = 0.15. We use open (periodic) boundary conditions along x (y, z). Bulk states are shown in yellow, gapless spinon edge states at the x = 0 (x = L) edge are shown in blue (red).

Full size image

Stronger interactions drive the system across a quantum phase transition to various Mott phases at a critical interaction strength Uc. In the Mott state, the fermionic degrees of freedom are fractionalized with charge (or number) and internal hyperfine degrees of freedom being split. While the number degrees of freedom are localized, the internal hyperfine degrees of freedom are deconfined. The resulting spinon bandstructure can remain topological across the Mott transition, which defines the sought after strong and weak topological Mott insulator (STMI and WTMI) phases. The dashed lines in Fig. 3(a) are only a guide for the eye indicating that the different Mott phases persist for . At even larger interaction strength magnetically ordered phases are likely to emerge (not shown). Note that the various complex hopping terms lead to a rather frustrated spin exchange and we expect the onset of magnetism for .

In Fig. 3(b) we present the interacting phase diagram at the critical interaction strength Uc as a function of λ/t and tz/t to show that WTMI and STMI occupy a large part of it. These phases exhibit a bulk gap of the order of 10% of the renormalized bandwidth (see Fig. 3(c)). Due to the topological nature of the spinon bandstructure, they feature gapless spinon surface states shown in Fig. 3(d–e) which is the defining property of the TMI phases.

Discussion

In our proposal, time-reversal is a non-local operation interchanging sites with even and odd jx in the new basis of djμ fermions defining the experimental lattice. Clearly, this is a consequence of the fact that the transformation depicted in Fig. 2 is non-local. However, on the scale of the system size, time-reversal and the mapping to the new lattice can be regarded as quasi-local operations and, in particular, do not interchange surface states at distinct surfaces. Thus any quasi-local perturbation of the fermions is also quasi-local in the basis of cjσ and the surface states in the new lattice localized on a given boundary form a time-reversal symmetric electronic system. This proves the protection12 of the surface states against localization by time-reversal symmetric disorder.

Let us now discuss how the TMI phases are reflected in observable quantities. The atoms are frozen in a Mott insulating state which can be detected via standard time-of-flight measurements. The most straightforward way to detect the spinon surface states is to measure the spin-dependent spectral function. Alternatively, one might consider transport or thermodynamic quantities60,61. In contrast to the non-interacting topological insulators, the TMIs do not exhibit a finite “charge” (or number) response at low frequencies due to the Mott gap. The gapless spinon surface states, however, carry entropy resulting in a thermal conductivity that is linear in temperature36 and a heat capacity that scales as T ln(1/T)38. Distinguishing between WTMI and STMI is possible as they feature a different number of surface states. The specific heat and thermal conductivity will thus be quantitatively different in the two phases.

To conclude, we have presented an experimentally feasible tight-binding Hamiltonian which allows performing a quantum simulation of a dimensional crossover from a 2D quantum spin Hall to 3D strong and weak topological insulator phases. Considering the effect of on-site interactions, we have shown that weak and strong topological Mott insulator phases can be realized. We are confident that our results will lead to a subsequent experimental realization of this exotic state of matter.

Methods

To derive the interacting phase diagrams reported in the main part of the paper, we apply slave-rotor theory47,48,49 to the Hamiltonian restated in the new basis (djμ fermions) where the interaction is an on-site Hubbard term. Within this approach, one introduces phases θj conjugate to the total number of fermions on one site of the lattice and auxiliary fermionic operators fjμ via

To restrict the theory to the physical part of the Hilbert space, we have to impose the constraint

This condition will be treated on average and accounted for by introducing Lagrange multipliers hj. Passing to a path-integral description, the associated action is given by

where fjμ, denote Grassmann variables and represent the hopping matrix elements in the new lattice.

In this work, we use the sigma-model description of Ref. 47, where the phase degrees of freedom are represented by complex bosonic fields

with the nonlinear constraint

The latter will be ensured on average via additonal Lagrange multipliers ρj. Applying a mean-field approximation to the hopping terms in Eq. (10), we arrive at the bosonic and fermionic actions

and

that are coupled via the self-consistency equations

Here we have already taken into account that, at half-filling, the constraint (9) is satisfied on average by choosing hj = 0. Note that only three of the Lagrange multipliers ρj, e.g. ρ0,0,0, ρ2,0,0 and ρ4,0,0, are independent which is due to the combination of translation symmetry and time-reversal invariance.

For any given U one has to solve the set of equations (15, 16) and find the correct Lagrange multipliers ρ0,0,0, ρ2,0,0, ρ4,0,0 yielding the renormalization factors Z(j, j′) of the auxiliary fermions fjμ. The band structure resulting from Eq. (14) can again be classified exactly as for non-interacting fermions leading to the variety of different correlated topologically trivial and non-trivial phases discussed in the main part of the paper. The transition into the Mott phase, where the local number degree of freedom is frozen out, occurs when the gap of the bosons in Eq. (13) closes. This point marks the transition from a ferromagnetic to a paramagnetic state of the rotors Xj. Even in the paramagnetic state there exist short range correlations between the rotors which implies that the renormalization factors Z(j, j′) are non-zero. A topologically non-trivial spinon bandstructure while having paramagnetic rotors defines the weak and strong topological Mott insulator phases.

To obtain the phase diagram in Fig. 3(b) in the main text, we have calculated the renormalization factors (16) right at the Mott transition (U = Uc) and determined the topological invariant of the auxiliary fermions fjμ using the numerical approach of Ref. 51. The phase boundaries for 0 < U < Uc, shown in Fig. 3(a) in the main text, have been obtained via linear interpolation of the renormalization factors,