Magnetic Chern Insulators in a monolayer of Transition Metal Trichalcogenides

A monolayer of transition metal trichalcogenides has received a lot of attention as potential two dimensional magnetic materials. The system has a honeycomb structure of transition metal ions, where both spin-orbit coupling and electron correlation effect play an important role. Here, motivated by these transition metal series with effective doping or mixed valence case, we propose the possible realization of magnetic Chern insulators at quarter filled honeycomb lattice. We show that the interplay of intrinsic spin-orbit coupling and electron correlation opens a wide region of ferromagnetic Chern insulating phases in between metals and normal insulators. Within the mean field approximation, we present the phase diagram of a quarter filled Kane-Mele Hubbard model and also discuss the effects of Rashba spin-orbit coupling and nearest neighbor interactions on it.

Topologically distinct phases introduced by Haldane showed that the quantum Hall phenomena could occur purely from the band structure in the absence of any external magnetic field, as a realization of the parity anomaly in (2 + 1) dimensional relativistic field theory 38 . By introducing the staggered flux on a honeycomb lattice, the system becomes an insulator with a non-zero topological invariant, termed as a Chern insulator. Experimental realization of Haldane model has been recently proposed in ultra cold atom system 39 , yet none of them has been reported in any solid state systems.
Here, we propose possible realization of Chern insulators in two dimensional van der Waals materials, especially in transition metal trichalcogenides. The van der Waals materials are characterized with layered crystals where individual layers are weakly coupled via van der Waals forces but with strong covalent bonding in the layer. Thus it is possible to peel away a single layer breaking the van der Waals bonds. One of the most well known examples is a single layer of graphene, peeled away from bulk graphite using scotch tape [40][41][42] . Using scotch tape technique, a variety of van der Waals materials have been successfully exfoliated into atomically thin layers. It turns out that pure two dimensional materials are not just limited to graphene, rather, 2D hexagonal boron nitride and the family of transition-metal chalcogenides are also present [43][44][45][46][47][48][49][50][51][52] . In particular, the transition-metal trichalcogenide (TMTC) series (with chemical formula TMBX 3 where TM represents transition metals, B = P, Si or Ge and X represents chalcogens) are recently receiving a great attention in both theoretical studies 45,46,53,54 and experiments [47][48][49][50][55][56][57][58] .
In TMTCs, the transition metal ions form a layered honeycomb structure, thus, a single 2D unit consisting of these transition metal atoms has similar lattice structure as that of graphene. However, unlike the case of graphene which has a zero bandgap, TMTC series have a sizable variation of bandgap ranging from 0.5 eV to 3.5 eV depending on the transition metal atoms 59 . In addition, the transition metal compounds possess large spin-orbit coupling and strong electron correlations compared to the case of graphene. Hence, these monolayers of TMTC series open a whole zoo of new exotic phases in two dimensional honeycomb lattice allowing possible control of both electron interaction and spin-orbit coupling. So far, there have been many recent studies on TMTC materials especially with 3d transition metal ions but not much attention on TMTCs with 4d and 5d transition metal ions. In this paper, we study the interplay of spin-orbit coupling and electron correlation motivated by TMTC materials with 4d and 5d transition metal ions. Especially, we study quarter-(or three-quarter) filled system with effective pseudospin-1/2 model. As a minimal model, we consider the Kane-Mele Hubbard model [21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37] . At quarter filling, we found several metallic and insulating phases within mean field approximation; ferromagnetic Chern metals, ferromagnetic Chern insulators and ferromagnetic normal insulators with broken time reversal and inversion symmetries. In particular, we point out that the magnetic Chern insulators could naturally arise when both spin-orbit coupling and electron interactions are present at quarter filling. In addition, we also found the possible transition from Chern insulator to normal insulator as originally proposed in the Haldane model 38 . We also investigate the stability of these phases in the presence of nearest-neighbor interaction and Rashba spin-orbit coupling.
In 4d or 5d TMTCs, the presence of strong spin-orbit coupling and crystal field splitting can split t 2g orbitals of transition metals ions (octahedral sites) into lower quartet orbitals with the effective total angular momentum j = 3/2 and upper doublet with j = 1/2 in the atomic limit 60,61 . When there are 9 or 11 electrons per unit cell (two sites) of honeycomb lattice, the j = 3/2 orbitals are fully filled, while the j = 1/2 orbitals of two sites have one or three electrons in total, resulting in effective quarter-or three-quarter fillings with pseudospin-1/2 model. Such fillings that include odd number of electrons per unit cell, can be realized by combinations of the two transition metal ions in a unit cell. Among Mo, W, Ru, Os, Tc, Re ions, one can combine two ions which satisfy d 4 (or d 6 ) and d 5 in each sublattice. It can also be realized by doping via gating or hydrogen substitution.
Before we study the quarter filled case, we briefly summarize the earlier work related to the Kane-Mele (Hubbard) model. The Kane-Mele model was first proposed to study the quantum spin Hall (QSH) effect in graphene, but, due to very small spin-orbit coupling, the topological properties were not clearly visible. Instead, the search was extended to real materials with strong spin-orbit coupling 19,[62][63][64][65][66] . There were also studies of possible QSH phases due to the spontaneous spin SU(2) symmetry breaking, induced by electron interactions even in the absence of spin-orbit coupling 67,68 . Further related work on topological phase transitions in the presence or absence of spin-orbit coupling have been studied in other lattices like kagome, decorated honeycomb and diamond etc [69][70][71][72] . The electron correlation effects on the Kane-Mele model were also extensively studied at half filling [21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36] . Away from half filling, possible pairing mechanism of superconductivity has received attention which could occur at 3/8 or 5/8 filling near the Van-Hove singularity in doped Kane-Mele model 37,73,74 . However, few studies related to the interplay of strong intrinsic spin-orbit coupling and electron correlations have been discussed for the case of 1/4 or 3/4 filling 75 and there are no detailed study of the full phase diagram at these fillings.

Results
We start by introducing the Kane-Mele Hubbard model. The Hamiltonian is, . There are two vector order parameters represented by M a where a labels the two sublattices (A, B) of honeycomb lattice. This mean-field Hamiltonian  MF has the following form up to a constant term ) with k 1 , k 2 being the momentum components along the basis vectors ê 1 and ê 2 in a honeycomb lattice and In the absence of both spin-orbit coupling and onsite interaction (λ so = U = 0), the system is in a metallic phase. With increasing U but λ so = 0, the magnetic moment is being developed and the system goes into magnetically ordered phases. In the range 6.8 < U < 7.5, the ferromagnetic metal is stabilized where M A = M B ≠ 0 with broken time reversal symmetry. In this phase, h MF (k) is represented as two copies of graphene Hamiltonian with spin up and down, and their energies are separated proportional to the magnetization values M A = M B . At quarter filling, hence, the lowest energy band remains gapless i.e. metallic. At U = U c = 7.5, there is a second order phase transition into a ferromagnetic metal where the system starts developing magnetization M A ≠ M B with broken inversion symmetry. In this case, both M and m in Eq. 3 are non-zero and thus, the lowest two energy bands are separated at every momentum value, but, the bands still cross the Fermi level. On further increasing U, the magnetization keep increasing opening a band gap between the lowest two bands and a ferromagnetic insulator with inversion symmetry broken is stabilized. All of these phases at λ so = 0 are topologically trivial cases, thus we labeled these phases as paramagnetic metal 'PM' , ferromagnetic metal with inversion symmetry 'FM' , ferromagnetic metal with inversion symmetry broken '  FM' and ferromagnetic normal insulator with broken inversion symmetry '  FNI' as shown in Fig. 1. In the presence of spin-orbit coupling (λ so ≠ 0), the non-interacting Kane-Mele model is just two copies of Haldane model with the phase factor φ = π/2 discussed in ref. 38 and opposite sign for spin up and down. Here, there is no extra mass term related to inversion symmetry breaking in the Hamiltonian and the system is metallic at quarter filling. With increasing interaction U, the magnetization is being developed along z direction i.e.
above U c which depends on the value of λ so . In Eq. (3), this is equivalent to M z ≠ 0 and M x,y and m are zero. The preference of magnetization along z direction can be easily understood by comparing the energies of single particle mean field Hamiltonian h MF (k) for two different cases, In the former case, the single particle energy of the lowest band is − . Near the second order transition point from paramagnetic metal to ferromagnetic metal, the magnetization value M is small and we see that the mean field solution with M z ≠ 0 has lower energy than the case with M x(y) ≠ 0, thus magnetic order along z direction is favored. With a finite M, the degenerate bands are separated having non zero Chern number ±1.
Although the lowest two bands are well separated at each momentum, the bands still cross the Fermi level at quarter filling, thus a ferromagnetic Chern metal phase, 'FCM z ' , is stabilized. With further increasing U, the lowest two bands are eventually separated, resulting in a second order phase transition from 'FCM z ' phase to ferromagnetic Chern insulating phase denoted as 'FCI z ' with M z ≠ 0 (See Fig. 1). In this phase, quarter filling corresponds to filling the lowest band that has the Chern number +1, resulting in the Hall conductivity σ H = e 2 /h. This Chern insulating phase can also be confirmed by the edge state calculation. We consider the honeycomb lattice with periodic boundary condition along one of the basis vector and zigzag edge along the direction of other basis vector. Based on the mean field Hamiltonian, we can plot the energy dispersion of the bands with edge states as shown in Fig. 2. Figure 2(a) shows the energy dispersion in FCI z at U = 8 and λ so = 0.3. We see that there are gapless edge states for the FCI z phase along with the bulk energy gap thus indicating it to be a non-trivial phase. As shown in Fig. 1, it is remarkable that the presence of both spin-orbit coupling and Coulomb repulsion naturally opens a wide range of ferromagnetic Chern insulator at quarter filling. This is very distinct situation compared to the half filled case where the QSH phase exists even without electron correlation.
For . In this phase, the system still has non-trivial topology and becomes ferromagnetic Chern insulator with broken inversion symmetry denoted as '  FCI z ' . With further increasing U, there exist a critical value of m z where the lowest two bands cross and then the system goes into a ferromagnetic normal insulator with broken inversion symmetry denoted as '  FNI z ' . Such phase transitions between FCI z or  FCI z and  FNI z are exactly consistent with the phase transition in the original Haldane model that is induced by inversion breaking mass term at the phase φ = π/2 38 . These phases and phase transitions are only stable for weak spin-orbit coupling. For λ so < 0.06, there are two metallic phases with broken inversion symmetry: ferromagnetic Chern metal '  FCM z ' and ferromagnetic normal metal '  FNM z ' as shown in Fig. 1. For very large values of U, the system stabilizes a ferromagnetic normal insulator with magnetization in xy plane but with broken inversion symmetry denoted as  FNI xy . (See Fig. 1) The  FNI xy phase can be understood by considering the spin-orbit coupling term as a perturbation in the large U limit. In the absence of spin-orbit coupling, the single particle mean-field energies are exactly the same irrespective of whether the magnetization is along z direction (case I) or xy plane (case II) due to SU(2) symmetry. When the spin-orbit coupling is small but finite, the second order correction in energy is − . In the ferromagnetic phase with broken inversion symmetry, increase of U leads to large M ≈ m. Hence, the second order correction for the case I is proportional to 1/M 3 and for the case II the correction is proportional to 1/M. Therefore, at large U limit, when the spin-orbit coupling term is present, the magnetization along xy plane is preferred. For small  λ . 0 2 so , there is a first order phase transition from  FNI z phase to  FNI xy , while for  λ . 0 2 so , first order phase transition is directly from FCI z phase to  FNI xy phase. (See Fig. 1). Figure 2(b) shows the energy band for the honeycomb lattice with zigzag edge in  FNI xy phase, at U = 13 and λ so = 0.3. There is no edge state crossing at quarter filling which indicates the system in a trivial insulating phase.
Finally, we also discuss the stability of the phase diagram in the presence of Rashba spin-orbit coupling and the nearest neighbor Coulomb interaction. The nearest-neighbor Coulomb interaction favors charge order and develops a mass term which breaks the inversion symmetry of the lattice. Thus, the phase with ferromagnetic order M A ≠ M B is further stabilized and the area of inversion broken ferromagnetic normal insulator phase is increased in the phase diagram. When both intrinsic (λ so ) and Rashba (λ R ) spin-orbit couplings are present, the particle hole symmetry is broken. In this case, the energy dispersion for non-interacting Hamiltonian ε λ λ ( , ) k n so R is independent of the sign of λ R but depends on the sign of λ so ; n∈ [1,4] is the band index from the lowest to highest energy bands. Here, ε λ λ . This can be easily seen from the energy form of the single particle Hamiltonian with both λ so and λ R . We found that the phase diagram remains almost unchanged for λ R ≤ |λ so | but the magnetization value decreases with the increase of λ R . For λ R > |λ so | and negative λ so , the system in  FNI xy phase undergoes a first order phase transition to FCI z phase and the critical λ R value for this transition increases with increase in U. For λ R > λ so with positive λ so , there is a first order transition from  FNI xy to  FNI z phase. For the latter case, the system in FCI z phase goes to  FNI z under first order phase transition for intermediate U.

Conclusion
In conclusion, we have studied the interplay of intrinsic spin-orbit coupling and onsite Coulomb interactions at quarter filled honeycomb lattice and predicted possible phases realizable in series of transition metal trichalcogenides under optimal doping. Especially with 4d and 5d transition metal ions, we have focused on the Kane-Mele Hubbard model at quarter filling and have shown possible realization of magnetic Chern insulators due to the presence of both electron interaction and spin-orbit coupling, as shown in Fig. 1. Within mean field approximation, we found that the magnetic Chern insulating phase is naturally opened for a wide range of interaction strength and spin orbit coupling as shown in Fig. 1. Furthermore, it can also lead to the phase transition between Chern insulator and normal insulator that was originally proposed by Haldane 38 , by stabilizing magnetic order even in the presence of Rashba spin-orbit coupling and nearest neighbor Coulomb interaction.
The realization of such topological phases in real materials will be very interesting as a future work. In addition, the effect of magnetic field on these systems with spin-orbit coupling and considerable electron correlation is also interesting which is beyond the scope of this paper. In the presence of finite temperature, the magnetic Chern insulating phase with magnetization along the z-direction are robust when both the interaction and the spin orbit coupling strength is large. On increasing the temperature, the fluctuation effect becomes more dominant and above a critical value, these phases may no longer remain robust. In the presence of finite temperature, the spontaneous symmetry breaking of the continuous symmetry is prohibited, hence, the phase transition to ferromagnetic phases with magnetization along x or y direction may not be a stable phase though quasiordering is possible.

Methods
We adopt the mean field approximation to solve the Kane-Mele Hubbard model Eq. (1), resulting in the phase diagram shown in Fig. 1. The mean field Hamiltonian is given in Eq. (2). Here the fluctuation term is neglected as in mean field theory and we assume that the deviation of the operator from its average value is very small. Except near the transition points, the fluctuations are very small and thus can be ignored. Within the mean field approximation, our order parameter is given as . The whole process from step (ii) to (iv) is repeated until all the quantities converge. We repeat this process for various initial guesses and sometimes find different mean field solutions. Comparing the energies of these solutions, we pick up the lowest energy state as the ground state of the Hamiltonian.