# Magnetic Chern Insulators in a monolayer of Transition Metal Trichalcogenides

## Abstract

A monolayer of transition metal trichalcogenides has received a lot of attention as potential two dimensional magnetic materials. The system has a honeycomb structure of transition metal ions, where both spin-orbit coupling and electron correlation effect play an important role. Here, motivated by these transition metal series with effective doping or mixed valence case, we propose the possible realization of magnetic Chern insulators at quarter filled honeycomb lattice. We show that the interplay of intrinsic spin-orbit coupling and electron correlation opens a wide region of ferromagnetic Chern insulating phases in between metals and normal insulators. Within the mean field approximation, we present the phase diagram of a quarter filled Kane-Mele Hubbard model and also discuss the effects of Rashba spin-orbit coupling and nearest neighbor interactions on it.

## Introduction

The effect of spin-orbit coupling plays an important role in the electronic structures of solids. In particular, spin-orbit coupling is essential for the realization of topological insulators where the gapless edge states are protected by time reversal symmetry1,2,3,4,5,6,7,8,9,10. Both theoretical prediction4,11,12 and experimental realization13,14,15,16,17,18 of topological insulators in real materials have been extended to the study of topologically non-trivial phases. More recently, it is also pointed out that the electron interaction effect could induce exotic phases such as topological Mott insulators19,20. Thus, the interplay of spin-orbit coupling and electron interaction has garnered a lot of attention, leading to a new discovery of materials and theoretical studies21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37.

Topologically distinct phases introduced by Haldane showed that the quantum Hall phenomena could occur purely from the band structure in the absence of any external magnetic field, as a realization of the parity anomaly in (2 + 1) dimensional relativistic field theory38. By introducing the staggered flux on a honeycomb lattice, the system becomes an insulator with a non-zero topological invariant, termed as a Chern insulator. Experimental realization of Haldane model has been recently proposed in ultra cold atom system39, yet none of them has been reported in any solid state systems.

Here, we propose possible realization of Chern insulators in two dimensional van der Waals materials, especially in transition metal trichalcogenides. The van der Waals materials are characterized with layered crystals where individual layers are weakly coupled via van der Waals forces but with strong covalent bonding in the layer. Thus it is possible to peel away a single layer breaking the van der Waals bonds. One of the most well known examples is a single layer of graphene, peeled away from bulk graphite using scotch tape40,41,42. Using scotch tape technique, a variety of van der Waals materials have been successfully exfoliated into atomically thin layers. It turns out that pure two dimensional materials are not just limited to graphene, rather, 2D hexagonal boron nitride and the family of transition-metal chalcogenides are also present43,44,45,46,47,48,49,50,51,52. In particular, the transition-metal trichalcogenide (TMTC) series (with chemical formula TMBX 3 where TM represents transition metals, B = P, Si or Ge and X represents chalcogens) are recently receiving a great attention in both theoretical studies45,46,53,54 and experiments47,48,49,50,55,56,57,58.

In TMTCs, the transition metal ions form a layered honeycomb structure, thus, a single 2D unit consisting of these transition metal atoms has similar lattice structure as that of graphene. However, unlike the case of graphene which has a zero bandgap, TMTC series have a sizable variation of bandgap ranging from 0.5 eV to 3.5 eV depending on the transition metal atoms59. In addition, the transition metal compounds possess large spin-orbit coupling and strong electron correlations compared to the case of graphene. Hence, these monolayers of TMTC series open a whole zoo of new exotic phases in two dimensional honeycomb lattice allowing possible control of both electron interaction and spin-orbit coupling. So far, there have been many recent studies on TMTC materials especially with 3d transition metal ions but not much attention on TMTCs with 4d and 5d transition metal ions.

In this paper, we study the interplay of spin-orbit coupling and electron correlation motivated by TMTC materials with 4d and 5d transition metal ions. Especially, we study quarter- (or three-quarter) filled system with effective pseudospin-1/2 model. As a minimal model, we consider the Kane-Mele Hubbard model21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37. At quarter filling, we found several metallic and insulating phases within mean field approximation; ferromagnetic Chern metals, ferromagnetic Chern insulators and ferromagnetic normal insulators with broken time reversal and inversion symmetries. In particular, we point out that the magnetic Chern insulators could naturally arise when both spin-orbit coupling and electron interactions are present at quarter filling. In addition, we also found the possible transition from Chern insulator to normal insulator as originally proposed in the Haldane model38. We also investigate the stability of these phases in the presence of nearest-neighbor interaction and Rashba spin-orbit coupling.

In 4d or 5d TMTCs, the presence of strong spin-orbit coupling and crystal field splitting can split t2g orbitals of transition metals ions (octahedral sites) into lower quartet orbitals with the effective total angular momentum j = 3/2 and upper doublet with j = 1/2 in the atomic limit60,61. When there are 9 or 11 electrons per unit cell (two sites) of honeycomb lattice, the j = 3/2 orbitals are fully filled, while the j = 1/2 orbitals of two sites have one or three electrons in total, resulting in effective quarter- or three-quarter fillings with pseudospin-1/2 model. Such fillings that include odd number of electrons per unit cell, can be realized by combinations of the two transition metal ions in a unit cell. Among Mo, W, Ru, Os, Tc, Re ions, one can combine two ions which satisfy d4 (or d6) and d5 in each sublattice. It can also be realized by doping via gating or hydrogen substitution.

Before we study the quarter filled case, we briefly summarize the earlier work related to the Kane-Mele (Hubbard) model. The Kane-Mele model was first proposed to study the quantum spin Hall (QSH) effect in graphene, but, due to very small spin-orbit coupling, the topological properties were not clearly visible. Instead, the search was extended to real materials with strong spin-orbit coupling19,62,63,64,65,66. There were also studies of possible QSH phases due to the spontaneous spin SU(2) symmetry breaking, induced by electron interactions even in the absence of spin-orbit coupling67,68. Further related work on topological phase transitions in the presence or absence of spin-orbit coupling have been studied in other lattices like kagome, decorated honeycomb and diamond etc69,70,71,72. The electron correlation effects on the Kane-Mele model were also extensively studied at half filling21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36. Away from half filling, possible pairing mechanism of superconductivity has received attention which could occur at 3/8 or 5/8 filling near the Van-Hove singularity in doped Kane-Mele model37,73,74. However, few studies related to the interplay of strong intrinsic spin-orbit coupling and electron correlations have been discussed for the case of 1/4 or 3/4 filling75 and there are no detailed study of the full phase diagram at these fillings.

## Results

We start by introducing the Kane-Mele Hubbard model. The Hamiltonian is,

$$\begin{array}{rcl} {\mathcal H} & = & -t\sum _{\langle ij\rangle ,\alpha }({c}_{i\alpha }^{\dagger }{c}_{j\alpha }+h\mathrm{.}c\mathrm{.})\\ & & -i{\lambda }_{so}\sum _{\langle \langle ij\rangle \rangle ,\sigma }({c}_{i\alpha }^{\dagger }{\nu }_{ij}{\sigma }_{\alpha \beta }^{z}{c}_{j\beta }+h\mathrm{.}c\mathrm{.})+U\sum _{i}{n}_{i\uparrow }{n}_{i\downarrow },\end{array}$$
(1)

where $${c}_{i\alpha }^{\dagger }$$ (c ) is the electron creation (annihilation) operator at site i with spin α {↑, ↓} on a honeycomb lattice, $${n}_{i\alpha }={c}_{i\alpha }^{\dagger }{c}_{i\alpha }$$ is the number density operator, σz is a Pauli matrix, 〈ij〉 and 〈〈ij〉〉 denotes pairs of nearest-neighbor and next-nearest-neighbor sites respectively. t, U and λ so are the nearest-neighbor hopping energy, the strength of the on-site Coulomb repulsion and the second-neighbor spin-orbit coupling strength respectively. Throughout this paper, we set the hopping amplitude t ≡ 1. ν ij  = −ν ji  = ±1, depending on whether the electron traversing from i to j makes a right (+1) or a left (−1) turn.

At quarter filling, the system remains metallic with or without spin-orbit coupling for the non-interacting case U = 0. At this filling, irrespective of the spin-orbit coupling strength, the perfect nesting wave vectors are absent at the Fermi surface. Thus, we neglect the instability of any charge density wave or spin density wave with finite momentum when the onsite repulsion U is turned on. The interaction term in the Hamiltonian Eq. (1) can be rewritten in terms of spin operator $${\boldsymbol{S}}={c}_{i\alpha }^{\dagger }\frac{{{\boldsymbol{\sigma }}}_{\alpha \beta }}{2}{c}_{i\beta }$$ and the total number of electrons in the system is N e ; $$U\,{\sum }_{i}{n}_{i\uparrow }{n}_{i\downarrow }=-{\sum }_{i}\frac{2U}{3}{{\boldsymbol{S}}}_{i}^{2}+\frac{U{N}_{e}}{2}$$. The last term $$\frac{U{N}_{e}}{2}$$ is a constant and just shifts the total energy of the system, thus can be ignored. We solve this interacting Hamiltonian using mean field approximation and the mean field Hamiltonian can be written as,

$${ {\mathcal H} }_{MF}={ {\mathcal H} }_{0}+\sum _{i}{{\boldsymbol{M}}}_{i}\cdot {{\boldsymbol{S}}}_{i}+\frac{3}{8U}\sum _{i}{({{\boldsymbol{M}}}_{i})}^{2}$$
(2)

where $${ {\mathcal H} }_{0}$$ is the non-interacting part of the Hamiltonian and $${{\boldsymbol{M}}}_{{i}}=-\frac{4U}{3}\langle {{\boldsymbol{S}}}_{i}\rangle$$. There are two vector order parameters represented by M a where a labels the two sublattices (A, B) of honeycomb lattice. This mean-field Hamiltonian $${ {\mathcal H} }_{MF}$$ has the following form up to a constant term

$${ {\mathcal H} }_{MF}=\sum _{{\boldsymbol{k}}}{c}_{{\boldsymbol{k}}}^{\dagger }{h}_{MF}({\boldsymbol{k}}){c}_{{\boldsymbol{k}}},$$
(3)
$$\begin{array}{rcl}{h}_{MF}({\boldsymbol{k}}) & = & {d}_{1}({\boldsymbol{k}}\mathrm{)\ }{\tau }^{x}\otimes {\mathbb{I}}+{d}_{2}({\boldsymbol{k}}\mathrm{)\ }{\tau }^{y}\otimes {\mathbb{I}}+{d}_{3}({\boldsymbol{k}}\mathrm{)\ }{\tau }^{z}\otimes {\sigma }^{z}\\ & & +\sum _{\mu \in \{x,y,z\}}({M}^{\mu }{\mathbb{I}}\otimes {\sigma }^{\mu }+{m}^{\mu }{\tau }^{z}\otimes {\sigma }^{\mu })\end{array}$$
(4)

where c k  = (cA(k), cB(k), cA(k), cB(k))T is the basis for honeycomb (A, B) sublattices with spins ↑, ↓. The Hamiltonian matrix h MF (k) can be represented in terms of two Pauli matrices τ = (τx, τy, τz) and σ = (σx, σy, σz) for A, B sublattices and spin ↑, ↓ respectively. d1(k) = (1 + cos k1 + cos k2), d2(k) = (sin k1 − sin k2) and d3(k) = 2λ so (sin k1 + sin k2 − sin (k1 + k2)) with k1, k2 being the momentum components along the basis vectors $${\hat{{\boldsymbol{e}}}}_{1}$$ and $${\hat{{\boldsymbol{e}}}}_{2}$$ in a honeycomb lattice and $${M}^{\mu }\equiv ({M}_{A}^{\mu }+{M}_{B}^{\mu }\mathrm{)/4}$$, $${m}^{\mu }\equiv ({M}_{A}^{\mu }-{M}_{B}^{\mu }\mathrm{)/4}$$ with μ = x, y, z.

Figure 1 is the phase diagram as a function of U and λ so at quarter filling, based on solving the self consistency equation. The phases in the figure are classified in terms of the magnetization. The phase where both M and m are zero is the paramagnetic phase. While M ≠ 0, m = 0 M A  = M B phase corresponds to the ferromagnetic phase, M ≠ 0, m ≠ 0 M A  ≠ M B corresponds to the ferromagnetic phase with broken inversion symmetry. The phase diagram is explained in detail below.

In the absence of both spin-orbit coupling and onsite interaction (λ so  = U = 0), the system is in a metallic phase. With increasing U but λ so  = 0, the magnetic moment is being developed and the system goes into magnetically ordered phases. In the range 6.8 < U < 7.5, the ferromagnetic metal is stabilized where M A  = M B  ≠ 0 with broken time reversal symmetry. In this phase, h MF (k) is represented as two copies of graphene Hamiltonian with spin up and down, and their energies are separated proportional to the magnetization values M A  = M B . At quarter filling, hence, the lowest energy band remains gapless i.e. metallic. At U = U c  = 7.5, there is a second order phase transition into a ferromagnetic metal where the system starts developing magnetization M A  ≠ M B with broken inversion symmetry. In this case, both M and m in Eq. 3 are non-zero and thus, the lowest two energy bands are separated at every momentum value, but, the bands still cross the Fermi level. On further increasing U, the magnetization keep increasing opening a band gap between the lowest two bands and a ferromagnetic insulator with inversion symmetry broken is stabilized. All of these phases at λ so  = 0 are topologically trivial cases, thus we labeled these phases as paramagnetic metal ‘PM’, ferromagnetic metal with inversion symmetry ‘FM’, ferromagnetic metal with inversion symmetry broken ‘$$\tilde{F}M$$’ and ferromagnetic normal insulator with broken inversion symmetry ‘$$\tilde{F}NI$$’ as shown in Fig. 1.

In the presence of spin-orbit coupling (λ so  ≠ 0), the non-interacting Kane-Mele model is just two copies of Haldane model with the phase factor ϕ = π/2 discussed in ref.38 and opposite sign for spin up and down. Here, there is no extra mass term related to inversion symmetry breaking in the Hamiltonian and the system is metallic at quarter filling. With increasing interaction U, the magnetization is being developed along z direction i.e. $${M}_{A}^{z}={M}_{B}^{z}\ne 0$$ above U c which depends on the value of λ so . In Eq. (3), this is equivalent to Mz ≠ 0 and Mx,y and m are zero. The preference of magnetization along z direction can be easily understood by comparing the energies of single particle mean field Hamiltonian h MF (k) for two different cases, $${M}_{A}^{z}={M}_{B}^{z}=M$$ and $${M}_{A}^{x(y)}={M}_{B}^{x(y)}=M$$. In the former case, the single particle energy of the lowest band is $$-\sqrt{{d}_{1}^{2}({\boldsymbol{k}})+{d}_{2}^{2}({\boldsymbol{k}})+{d}_{3}^{2}({\boldsymbol{k}})}-M\mathrm{/2}$$. In the latter case, the energy of the lowest band is $$-\sqrt{{[{({d}_{1}^{2}({\boldsymbol{k}})+{d}_{2}^{2}({\boldsymbol{k}}))}^{\frac{1}{2}}+M\mathrm{/2}]}^{2}+{d}_{3}^{2}({\boldsymbol{k}})}$$. Near the second order transition point from paramagnetic metal to ferromagnetic metal, the magnetization value M is small and we see that the mean field solution with Mz ≠ 0 has lower energy than the case with Mx(y) ≠ 0, thus magnetic order along z direction is favored. With a finite M, the degenerate bands are separated having non zero Chern number ±1. Although the lowest two bands are well separated at each momentum, the bands still cross the Fermi level at quarter filling, thus a ferromagnetic Chern metal phase, ‘FCM z ’, is stabilized.

With further increasing U, the lowest two bands are eventually separated, resulting in a second order phase transition from ‘FCM z ’ phase to ferromagnetic Chern insulating phase denoted as ‘FCI z ’ with Mz ≠ 0 (See Fig. 1). In this phase, quarter filling corresponds to filling the lowest band that has the Chern number +1, resulting in the Hall conductivity σ H  = e2/h. This Chern insulating phase can also be confirmed by the edge state calculation. We consider the honeycomb lattice with periodic boundary condition along one of the basis vector and zigzag edge along the direction of other basis vector. Based on the mean field Hamiltonian, we can plot the energy dispersion of the bands with edge states as shown in Fig. 2. Figure 2(a) shows the energy dispersion in FCI z at U = 8 and λ so  = 0.3. We see that there are gapless edge states for the FCI z phase along with the bulk energy gap thus indicating it to be a non-trivial phase. As shown in Fig. 1, it is remarkable that the presence of both spin-orbit coupling and Coulomb repulsion naturally opens a wide range of ferromagnetic Chern insulator at quarter filling. This is very distinct situation compared to the half filled case where the QSH phase exists even without electron correlation.

For $$0.06\le {\lambda }_{so}\lesssim 0.2$$, there is another second order phase transition within mean field approximation. Increase of U leads to different magnetic phases where $${M}_{A}^{z}\ne {M}_{B}^{z}$$, thus both Mz and mz are non zero in Eq.(3). In this phase, the system still has non-trivial topology and becomes ferromagnetic Chern insulator with broken inversion symmetry denoted as ‘$$\tilde{F}C{I}_{z}$$’. With further increasing U, there exist a critical value of mz where the lowest two bands cross and then the system goes into a ferromagnetic normal insulator with broken inversion symmetry denoted as ‘$$\tilde{F}N{I}_{z}$$’. Such phase transitions between FCI z or $$\tilde{F}C{I}_{z}$$ and $$\tilde{F}N{I}_{z}$$ are exactly consistent with the phase transition in the original Haldane model that is induced by inversion breaking mass term at the phase ϕ = π/238. These phases and phase transitions are only stable for weak spin-orbit coupling. For λ so  < 0.06, there are two metallic phases with broken inversion symmetry: ferromagnetic Chern metal ‘$$\tilde{F}C{M}_{z}$$’ and ferromagnetic normal metal ‘$$\tilde{F}N{M}_{z}$$’ as shown in Fig. 1.

For very large values of U, the system stabilizes a ferromagnetic normal insulator with magnetization in xy plane but with broken inversion symmetry denoted as $$\tilde{F}N{I}_{xy}$$. (See Fig. 1) The $$\tilde{F}N{I}_{xy}$$ phase can be understood by considering the spin-orbit coupling term as a perturbation in the large U limit. In the absence of spin-orbit coupling, the single particle mean-field energies are exactly the same irrespective of whether the magnetization is along z direction (case I) or xy plane (case II) due to SU(2) symmetry. When the spin-orbit coupling is small but finite, the second order correction in energy is $$-4{d}_{3}^{2}({d}_{1}^{2}({\boldsymbol{k}})+{d}_{2}^{2}({\boldsymbol{k}}))/{({m}^{2}+4({d}_{1}^{2}({\boldsymbol{k}})+{d}_{2}^{2}({\boldsymbol{k}})))}^{\mathrm{3/2}}$$ for case I, whereas for case II, the correction is $$-{d}_{3}^{2}/(M+\sqrt{{m}^{2}+4({d}_{1}^{2}({\boldsymbol{k}})+{d}_{2}^{2}({\boldsymbol{k}}))})$$. In the ferromagnetic phase with broken inversion symmetry, increase of U leads to large M ≈ m. Hence, the second order correction for the case I is proportional to 1/M3 and for the case II the correction is proportional to 1/M. Therefore, at large U limit, when the spin-orbit coupling term is present, the magnetization along xy plane is preferred. For small $${\lambda }_{so}\lesssim 0.2$$, there is a first order phase transition from $$\tilde{F}N{I}_{z}$$ phase to $$\tilde{F}N{I}_{xy}$$, while for $${\lambda }_{so}\gtrsim 0.2$$, first order phase transition is directly from FCI z phase to $$\tilde{F}N{I}_{xy}$$ phase. (See Fig. 1). Figure 2(b) shows the energy band for the honeycomb lattice with zigzag edge in $$\tilde{F}N{I}_{xy}$$ phase, at U = 13 and λ so  = 0.3. There is no edge state crossing at quarter filling which indicates the system in a trivial insulating phase.

Finally, we also discuss the stability of the phase diagram in the presence of Rashba spin-orbit coupling and the nearest neighbor Coulomb interaction. The nearest-neighbor Coulomb interaction favors charge order and develops a mass term which breaks the inversion symmetry of the lattice. Thus, the phase with ferromagnetic order MA ≠ MB is further stabilized and the area of inversion broken ferromagnetic normal insulator phase is increased in the phase diagram. When both intrinsic (λ so ) and Rashba (λ R ) spin-orbit couplings are present, the particle hole symmetry is broken. In this case, the energy dispersion for non-interacting Hamiltonian $${\varepsilon }_{{\boldsymbol{k}}}^{n}({\lambda }_{so},{\lambda }_{R})$$ is independent of the sign of λ R but depends on the sign of λ so ; n[1, 4] is the band index from the lowest to highest energy bands. Here, $${\varepsilon }_{{\boldsymbol{k}}}^{n}({\lambda }_{so},{\lambda }_{R})=-{\varepsilon }_{{\boldsymbol{k}}}^{5-n}(-{\lambda }_{so},{\lambda }_{R})$$. This can be easily seen from the energy form of the single particle Hamiltonian with both λ so and λ R . We found that the phase diagram remains almost unchanged for λ R  ≤ |λ so | but the magnetization value decreases with the increase of λ R . For λ R  > |λ so | and negative λ so , the system in $$\tilde{F}N{I}_{xy}$$ phase undergoes a first order phase transition to FCI z phase and the critical λ R value for this transition increases with increase in U. For λ R  > λ so with positive λ so , there is a first order transition from $$\tilde{F}N{I}_{xy}$$ to $$\tilde{F}N{I}_{z}$$ phase. For the latter case, the system in FCI z phase goes to $$\tilde{F}N{I}_{z}$$ under first order phase transition for intermediate U.

## Conclusion

In conclusion, we have studied the interplay of intrinsic spin-orbit coupling and onsite Coulomb interactions at quarter filled honeycomb lattice and predicted possible phases realizable in series of transition metal trichalcogenides under optimal doping. Especially with 4d and 5d transition metal ions, we have focused on the Kane-Mele Hubbard model at quarter filling and have shown possible realization of magnetic Chern insulators due to the presence of both electron interaction and spin-orbit coupling, as shown in Fig. 1. Within mean field approximation, we found that the magnetic Chern insulating phase is naturally opened for a wide range of interaction strength and spin orbit coupling as shown in Fig. 1. Furthermore, it can also lead to the phase transition between Chern insulator and normal insulator that was originally proposed by Haldane38, by stabilizing magnetic order even in the presence of Rashba spin-orbit coupling and nearest neighbor Coulomb interaction.

The realization of such topological phases in real materials will be very interesting as a future work. In addition, the effect of magnetic field on these systems with spin-orbit coupling and considerable electron correlation is also interesting which is beyond the scope of this paper. In the presence of finite temperature, the magnetic Chern insulating phase with magnetization along the z-direction are robust when both the interaction and the spin orbit coupling strength is large. On increasing the temperature, the fluctuation effect becomes more dominant and above a critical value, these phases may no longer remain robust. In the presence of finite temperature, the spontaneous symmetry breaking of the continuous symmetry is prohibited, hence, the phase transition to ferromagnetic phases with magnetization along x or y direction may not be a stable phase though quasiordering is possible.

## Methods

We adopt the mean field approximation to solve the Kane-Mele Hubbard model Eq. (1), resulting in the phase diagram shown in Fig. 1. The mean field Hamiltonian is given in Eq. (2). Here the fluctuation term is neglected as in mean field theory and we assume that the deviation of the operator from its average value is very small. Except near the transition points, the fluctuations are very small and thus can be ignored. Within the mean field approximation, our order parameter is given as

$${{\boldsymbol{M}}}_{a}=-\frac{4U}{3}\langle {{\boldsymbol{S}}}_{a}\rangle =-\frac{4U}{3}{\langle {c}_{ka\alpha }^{\dagger }\frac{{{\boldsymbol{\sigma }}}_{\alpha \beta }}{2}{c}_{ka\beta }\rangle }_{MF}\mathrm{.}$$

In order to solve a self-consistent equation for M, (i) start the iteration with a random initial guess for each components of M a , (ii) diagonalize H MF using M a and find the energy and eigenfunctions, H MF |ψ n (k, M a )〉 = ε n (k, M a )|ψ n (k, M a )〉, where k takes the value in the N × N Brillouin Zone mesh and n is the band index. Here we took the value of N upto 150 and checked the phases are robust against the change of N. (iii) Tune the chemical potential μ to quarter filling by fixing the number of particles, $$1=\frac{1}{{N}^{2}}{\sum }_{k,n}{n}_{F}[{\varepsilon }_{n}({\bf{k}},{{\bf{M}}}_{{\bf{a}}})\mu ]$$ where n F [ε n (k, M a ), μ] is the Fermi distribution function. (iv) Using the eigenfunctions of the mean field Hamiltonian, calculate the expectation value of the spin vector on each site in the unit cell and compute the new values of $${{\boldsymbol{M}}}_{a}=-\frac{4U}{3}\langle {{\boldsymbol{S}}}_{a}\rangle =-\frac{4U}{3}\frac{1}{{N}^{2}}{\sum }_{k}\langle {c}_{ka\alpha }^{\dagger }\frac{{{\boldsymbol{\sigma }}}_{\alpha \beta }}{2}{c}_{ka\beta }\rangle {n}_{F}[{\varepsilon }_{n}({\bf{k}},{{\bf{M}}}_{{\bf{a}}}),\mu ]$$. The whole process from step (ii) to (iv) is repeated until all the quantities converge. We repeat this process for various initial guesses and sometimes find different mean field solutions. Comparing the energies of these solutions, we pick up the lowest energy state as the ground state of the Hamiltonian.

## References

1. 1.

Kane, C. L. & Mele, E. J. Quantum spin hall effect in graphene. Phys. Rev. Lett. 95, 226801 (2005).

2. 2.

Kane, C. L. & Mele, E. J. Z 2 topological order and the quantum spin hall effect. Phys. Rev. Lett. 95, 146802 (2005).

3. 3.

Fu, L., Kane, C. L. & Mele, E. J. Topological insulators in three dimensions. Phys. Rev. Lett. 98, 106803 (2007).

4. 4.

Fu, L. & Kane, C. L. Topological insulators with inversion symmetry. Phys. Rev. B 76, 045302 (2007).

5. 5.

Büttiker, M. Edge-state physics without magnetic fields. Science 325, 278–279 (2009).

6. 6.

Roy, R. Topological phases and the quantum spin hall effect in three dimensions. Phys. Rev. B 79, 195322 (2009).

7. 7.

Moore, J. E. The birth of topological insulators. Nature 464, 194–198 (2010).

8. 8.

Hasan, M. Z. & Kane, C. L. Colloquium: topological insulators. Rev. of Mod. Phys. 82, 3045 (2010).

9. 9.

Qi, X.-L. & Zhang, S.-C. The quantum spin hall effect and topological insulators. Physics Today 63, 33–38 (2010).

10. 10.

Ren, Y., Qiao, Z. & Niu, Q. Topological phases in two-dimensional materials: a review. Reports on Progress in Physics 79, 066501 (2016).

11. 11.

Bernevig, B. A., Hughes, T. L. & Zhang, S.-C. Quantum spin hall effect and topological phase transition in hgte quantum wells. Science 314, 1757–1761 (2006).

12. 12.

Teo, J. C. Y., Fu, L. & Kane, C. L. Surface states and topological invariants in three-dimensional topological insulators: Application to bi1−xsb x . Phys. Rev. B 78, 045426 (2008).

13. 13.

Chen, Y. et al. Experimental realization of a three-dimensional topological insulator, Bi2Te3. Science 325, 178–181 (2009).

14. 14.

König, M. et al. Quantum spin hall insulator state in hgte quantum wells. Science 318, 766–770 (2007).

15. 15.

Hsieh, D. et al. A topological dirac insulator in a quantum spin hall phase. Nature 452, 970–974 (2008).

16. 16.

Hsieh, D. et al. Observation of unconventional quantum spin textures in topological insulators. Science 323, 919–922 (2009).

17. 17.

Xia, Y. et al. Observation of a large-gap topological-insulator class with a single dirac cone on the surface. Nature Physics 5, 398–402 (2009).

18. 18.

Hsieh, D. et al. A tunable topological insulator in the spin helical dirac transport regime. Nature 460, 1101–1105 (2009).

19. 19.

Pesin, D. & Balents, L. Mott physics and band topology in materials with strong spin–orbit interaction. Nature Physics 6, 376–381 (2010).

20. 20.

Liu, T., Douçot, B. & Le Hur, K. Realizing topological mott insulators from the rkky interaction. Phys. Rev. B 93, 195153 (2016).

21. 21.

Zheng, D., Zhang, G.-M. & Wu, C. Particle-hole symmetry and interaction effects in the kane-mele-hubbard model. Phys. Rev. B 84, 205121 (2011).

22. 22.

Hohenadler, M. et al. Quantum phase transitions in the Kane Mele Hubbard model. Phys. Rev. B 85, 115132 (2012).

23. 23.

Hung, H.-H., Wang, L., Gu, Z.-C. & Fiete, G. A. Topological phase transition in a generalized kane-mele-hubbard model: A combined quantum monte carlo and green’s function study. Phys. Rev. B 87, 121113 (2013).

24. 24.

Laubach, M., Reuther, J. & Thomale, R. & Rachel, S. Rashba spin-orbit coupling in the kane-mele-hubbard model. Phys. Rev. B 90, 165136 (2014).

25. 25.

Griset, C. & Xu, C. Phase diagram of the kane-mele-hubbard model. Phys. Rev. B 85, 045123 (2012).

26. 26.

Vaezi, A., Mashkoori, M. & Hosseini, M. Phase diagram of the strongly correlated kane-mele-hubbard model. Phys. Rev. B 85, 195126 (2012).

27. 27.

Hohenadler, M., Toldin, F. P., Herbut, I. & Assaad, F. Phase diagram of the kane-mele-coulomb model. Phys. Rev. B 90, 085146 (2014).

28. 28.

Yu, S.-L., Xie, X. & Li, J.-X. Mott physics and topological phase transition in correlated dirac fermions. Phys. Rev. Lett. 107, 010401 (2011).

29. 29.

Meng, Z. Y., Hung, H.-H. & Lang, T. C. The characterization of topological properties in quantum monte carlo simulations of the kane–mele–hubbard model. . Modern Physics Letters B 28, 1430001 (2014).

30. 30.

Hohenadler, M., Lang, T. & Assaad, F. Correlation effects in quantum spin-hall insulators: a quantum monte carlo study. Phys. Rev. Lett. 106, 100403 (2011).

31. 31.

Chung, C.-H., Lee, D.-H. & Chao, S.-P. Kane-mele hubbard model on a zigzag ribbon: stability of the topological edge states and quantum phase transitions. Phys. Rev. B 90, 035116 (2014).

32. 32.

Budich, J. C., Thomale, R., Li, G., Laubach, M. & Zhang, S.-C. Fluctuation-induced topological quantum phase transitions in quantum spin-hall and anomalous-hall insulators. Phys. Rev. B 86, 201407 (2012).

33. 33.

Rachel, S. & Le Hur, K. Topological insulators and mott physics from the hubbard interaction. Phys. Rev. B 82, 075106 (2010).

34. 34.

Wu, W., Rachel, S., Liu, W.-M. & Le Hur, K. Quantum spin hall insulators with interactions and lattice anisotropy. Phys. Rev. B 85, 205102 (2012).

35. 35.

Lang, T. C., Essin, A. M., Gurarie, V. & Wessel, S. Z 2 topological invariants in two dimensions from quantum monte carlo. Phys. Rev. B 87, 205101 (2013).

36. 36.

Araújo, M. A., Castro, E. V. & Sacramento, P. D. Change of an insulator’s topological properties by a hubbard interaction. Phys. Rev. B 87, 085109 (2013).

37. 37.

Wen, J., Kargarian, M., Vaezi, A. & Fiete, G. A. Doping the kane-mele-hubbard model: A slave-boson approach. Phys. Rev. B 84, 235149 (2011).

38. 38.

Haldane, F. D. M. Model for a quantum hall effect without landau levels: Condensed-matter realization of the “parity anomaly”. Phys. Rev. Lett. 61, 2015 (1988).

39. 39.

Jotzu, G. et al. Experimental realization of the topological haldane model with ultracold fermions. Nature 515, 237–240 (2014).

40. 40.

Geim, A. K. & Novoselov, K. S. The rise of graphene. Nature materials 6, 183–191 (2007).

41. 41.

Geim, A. K. & Grigorieva, I. V. Van der waals heterostructures. Nature 499, 419–425 (2013).

42. 42.

Novoselov, K., Mishchenko, A., Carvalho, A. & Neto, A. C. 2d materials and van der waals heterostructures. Science 353, aac9439 (2016).

43. 43.

Wang, Q. H., Kalantar-Zadeh, K., Kis, A., Coleman, J. N. & Strano, M. S. Electronics and optoelectronics of two-dimensional transition metal dichalcogenides. Nature nanotechnology 7, 699–712 (2012).

44. 44.

Xu, X., Yao, W., Xiao, D. & Heinz, T. F. Spin and pseudospins in layered transition metal dichalcogenides. Nature Physics 10, 343–350 (2014).

45. 45.

Chittari, B. L. et al. Electronic and magnetic properties of single-layer mPX3 metal phosphorous trichalcogenides. Phys. Rev. B 94, 184428 (2016).

46. 46.

Sivadas, N., Daniels, M. W., Swendsen, R. H., Okamoto, S. & Xiao, D. Magnetic ground state of semiconducting transition-metal trichalcogenide monolayers. Phys. Rev. B 91, 235425 (2015).

47. 47.

Du, K.-Z. et al. Weak van der waals stacking, wide-range band gap, and raman study on ultrathin layers of metal phosphorus trichalcogenides. ACS nano 10, 1738–1743 (2015).

48. 48.

Lin, M.-W. et al. Ultrathin nanosheets of crsite 3: a semiconducting two-dimensional ferromagnetic material. Journal of Materials Chemistry C 4, 315–322 (2016).

49. 49.

Kuo, C.-T. et al. Exfoliation and raman spectroscopic fingerprint of few-layer nips3 van der waals crystals. Scientific reports 6 (2016).

50. 50.

Lee, J.-U. et al. Ising-type magnetic ordering in atomically thin feps3. Nano Letters 16, 7433–7438 (2016).

51. 51.

Tian, Y., Gray, M. J., Ji, H., Cava, R. & Burch, K. S. Magneto-elastic coupling in a potential ferromagnetic 2d atomic crystal. 2D Materials 3, 025035 (2016).

52. 52.

Yang, H., Kim, S. W., Chhowalla, M. & Lee, Y. H. Structural and quantum-state phase transition in van der waals layered materials. Nature Physics 17, 467–475 (2017).

53. 53.

Sugita, Y., Miyake, T. & Motome, Y. Multiple dirac cones and topological magnetism in honeycomb-monolayer transition metal trichalcogenides. arXiv preprint arXiv:1704.00318 (2017).

54. 54.

Sugita, Y., Miyake, T. & Motome, Y. Electronic band structure of 4d and 5d transition metal trichalcogenides. Physica B: Condensed Matter (2017).

55. 55.

Li, X., Cao, T., Niu, Q., Shi, J. & Feng, J. Coupling the valley degree of freedom to antiferromagnetic order. Proceedings of the National Academy of Sciences 110, 3738–3742 (2013).

56. 56.

Park, J.-G. Opportunities and challenges of two-dimensional magnetic van der waals materials: magnetic graphene? Journal of Physics Condensed Matter 28, 301001 (2016).

57. 57.

Gong, C. et al. Discovery of intrinsic ferromagnetism in two-dimensional van der waals crystals. Nature 546, 265–269 (2017).

58. 58.

Kim, S. Y. et al. Charge-spin correlation and self-doped ground state in van der waals antiferromagenet NiPS3. arXiv preprint arXiv:1706.06259 (2017).

59. 59.

Bullett, D. Variation of electronic properties with structure of transition metal trichalcogenides. Journal of Physics C: Solid State Physics 12, 277 (1979).

60. 60.

Sugano, S. Multiplets Of Transition-Metal Ions In Crystals (Elsevier, 2012).

61. 61.

McGlynn, E. Electron paramagnetic resonance of transition ions, oxford classic texts in the physical sciences, by a. abragam and b. bleaney. Contemporary Physics 54, 115–116 (2013).

62. 62.

Jackeli, G. & Khaliullin, G. Mott insulators in the strong spin-orbit coupling limit: from heisenberg to a quantum compass and kitaev models. Phys. Rev. Lett. 102, 017205 (2009).

63. 63.

Shitade, A. et al. Quantum spin hall effect in a transition metal oxide Na 2 IrO 3. Phys. Rev. Lett. 102, 256403 (2009).

64. 64.

Wunderlich, J., Kaestner, B., Sinova, J. & Jungwirth, T. Experimental observation of the spin-hall effect in a two-dimensional spin-orbit coupled semiconductor system. Phys. Rev. Lett. 94, 047204 (2005).

65. 65.

Moon, E.-G., Xu, C., Kim, Y. B. & Balents, L. Non-fermi-liquid and topological states with strong spin-orbit coupling. Phys. Rev. Lett. 111, 206401 (2013).

66. 66.

Sinova, J. et al. Universal intrinsic spin hall effect. Phys. Rev. Lett. 92, 126603 (2004).

67. 67.

Raghu, S., Qi, X.-L., Honerkamp, C. & Zhang, S.-C. Topological mott insulators. Phys. Rev. Lett. 100, 156401 (2008).

68. 68.

Weeks, C. & Franz, M. Interaction-driven instabilities of a dirac semimetal. Phys. Rev. B 81, 085105 (2010).

69. 69.

Guo, H.-M. & Franz, M. Topological insulator on the kagome lattice. Phys. Rev. B 80, 113102 (2009).

70. 70.

Rüegg, A., Wen, J. & Fiete, G. A. Topological insulators on the decorated honeycomb lattice. Phys. Rev. B 81, 205115 (2010).

71. 71.

Wen, J., Rüegg, A., Wang, C.-C. J. & Fiete, G. A. Interaction-driven topological insulators on the kagome and the decorated honeycomb lattices. Phys. Rev. B 82, 075125 (2010).

72. 72.

Beugeling, W., Everts, J. C. & Morais Smith, C. Topological phase transitions driven by next-nearest-neighbor hopping in two-dimensional lattices. Phys. Rev. B 86, 195129 (2012).

73. 73.

Ma, T., Lin, H.-Q. & Gubernatis, J. E. Triplet p+ ip pairing correlations in the doped kane-mele-hubbard model: A quantum monte carlo study. EPL (Europhysics Letters) 111, 47003 (2015).

74. 74.

Fukaya, Y., Yada, K., Hattori, A. & Tanaka, Y. Pairing mechanism of unconventional superconductivity in doped kane–mele model. Journal of the Physical Society of Japan 85, 104704 (2016).

75. 75.

Murthy, G., Shimshoni, E., Shankar, R. & Fertig, H. A. Quarter-filled honeycomb lattice with a quantized hall conductance. Phys. Rev. B 85, 073103 (2012).

## Acknowledgements

The authors acknowledge supports from KAIST startup and National Research Foundation Grant (NRF-2017R1A2B4008097). A. Mishra is supported by BK21 plus. The authors would like to thank Prof. Kenneth Burch, Prof. Leon Balents, Prof. S. R. Hassan, Dr. Dibyakrupa Sahoo and Dr. Vinu Lukose for their useful comments.

## Author information

Authors

### Contributions

A.M., and S.B.L. formulated the problem and methodology. A.M. did the computations. A.M. and S.B.L. analyzed the results and wrote the paper. S.B.L. developed the ideas for possible experimental realization of the predicted results in the paper.

### Corresponding authors

Correspondence to Archana Mishra or SungBin Lee.

## Ethics declarations

### Competing Interests

The authors declare that they have no competing interests.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Mishra, A., Lee, S. Magnetic Chern Insulators in a monolayer of Transition Metal Trichalcogenides. Sci Rep 8, 799 (2018). https://doi.org/10.1038/s41598-017-18880-z

• Accepted:

• Published:

• ### Interplay of Magnetism and Topological Superconductivity in Bilayer Kagome Metals

• Santu Baidya
• , Aabhaas Vineet Mallik
• , Subhro Bhattacharjee
•  & Tanusri Saha-Dasgupta

Physical Review Letters (2020)

• ### Magnetic anisotropy and low-field magnetic phase diagram of the quasi-two-dimensional ferromagnet Cr2Ge2Te6

• S. Selter
• , G. Bastien
• , A. U. B. Wolter
• , S. Aswartham
•  & B. Büchner

Physical Review B (2020)

• ### Stacking sensitive topological phases in a bilayer Kane-Mele-Hubbard model at quarter filling

• Archana Mishra
•  & SungBin Lee

Physical Review B (2019)

• ### Topological multiferroic phases in the extended Kane-Mele-Hubbard model in the Hofstadter regime

• Archana Mishra
•  & SungBin Lee

Physical Review B (2018)