Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Fermionic order by disorder in a van der Waals antiferromagnet

## Abstract

CeTe3 is a unique platform to investigate the itinerant magnetism in a van der Waals (vdW) coupled metal. Despite chemical pressure being a promising route to boost quantum fluctuation in this system, a systematic study on the chemical pressure effect on Ce3+(4f1) states is absent. Here, we report on the successful growth of a series of Se doped single crystals of CeTe3. We found a fluctuation driven exotic magnetic rotation from the usual easy-axis ordering to an unusual hard-axis ordering. Unlike in localized magnetic systems, near-critical magnetism can increase itinerancy hand-in-hand with enhancing fluctuation of magnetism. Thus, seemingly unstable hard-axis ordering emerges through kinetic energy gain, with the self-consistent observation of enhanced magnetic fluctuation (disorder). As far as we recognize, this order-by-disorder process in fermionic system is observed for the first time within vdW materials. Our finding opens a unique experimental platform for direct visualization of the rich quasiparticle Fermi surface deformation associated with the Fermionic order-by-disorder process. Also, the search for emergent exotic phases by further tuning of quantum fluctuation is suggested as a promising future challenge.

## Introduction

In the search for interesting metallic states in quantum materials, controlling magnetic orientation, fluctuations, and their interaction with conduction electrons has been of utmost importance. Consequent exotic phases are rich, ranging from unconventional superconductivity1,2,3,4 to giant magneto-transport effects, including metal-to-insulator transition5, magnetically driven 2D confinement6, and exotic Hall effects7,8,9,10,11,12,13,14. One of the most important recent challenges within this field is exploring the exotic magnetic metallic state in van del Waals (vdW) coupled materials15,16,17,18,19,20,21. A missing but crucial component in this challenge is establishing various approaches to controlling the interaction between magnetism and conduction and the mechanism by which they operate.

CeTe3 provides a unique platform to investigate the itinerant magnetism in a van der Waals (vdW) coupled metal22,23,24. The crystal structure of CeTe3 features a square net of Te0.5− and rock-salt type layer of Ce3+ and Te2− (Fig. 1a), combined to realize a highly two-dimensional vdW motif21,25. The conduction band mainly originates from the Te0.5− square net; Ce3+ is responsible for magnetism. Strong nesting of the Fermi surface derived from the square net Te sheet causes a 1D-like charge density wave (CDW) transition below 400 K26,27,28,29,30, whose temperature scale is well separated from the much lower magnetic transition temperature. Based on previous studies23,31, the magnetic moment of CeTe3 can be described as a pseudo spin 1/2 system with strongly easy-plane (XY) character, and long range antiferromagnetic order along the easy-plane is observed below 1.3 K.

Recently, CeSeTe2 has been identified as a compound closely related to CeTe3. In CeSeTe2, Se2− selectively substitutes into Te2− sites in CeTe332,33. Consistent with the chemical and structural similarity of CeSeTe2 to CeTe3 (Fig. 1a), a CDW transition and strongly easy-plane- like magnetism in the paramagnetic regime are observed in both materials31. In CeSeTe2 however, the reported magnetism in the ordered state is consistent with a collinear structure pointing in the out-of-plane direction31, which is strongly unfavorable in the paramagnetic state. Thus, by defining the magnetic hard and easy axis/plane of the system based on its paramagnetic state, surprising hard-axis ordering is suggested to emerge in CeSeTe2 below the antiferromagnetic phase transition. Despite this unusual direction of magnetic moment in the ordered state, no detailed measurements have been performed so far to elucidate the underlying mechanism.

To understand the origin of this peculiar hard-axis ordering, here we studied the effect of Se doping systematically through magnetization and heat capacity measurement of Ce(SexTe1−x)Te2. We confirmed that addition of Se applies chemical pressure to Ce3+, which results in exotic reorientation of the magnetic moment through enhanced quantum fluctuation. In this report, we present that the presence of hard-axis ordering is representative of the fermionic order by disorder process, in which a seemingly unstable direction is stabilized through itinerant energy gain.

## Results

### Crystal growth of Ce(SexTe1−x)Te2

Single crystals of Ce(SexTe1−x)Te2 were grown via the flux method (See method for detail). This formula represents that Se2− selectively replaces Te2− in the magnetic blocking layer (Fig. 1a). Thus, Se2− doping connects the two isostructural systems Ce3+ (Te2−)(Te0.5−)2 and Ce3+ (Se2−) (Te0.5−)2. Since the ionic radii of Se2− and Te2− are 1.98 Å and 2.21 Å, respectively34, systematic control of chemical pressure becomes possible in Ce(SexTe1−x)Te2 through variation of the doping ratio. The single crystals of Ce(SexTe1−x)Te2 present a plate-like morphology with shiny surfaces, reflecting the vdW coupling nature in our system (Fig. 1a). Figure 1b shows the doping evolution of the X-ray diffraction (XRD) pattern around the (080) peak. Figure 1c shows the relationship of the lattice constant along the out-of-plane b axis and the Se/Te ratio x as determined by XRD and energy dispersive X-ray spectrometry (EDX), respectively. As predicted by Vegard’s law, the monotonic linear behavior of the shrinkage of lattice constant with doping suggests the systematic replacement of smaller ionic radii Se2− into the larger ionic radii Te2− sites. Similar monotonic shrinkage of the out of plane lattice constant by doping with elements with smaller ionic radii is also seen in another vdW misfit compounds such as Bi-based cuprates34. In this study, using the obtained linear function, we calculated x from the out-of-plane lattice constant of the crystal flakes used for heat capacity and magnetic measurements.

### Magnetic order in CeSeTe2 and CeTe3

We first present a significant difference in the magnetism and heat capacity behavior of CeTe3 (Fig. 2a) and CeSeTe2 (Fig. 2b), despite the common easy-plane feature in the paramagnetic regime. In Fig. 2a,b, magnetic susceptibility χ(T) and heat capacity C(T) are shown on the left and right axes, respectively. For the parent compound CeTe3, we see two characteristic antiferromagnetism related transition temperatures TN1 and TN2. Only a broad peak is observed in C(T) around TN1 while around TN2 a sharp peak in C(T) accompanies stronger suppression of χ(T) along the in-plane (blue and red curves) than the out-of-plane (greed curve) direction. On the other hand, the doped system CeSeTe2 features a single antiferromagnetic transition with a sharp peak at TN3 in C(T) that accompanies stronger suppression of χ(T) along the out-of-plane (green curve) than the in-plane (blue and red curves) directions. Therefore, at the lowest temperature, the Néel vector points along the easy plane in CeTe3, whereas it points along the hard axis in CeSeTe2. This striking contrast is also evident from the suppression of susceptibility and spin flip and flop transitions (Fig. 2c,d) since they should appear along the Néel vector. Figure 2e is the experimentally determined magnetic hard axis and easy plane together with crystal axes. These directions are the same are the same for all samples shown in this study.

### Quasi two-dimensional magnetism of CeSeTe2 and CeTe3

The ordering phenomena in Ce(SexTe1−x)Te2 based on localized spin picture are briefly discussed before we emphasize the importance of itinerancy. The magnetism of CeTe3 and CeSeTe2 is supposed to originate from highly two-dimensional interactions with XY-like and Ising-like anisotropy, respectively. In CeTe3, the successive phase transitions associated with TN1 and TN2 are reminiscent of the 2D XY model perturbed by weak in-plane anisotropy35. While two-dimensional XY-like interactions can support only quasi long-range order at a finite temperature, genuine long-range order can appear by nearly fourfold symmetric anisotropies at a lower temperature. This picture is presumably captured by the broad and sharp peaks TN1 and TN2, respectively. In contrast, Ising-type long-range-order in CeSeTe2 can appear by breaking discrete Ising symmetry; here in-plane anisotropy plays no role in the magnetic ordering and a single transition has been observed. The abrupt suppression of χ(T) also reflects on the stability of the Ising-like collinear order. It should be noted that the observed collinear magnetism suggests that the Dzyaloshinski–Moriya (DM) interaction, which favors noncollinear spin arrangement regardless of anisotropy of the g factor, should play a minor role in the emergence of hard axis magnetic order. At this stage, it remains unclear why the effective interaction becomes Ising-like by doping even though the paramagnetic susceptibility remains XY-like. As we discuss hereafter, beyond the localized magnetic picture, kinetic energy gain plays a crucial role in explaining an essential part of our experimental observation.

### Magnetic fluctuation of Ce(SexTe1−x)Te2

If the ordering direction of CeSeTe2 is actually unfavorable, we should observe self-consistent consequences of enlarged quantum fluctuation. To check this, we investigated the systematic doping dependence of the magnetic fluctuation of Ce(SexTe1−x)Te2. 4f-electron derived entropy S(T) is estimated and plotted in Fig. 3a–f (right axis) based on the formula $$S(T) = \int_{0}^{T} {\frac{{C_{f} }}{T}dT}$$. Here, Cf(T) is the heat capacity from the 4f electron contribution (See method for detail). We further calculate magnetic entropy Sm for S(TN1) and S(TN3) as x < 0.54 and x > 0.54, respectively. Based on a fully localized picture, Sm should asymptotically approach Rln2 (~ 5.76 JK−1 mol−1). This value is simply derived from the ground state doublet of Ce3+ ions under the crystalline electric field. Thus, 1 − Sm/Rln2 can be taken as the measure of the strength of magnetic fluctuation for a given magnetic order.

### Evolution of magnetic fluctuation by doping

By constructing a phase diagram, a correlation exists between enhanced magnetic fluctuation and emergent hard-axis moment. Figure 4a represents the phase diagram with three characteristic temperatures (left axis) together with 1 − Sm/Rln2 (right axis). With increasing doping x, TN1 and TN2 are systematically decreased for x < 0.54. Above x = 0.54, we see only one transition temperature TN3. Therefore, combining magnetic measurements for CeTe3 (Fig. 2a) and CeSeTe2 (Fig. 2b), a first-order magnetic transition is expected to exist near x = 0.54, across which the direction of magnetic moment abruptly changes from the easy plane to the hard axis. The enhancement of 1 − Sm/Rln2 towards the first-order transition point is partly due to the chemical disorder effect since chemical entropy naturally reaches a maximum at x = 0.5 in our alloyed system. As a consequence, chemical disorder may contribute to a slight increase in 1 − Sm/Rln2 with doping levels above x = 0.54 (Fig. 4a). However, we emphasize that the values 1 − Sm/Rln2 for two nearly stoichiometric samples with x = 0 and x = 0.96 are 0.25 and 0.36, respectively. Based on these values, the magnetic fluctuation of stoichiometric CeSeTe2 is expected to be ~ 1.4 times larger than that of the parent compound CeTe3. Therefore, the key feature to be captured in the phase diagram is an emergent hard-axis moment with a monotonic enhancement of magnetic fluctuation.

## Discussion

We propose the possible mechanism leading to emergent hard axis ordering in terms of fermionic order by disorder in itinerant systems36,37. Our observation is that the change in chemical pressure enhanced the magnetic quantum fluctuation and altered the direction of Néel vector to the apparent “hard axis”. This implies that magnetic quantum fluctuation itself drives the hard-axis ordering as in the case of “order by disorder” in frustrated magnets38. The relevant quantum fluctuation is unlikely to be Kondo screening in our case because a similar magnitude of saturated magnetic moment along the easy-axis with doping was observed experimentally (Fig. 2c,d). Instead, we propose that the enhanced magnetic fluctuation is a spin-wave excitation, which is the primary fluctuation from the magnetically ordered state. An important point to note is that the spin-wave excitation is a transverse fluctuation. Since the transverse direction from the hard axis is on the easy plane, hard-axis order can fluctuate more easily than easy-plane order. We surmise that the hard-axis order becomes favorable when chemical pressure further enhances quantum fluctuation of localized spin through Kondo coupling.

Compared to localized frustrated magnets, fermionic order by disorder requires kinetic energy gain of electrons in addition to transverse spin fluctuation (Fig. 4b). Such physics is directly captured by investigating electronic quasiparticle band deformation to lower the total energy of the system. This concept is consistent with the Fermionic order-by-disorder process, as predicted theoretically36,37. Regardless of theoretical proposals, however, experimental direct visualization of quasi-particle band deformation has been missing due to the lack of proper materials to investigate with high resolution. Compared with materials in existing studies such as YbRh2Si2, YbNi4P2, and CeT2Al10 (T = Ru, Os)39,40,41,42,43,44,45 the intrinsically cleavable nature and simple composite structure of our vdW material offers an excellent platform for unveiling rich quasiparticle band deformation using surface sensitive spectroscopies with changing external magnetic field, its orientation and temperature. In addition, inherent applicability of advanced exfoliation techniques46 will further enrich future challenges in searching for novel quantum phenomena.

In summary, we show magnetic fluctuation driven unusual magnetic rotation from the usual easy plane to unusual hard axis moment across the magnetic phase transition. As far as we recognize, this is the first report of the order-by-disorder process in fermionic system in a vdW coupled antiferromagnet, which opens various unique research directions for the future.

## Methods

### Synthesis of monocrystalline Ce(SexTe1−x)Te2

Single crystals of Ce(SexTe1−x)Te2 are grown using self-flux method. The mixture of the elemental powder of cerium (99.9%), selenium (99.999%), and tellurium (99.999%) in a molar ratio of 1:2x/3:40-2x/3 was placed inside the alumina crucible and sealed in an evacuated quartz tube. The ampoule was heated to 900 °C at a rate of 75 °C/hour, kept at 900 °C for 24 h, and then slowly cooled down to 550 °C at a rate of 2 °C/hour followed by centrifugation to remove crystals from the melt.

### Magnetization and heat capacity measurement

DC magnetization and heat capacity were measured using a Magnetic Property Measurement System (Quantum Design) with a 3He insert47 and a Physical Property Measurement System PPMS (Quantum Design) with a 3He insert, respectively.

### Estimation of magnetic heat capacity

We extracted the magnetic fluctuation strength from heat capacity C(T) (Fig. 3a–f left axis) based on the method described as follows. To obtain the 4f electron contribution Cf(T), first we subtract C(T) of LaTe3 from that of Ce(SexTe1−x)Te2. This subtraction relies on the assumption that phonon and electronic contributions not arising from the 4f electrons are the same between LaTe3 and Ce(SexTe1−x)Te2 (see supplemental information). From comparison, the specific heat is dominated by 4f electron derived signal, and phonon subtraction is a minor correction within the main purpose of this study. Using the obtained Cf(T), Here, we set Cf(0) = 0 to calculate S(T). This assumption is physically not exactly accurate since heat capacity is finite due to itinerant 4f-electron nature; It does not however have a large impact within the purpose of extracting magnetic entropy since electronic contribution relative to magnetic contribution is much lower.

## Data availability

The datasets collected and analysis performed by this study are available from the corresponding author upon reasonable request.

## References

1. Keimer, B., Kivelson, S. A., Norman, M. R., Uchida, S. & Zaanen, J. From quantum matter to high-temperature superconductivity in copper oxides. Nature 518, 179–186 (2015).

2. Stewart, G. R. Superconductivity in iron compounds. Rev. Mod. Phys. 83, 1589–1652 (2011).

3. Stewart, G. R. Heavy-fermion systems. Rev. Mod. Phys. 56, 755 (1984).

4. Coleman, P. Heavy Fermions and the Kondo Lattice: a 21st Century Perspective. arXiv:1509.05769

5. Imada, M., Fujimori, A. & Tokura, Y. Metal-insulator transitions. Rev. Mod. Phys. 70, 1039 (1998).

6. Masuda, H. et al. Quantum Hall effect in a bulk antiferromagnet EuMnBi2 with magnetically confined two-dimensional Dirac fermions. Sci. Adv. 2, 1501117 (2016).

7. Kida, T. et al. The giant anomalous Hall effect in the ferromagnet Fe3Sn2—a frustrated kagome metal. J. Phys.: Condens. Matter 23, 112205 (2011).

8. Nakatsuji, S., Kiyohara, N. & Higo, T. Large anomalous Hall effect in a non-collinear antiferromagnet at room temperature. Nature 527, 212 (2015).

9. Liu, E. et al. Giant anomalous Hall effect in a ferromagnetic kagome-lattice semimetal. Nat. Phys. 14, 1125 (2018).

10. Fang, Z. et al. The anomalous hall effect and magnetic monopoles in momentum space. Science 302, 92 (2003).

11. Soumyanarayanan, A., Reyren, N., Fert, A. & Panagopoulos, C. Emergent phenomena induced by spin–orbit coupling at surfaces and interfaces. Nature 539, 509 (2016).

12. Matsuno, J. et al. Interface-driven topological hall effect in SrRuO3–SrIrO3 bilayer. Sci. Adv. 2, e1600304 (2016).

13. Soumyanarayanan, A. et al. Tunable room temperature magnetic skyrmions in Ir/Fe/Co/Pt multilayers. Nat. Mat. 16, 898–904 (2017).

14. Nagaosa, N. & Tokura, Y. Topological properties and dynamics of magnetic skyrmions. Nat. Nanotech. 8, 899 (2013).

15. Geim, A. K. & Van der Grigorieva, I. V. Waals heterostructures. Nature 499, 419 (2013).

16. Zhang, Y. et al. Emergence of Kondo lattice behavior in a van der Waals itinerant ferromagnet, Fe3GeTe2. Sci. Adv. 4, eaao6791 (2018).

17. Fei, Z. et al. Two-dimensional itinerant ferromagnetism in atomically thin Fe3GeTe2. Nat. Mater. 17, 778–782 (2018).

18. Kim, K. et al. Large anomalous Hall current induced by topological nodal lines in a ferromagnetic van der Waals semimetal. Nat. Mater. 17, 794 (2018).

19. Deng, Y. et al. Gate-tunable room-temperature ferromagnetism in two-dimensional Fe3GeTe2. Nature 563, 94 (2018).

20. May, A. F. et al. Ferromagnetism near room temperature in the cleavable van der Waals crystal Fe5GeTe2. ACS Nano 13, 4436–4442 (2019).

21. Lei, S. et al. High mobility in a van der Waals layered antiferromagnetic metal. Sci. Adv. 6, eaay6407 (2020).

22. Iyeiri, Y., Okumura, T., Michioka, C. & Suzuki, K. Magnetic properties of rare-earth metal tritellurides RTe3 (R = Ce, Pr, Nd, Gd, Dy). Phys. Rev. B 67, 144417 (2003).

23. Deguchi, K., Okada, T., Chen, G. F., Ban, S., Aso, N. & Sato, N. K. Magnetic order of rare-earth tritelluride CeTe3 at low temperature. J. Phys.: Conf. Ser. 150, 042023 (2009).

24. Ru, N. & Fisher, I. R. Thermodynamic and transport properties of YTe3, LaTe3, and CeTe3. Phys. Rev. B 73, 033101 (2006).

25. Klemenz, S., Lei, S. & Schoop, L. M. Topological semimetals in square-net materials. Annu. Rev. Mater. Res. 49, 185–206 (2019).

26. Schmitt, F. et al. Transient electronic structure and melting of a charge density wave in TbTe3. Science 321, 1649 (2008).

27. Brouet, V. et al. Fermi surface reconstruction in the CDW state of CeTe3 observed by photoemission. Phys. Rev. Lett. 93, 126405 (2004).

28. Brouet, V. et al. ARPES study of the evolution of band structure and charge density wave properties in RTe3 for R = Y, La, Ce, Sm, Gd, Tb and Dy. Phys. Rev. B 77, 235104 (2008).

29. DiMasi, E., Aronson, M. C., Mansfield, J. F., Foran, B. & Lee, S. Chemical pressure and charge-density waves in rare-earth tritellurides. Phys. Rev. B 52, 14516 (1995).

30. Ru, N. et al. Effect of chemical pressure on the charge density wave transition in rare-earth tritellurides RTe3. Phys. Rev. B 77, 035114 (2008).

31. Wang, P. P. et al. Anisotropic transport and magnetic properties of charge-density-wave materials RSeTe2 (R = La, Ce, Pr, Nd). Chin. Phys. Lett. 32, 087101 (2015).

32. Polequin, B., Tsinde, F. & Doert, T. The ternary rare-earth polychalcogenides LaSeTe2, CeSeTe2, PrSeTe2, NdSeTe2, and SmSeTe2: syntheses, crystal structures, electronic properties, and charge-density-wave-transitions. Solid State Sci. 7, 573–587 (2005).

33. Doert, T. et al. LaSeTe2—Temperature dependent structure investigation and electron holography on a charge-density-wave-hosting compound. Chem. Eur. J. 9, 5865–5872 (2003).

34. Okada, Y., Takeuchi, T., Baba, T., Shin, S. & Ikuta, H. Origin of the anomalously strong influence of out-of-plane disorder on high-Tc superconductivity. J. Phys. Soc. Jpn. 77, 074714 (2008).

35. José, J. V., Kadanoff, L. P., Kirkpatrick, S. & Nelson, D. R. Renormalization, vortices, and symmetry-breaking perturbations in the two-dimensional planar model. Phys. Rev. B 16, 1217 (1978).

36. Krüger, F., Pedder, C. J. & Green, A. G. Fluctuation-driven magnetic hard-axis ordering in metallic ferromagnets. Phys. Rev. Lett. 113, 147001 (2014).

37. Green, A. G., Conduit, G. & Kruger, F. Quantum order-by-disorderin strongly correlated metal. Annu. Rev. Condens. Matter Phys. 9, 59–77 (2018).

38. Henley, C. L. Ordering due to disorder in a frustrated vector antiferromagnet. Phys. Rev. Lett. 62, 2056 (1989).

39. Lausberg, S. et al. Doped YbRh2Si2: not only ferromagnetic correlations but ferromagnetic order. Phys. Rev. Lett. 110, 256402 (2013).

40. Steppke, A. et al. Ferromagnetic quantum critical point in the heavy-fermion metal YbNi4(P1−xAsx). Science 339, 933 (2013).

41. Ishida, K. et al. YbRh2Si2: spin fluctuations in the vicinity of a quantum critical point at low magnetic field. Phys. Rev. Lett. 89, 107202 (2002).

42. Klingner, C. et al. Evolution of magnetism in Yb(Rh1−xCox)2Si2. Phys. Rev. B 83, 144405 (2011).

43. Andrade, E. C., Brando, M., Geibel, C. & Vojta, M. Competing orders, competing anisotropies, and multicriticality: The case of Co-doped YbRh2Si2. Phys. Rev. B 90, 075138 (2014).

44. Khalyavin, D. D. et al. Change of magnetic ground state by light electron doping in CeOs2Al10. Phys. Rev. B 88, 060403(R) (2013).

45. Bhattacharyya, A. et al. Anomalous change of the magnetic moment direction by hole doping in CeRu2Al10. Phys. Rev. B 90, 174412 (2014).

46. Watanabe, M. et al. Quantum oscillations with magnetic hysteresis observed in CeTe3 thin films. APEX (submitted)

47. Sato, Y. et al. Development of a low-temperature insert for precise magnetization measurement below T = 2 K with a superconducting quantum interference device magnetometer. Jpn. J. Appl. Phys. 52, 106702 (2013).

48. Momma, K. & Izumi, F. J. Appl. Cryst. 44, 1272–1276 (2011).

## Acknowledgements

The cell structure was visualized using VESTA48. Heat capacity measurements were carried out under the Visiting Researcher’s Program of Institute for Solid State Physics, The University of Tokyo. We thank S. Asai and S. Hasegawa for helping with heat capacity measurements.

## Author information

Authors

### Contributions

Y. O. and R. K. conceived this project. D. U, S. K., Y. F., R. K., Y. O., B. S., Y. I., W. S., T. M., and T. K. corrected experimental data. C. H., H. L., and F. C., theoretically considered electronic structure. R. O. and Y. O. interpret the data. Y. O., R. O., and B. S. wrote the manuscript. We all discussed.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

### Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Okuma, R., Ueta, D., Kuniyoshi, S. et al. Fermionic order by disorder in a van der Waals antiferromagnet. Sci Rep 10, 15311 (2020). https://doi.org/10.1038/s41598-020-72300-3

• Accepted:

• Published:

• DOI: https://doi.org/10.1038/s41598-020-72300-3

• ### Entropy-driven order in an array of nanomagnets

• Hilal Saglam
• Ayhan Duzgun
• Peter Schiffer

Nature Physics (2022)