Fermionic order by disorder in a van der Waals antiferromagnet

CeTe3 is a unique platform to investigate the itinerant magnetism in a van der Waals (vdW) coupled metal. Despite chemical pressure being a promising route to boost quantum fluctuation in this system, a systematic study on the chemical pressure effect on Ce3+(4f1) states is absent. Here, we report on the successful growth of a series of Se doped single crystals of CeTe3. We found a fluctuation driven exotic magnetic rotation from the usual easy-axis ordering to an unusual hard-axis ordering. Unlike in localized magnetic systems, near-critical magnetism can increase itinerancy hand-in-hand with enhancing fluctuation of magnetism. Thus, seemingly unstable hard-axis ordering emerges through kinetic energy gain, with the self-consistent observation of enhanced magnetic fluctuation (disorder). As far as we recognize, this order-by-disorder process in fermionic system is observed for the first time within vdW materials. Our finding opens a unique experimental platform for direct visualization of the rich quasiparticle Fermi surface deformation associated with the Fermionic order-by-disorder process. Also, the search for emergent exotic phases by further tuning of quantum fluctuation is suggested as a promising future challenge.

In the search for interesting metallic states in quantum materials, controlling magnetic orientation, fluctuations, and their interaction with conduction electrons has been of utmost importance. Consequent exotic phases are rich, ranging from unconventional superconductivity [1][2][3][4] to giant magneto-transport effects, including metal-toinsulator transition 5 , magnetically driven 2D confinement 6 , and exotic Hall effects [7][8][9][10][11][12][13][14] . One of the most important recent challenges within this field is exploring the exotic magnetic metallic state in van del Waals (vdW) coupled materials [15][16][17][18][19][20][21] . A missing but crucial component in this challenge is establishing various approaches to controlling the interaction between magnetism and conduction and the mechanism by which they operate.
CeTe 3 provides a unique platform to investigate the itinerant magnetism in a van der Waals (vdW) coupled metal [22][23][24] . The crystal structure of CeTe 3 features a square net of Te 0.5− and rock-salt type layer of Ce 3+ and Te 2− (Fig. 1a), combined to realize a highly two-dimensional vdW motif 21,25 . The conduction band mainly originates from the Te 0.5− square net; Ce 3+ is responsible for magnetism. Strong nesting of the Fermi surface derived from the square net Te sheet causes a 1D-like charge density wave (CDW) transition below 400 K [26][27][28][29][30] , whose temperature scale is well separated from the much lower magnetic transition temperature. Based on previous studies 23, 31 , the magnetic moment of CeTe 3 can be described as a pseudo spin 1/2 system with strongly easy-plane (XY) character, and long range antiferromagnetic order along the easy-plane is observed below 1.3 K.
Recently, CeSeTe 2 has been identified as a compound closely related to CeTe 3 . In CeSeTe 2 , Se 2− selectively substitutes into Te 2− sites in CeTe 3 32,33 . Consistent with the chemical and structural similarity of CeSeTe 2 to CeTe 3 (Fig. 1a), a CDW transition and strongly easy-plane-like magnetism in the paramagnetic regime are observed in both materials 31 . In CeSeTe 2 however, the reported magnetism in the ordered state is consistent with a collinear structure pointing in the out-of-plane direction 31 , which is strongly unfavorable in the paramagnetic state. Thus, by defining the magnetic hard and easy axis/plane of the system based on its paramagnetic state, surprising hard-axis ordering is suggested to emerge in CeSeTe 2 below the antiferromagnetic phase transition. Despite this unusual direction of magnetic moment in the ordered state, no detailed measurements have been performed so far to elucidate the underlying mechanism. www.nature.com/scientificreports/ To understand the origin of this peculiar hard-axis ordering, here we studied the effect of Se doping systematically through magnetization and heat capacity measurement of Ce(Se x Te 1−x )Te 2 . We confirmed that addition of Se applies chemical pressure to Ce 3+ , which results in exotic reorientation of the magnetic moment through enhanced quantum fluctuation. In this report, we present that the presence of hard-axis ordering is representative of the fermionic order by disorder process, in which a seemingly unstable direction is stabilized through itinerant energy gain.

Results
crystal growth of ce(Se x te 1−x )te 2 . Single crystals of Ce(Se x Te 1−x )Te 2 were grown via the flux method (See method for detail). This formula represents that Se 2− selectively replaces Te 2− in the magnetic blocking layer (Fig. 1a). Thus, Se 2− doping connects the two isostructural systems Ce 3+ (Te 2− )(Te 0.5− ) 2 and Ce 3+ (Se 2− ) (Te 0.5− ) 2 . Since the ionic radii of Se 2− and Te 2− are 1.98 Å and 2.21 Å, respectively 34 , systematic control of chemical pressure becomes possible in Ce(Se x Te 1−x )Te 2 through variation of the doping ratio. The single crystals of Ce(Se x Te 1−x )Te 2 present a plate-like morphology with shiny surfaces, reflecting the vdW coupling nature in our system (Fig. 1a). Figure 1b shows the doping evolution of the X-ray diffraction (XRD) pattern around the (080) peak.  www.nature.com/scientificreports/ shows the relationship of the lattice constant along the out-of-plane b axis and the Se/Te ratio x as determined by XRD and energy dispersive X-ray spectrometry (EDX), respectively. As predicted by Vegard's law, the monotonic linear behavior of the shrinkage of lattice constant with doping suggests the systematic replacement of smaller ionic radii Se 2− into the larger ionic radii Te 2− sites. Similar monotonic shrinkage of the out of plane lattice constant by doping with elements with smaller ionic radii is also seen in another vdW misfit compounds such as Bi-based cuprates 34 . In this study, using the obtained linear function, we calculated x from the out-ofplane lattice constant of the crystal flakes used for heat capacity and magnetic measurements.
Magnetic order in ceSete 2 and cete 3 . We first present a significant difference in the magnetism and heat capacity behavior of CeTe 3 (Fig. 2a) and CeSeTe 2 (Fig. 2b), despite the common easy-plane feature in the paramagnetic regime. In Fig. 2a,b, magnetic susceptibility χ(T) and heat capacity C(T) are shown on the left and right axes, respectively. For the parent compound CeTe 3 , we see two characteristic antiferromagnetism related transition temperatures T N1 and T N2 . Only a broad peak is observed in C(T) around T N1 while around T N2 a sharp peak in C(T) accompanies stronger suppression of χ(T) along the in-plane (blue and red curves) than the out-ofplane (greed curve) direction. On the other hand, the doped system CeSeTe 2 features a single antiferromagnetic transition with a sharp peak at T N3 in C(T) that accompanies stronger suppression of χ(T) along the out-of-plane (green curve) than the in-plane (blue and red curves) directions. Therefore, at the lowest temperature, the Néel vector points along the easy plane in CeTe 3 , whereas it points along the hard axis in CeSeTe 2 . This striking contrast is also evident from the suppression of susceptibility and spin flip and flop transitions (Fig. 2c,d) since they should appear along the Néel vector. Figure 2e is the experimentally determined magnetic hard axis and easy plane together with crystal axes. These directions are the same are the same for all samples shown in this study.
Quasi two-dimensional magnetism of ceSete 2 and cete 3 . The ordering phenomena in Ce(Se x Te 1−x ) Te 2 based on localized spin picture are briefly discussed before we emphasize the importance of itinerancy. The magnetism of CeTe 3 and CeSeTe 2 is supposed to originate from highly two-dimensional interactions with XYlike and Ising-like anisotropy, respectively. In CeTe 3 , the successive phase transitions associated with T N1 and T N2 are reminiscent of the 2D XY model perturbed by weak in-plane anisotropy 35 . While two-dimensional XYlike interactions can support only quasi long-range order at a finite temperature, genuine long-range order can appear by nearly fourfold symmetric anisotropies at a lower temperature. This picture is presumably captured by the broad and sharp peaks T N1 and T N2 , respectively. In contrast, Ising-type long-range-order in CeSeTe 2 can appear by breaking discrete Ising symmetry; here in-plane anisotropy plays no role in the magnetic ordering and a single transition has been observed. The abrupt suppression of χ(T) also reflects on the stability of the Ising-like collinear order. It should be noted that the observed collinear magnetism suggests that the Dzyaloshinski-Moriya (DM) interaction, which favors noncollinear spin arrangement regardless of anisotropy of the g factor, should play a minor role in the emergence of hard axis magnetic order. At this stage, it remains unclear why the effective interaction becomes Ising-like by doping even though the paramagnetic susceptibility remains XY-like. As we discuss hereafter, beyond the localized magnetic picture, kinetic energy gain plays a crucial role in explaining an essential part of our experimental observation. Here, C f (T) is the heat capacity from the 4f electron contribution (See method for detail). We further calculate magnetic entropy S m for S(T N1 ) and S(T N3 ) as x < 0.54 and x > 0.54, respectively. Based on a fully localized picture, S m should asymptotically approach Rln2 (~ 5.76 JK −1 mol −1 ). This value is simply derived from the ground state doublet of Ce 3+ ions under the crystalline electric field. Thus, 1 − S m /Rln2 can be taken as the measure of the strength of magnetic fluctuation for a given magnetic order.

Evolution of magnetic fluctuation by doping. By constructing a phase diagram, a correlation exists
between enhanced magnetic fluctuation and emergent hard-axis moment. Figure 4a represents the phase diagram with three characteristic temperatures (left axis) together with 1 − S m /Rln2 (right axis). With increasing doping x, T N1 and T N2 are systematically decreased for x < 0.54. Above x = 0.54, we see only one transition temperature T N3 . Therefore, combining magnetic measurements for CeTe 3 (Fig. 2a) and CeSeTe 2 (Fig. 2b), a firstorder magnetic transition is expected to exist near x = 0.54, across which the direction of magnetic moment abruptly changes from the easy plane to the hard axis. The enhancement of 1 − S m /Rln2 towards the first-order transition point is partly due to the chemical disorder effect since chemical entropy naturally reaches a maximum at x = 0.5 in our alloyed system. As a consequence, chemical disorder may contribute to a slight increase in 1 − S m /Rln2 with doping levels above x = 0.54 (Fig. 4a). However, we emphasize that the values 1 − S m /Rln2 for two nearly stoichiometric samples with x = 0 and x = 0.96 are 0.25 and 0.36, respectively. Based on these values, the magnetic fluctuation of stoichiometric CeSeTe 2 is expected to be ~ 1.4 times larger than that of the parent compound CeTe 3 . Therefore, the key feature to be captured in the phase diagram is an emergent hard-axis moment with a monotonic enhancement of magnetic fluctuation.

Discussion
We propose the possible mechanism leading to emergent hard axis ordering in terms of fermionic order by disorder in itinerant systems 36,37 . Our observation is that the change in chemical pressure enhanced the magnetic quantum fluctuation and altered the direction of Néel vector to the apparent "hard axis". This implies Scientific RepoRtS | (2020) 10:15311 | https://doi.org/10.1038/s41598-020-72300-3 www.nature.com/scientificreports/ that magnetic quantum fluctuation itself drives the hard-axis ordering as in the case of "order by disorder" in frustrated magnets 38 . The relevant quantum fluctuation is unlikely to be Kondo screening in our case because a similar magnitude of saturated magnetic moment along the easy-axis with doping was observed experimentally (Fig. 2c,d). Instead, we propose that the enhanced magnetic fluctuation is a spin-wave excitation, which is the primary fluctuation from the magnetically ordered state. An important point to note is that the spin-wave excitation is a transverse fluctuation. Since the transverse direction from the hard axis is on the easy plane, hard-axis www.nature.com/scientificreports/ order can fluctuate more easily than easy-plane order. We surmise that the hard-axis order becomes favorable when chemical pressure further enhances quantum fluctuation of localized spin through Kondo coupling. Compared to localized frustrated magnets, fermionic order by disorder requires kinetic energy gain of electrons in addition to transverse spin fluctuation (Fig. 4b). Such physics is directly captured by investigating electronic quasiparticle band deformation to lower the total energy of the system. This concept is consistent with the Fermionic order-by-disorder process, as predicted theoretically 36,37 . Regardless of theoretical proposals, however, experimental direct visualization of quasi-particle band deformation has been missing due to the lack of proper materials to investigate with high resolution. Compared with materials in existing studies such as YbRh 2 Si 2 , YbNi 4 P 2 , and CeT 2 Al 10 (T = Ru, Os) 39-45 the intrinsically cleavable nature and simple composite structure of our vdW material offers an excellent platform for unveiling rich quasiparticle band deformation using surface sensitive spectroscopies with changing external magnetic field, its orientation and temperature. In addition, inherent applicability of advanced exfoliation techniques 46 will further enrich future challenges in searching for novel quantum phenomena.
In summary, we show magnetic fluctuation driven unusual magnetic rotation from the usual easy plane to unusual hard axis moment across the magnetic phase transition. As far as we recognize, this is the first report of the order-by-disorder process in fermionic system in a vdW coupled antiferromagnet, which opens various unique research directions for the future.  estimation of magnetic heat capacity. We extracted the magnetic fluctuation strength from heat capacity C(T) (Fig. 3a-f left axis) based on the method described as follows. To obtain the 4f electron contribution C f (T), first we subtract C(T) of LaTe 3 from that of Ce(Se x Te 1−x )Te 2 . This subtraction relies on the assumption that phonon and electronic contributions not arising from the 4f electrons are the same between LaTe 3 and Ce(Se x Te 1−x )Te 2 (see supplemental information). From comparison, the specific heat is dominated by 4f electron derived signal, and phonon subtraction is a minor correction within the main purpose of this study. Using the obtained C f (T), Here, we set C f (0) = 0 to calculate S(T). This assumption is physically not exactly accurate since heat capacity is finite due to itinerant 4f-electron nature; It does not however have a large impact within the purpose of extracting magnetic entropy since electronic contribution relative to magnetic contribution is much lower.

Data availability
The datasets collected and analysis performed by this study are available from the corresponding author upon reasonable request.