Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Hyperbolic shear polaritons in low-symmetry crystals

## Abstract

The lattice symmetry of a crystal is one of the most important factors in determining its physical properties. Particularly, low-symmetry crystals offer powerful opportunities to control light propagation, polarization and phase1,2,3,4. Materials featuring extreme optical anisotropy can support a hyperbolic response, enabling coupled light–matter interactions, also known as polaritons, with highly directional propagation and compression of light to deeply sub-wavelength scales5. Here we show that monoclinic crystals can support hyperbolic shear polaritons, a new polariton class arising in the mid-infrared to far-infrared due to shear phenomena in the dielectric response. This feature emerges in materials in which the dielectric tensor cannot be diagonalized, that is, in low-symmetry monoclinic and triclinic crystals in which several oscillators with non-orthogonal relative orientations contribute to the optical response6,7. Hyperbolic shear polaritons complement previous observations of hyperbolic phonon polaritons in orthorhombic1,3,4 and hexagonal8,9 crystal systems, unveiling new features, such as the continuous evolution of their propagation direction with frequency, tilted wavefronts and asymmetric responses. The interplay between diagonal loss and off-diagonal shear phenomena in the dielectric response of these materials has implications for new forms of non-Hermitian and topological photonic states. We anticipate that our results will motivate new directions for polariton physics in low-symmetry materials, which include geological minerals10, many common oxides11 and organic crystals12, greatly expanding the material base and extending design opportunities for compact photonic devices.

## Main

Crystal symmetry plays a critical role in dictating the optical, electronic, mechanical and thermal properties of a material. Reduced symmetry is at the heart of numerous emergent phenomena, including structural phase transitions11, charge-density waves13 and topological physics14. The interaction of light with low-symmetry materials is particularly important, as it allows fine control over the phase, propagation direction and polarization1,2,3,4. This control can be especially pronounced for sub-diffractional surface waves, for instance, surface phonon polaritons (SPhPs)15 and surface plasmon polaritons (SPPs), supported at the surface of polar crystals and conductors, respectively. Both SPhPs and SPPs are quasiparticles comprising photons and coherently oscillating charges, that is, polar lattice vibrations or free-carrier plasmas, respectively, and they are strongly influenced by crystal symmetry. As a relevant example, low-symmetry polaritonic materials can support hyperbolic light propagation16, constituting an exotic class of light waves that are highly directional with very large momenta. Hyperbolic polaritons occur in materials in which the real part of the permittivity along at least one crystal direction is negative and positive along at least one other. This extreme anisotropy is associated with free carriers and optic phonons in anisotropic lattices. In turn, hyperbolic polaritons enable deeply sub-wavelength light confinement over broad bandwidths8,9. In polar crystals with symmetries that support a single optical axis (uniaxial), such as hexagonal boron nitride (hBN), hyperbolic polaritons (HPs) of type I or type II can arise5,8,9, for which the hyperbolic isofrequency surfaces do or do not intersect the optical axis, respectively. Materials or metamaterials exhibiting lower symmetry, in which all three major polarizability axes are different (biaxial) but orthogonal, such as alpha-phase molybdenum trioxide (α-MoO3)1,3, Li-intercalated vanadium pentoxide (V2O5)4 or nanostructured metasurfaces17, exhibit several distinct spectral regimes of hyperbolic modes propagating along different crystal axes. Notably, in-plane hyperbolicity within α-MoO3 films has been shown to be low loss1,3, with reconfigurable features2 and capable of supporting topological transitions2. More exotic polaritonic responses may be expected in crystals with further reduced symmetry, such as monoclinic and triclinic lattices.

Monoclinic crystals make up the largest crystal system, with around one-third of the minerals of Earth belonging to one of its three classes18. These low-symmetry Bravais lattices exhibit non-orthogonal principal crystal axes (Fig. 1a), in contrast to orthorhombic (for example, biaxial α-MoO3 (ref. 1)), tetragonal, hexagonal, trigonal (for example, uniaxial α-quartz, aQ, Fig. 1b) or cubic crystal systems. As a consequence, their dielectric permittivity tensor has major polarizability directions that strongly depend on the frequency, with off-diagonal terms that cannot be completely removed through coordinate rotation6,7, and exhibits shear terms analogous to viscous flow19. These features arise due to the non-trivial relative orientation (neither parallel nor orthogonal) of several optical transitions that, at a given frequency, contribute to a net polarization that cannot be aligned with the crystal axes. In turn, this property results in exotic light propagation not supported by higher-symmetry crystals6,7,20. Here we show exemplary consequences of these material features for nanophotonics, in particular, the emergence of a new form of waves — hyperbolic shear polaritons (HShPs) — which have not been previously observed.

In this work, we theoretically and experimentally demonstrate the emergence of HShPs in monoclinic crystals. As an exemplary material to demonstrate this phenomenon, we study beta-phase Ga2O3 (bGO), which has gained a large amount of research and industrial attention for its high breakdown field21 and applications in photovoltaics22, optical displays23 and gas sensors24. In the low-energy range, bGO features several strong infrared-active, non-orthogonal phonon resonances6, making the permittivity tensor of bGO naturally non-diagonalizable. Its low symmetry has two consequences on the polariton propagation when compared with more conventional hyperbolic materials with a diagonal permittivity tensor, such as hBN, aQ and α-MoO3. First, both the bGO polariton wavelength and the propagation direction strongly disperse with frequency. Second, we demonstrate that the asymmetric nature of optical loss in such crystals gives rise to shear, resulting in polariton propagation with tilted wavefronts. Such tilted wavefronts are a direct consequence of the low symmetry of the material and are one of the most notable and unique features of HShPs. New opportunities for polaritonics arise for HShPs stemming directly from their non-Hermitian and topological nature. Yet, surprisingly, they can be observed in low-loss, naturally occurring materials, without the need for artificial structuring of a material surface17.

To highlight the role of the asymmetry of monoclinic crystals in their polariton response, we compare HShPs with HPs supported by higher-symmetry anisotropic crystals, such as aQ25. In this vein, we compare the crystal structure of monoclinic bGO in Fig 1a with the trigonal crystal of aQ in Fig. 1b, illustrating the low crystal symmetry present in bGO. In general, the description of the dielectric response of monoclinic crystals requires inclusion of identical off-diagonal elements in the monoclinic plane within the frequency-dependent, complex-valued dielectric tensor.

$$\overline{\overline{\varepsilon (\omega )}}=[\begin{array}{ccc}{\varepsilon }_{xx}(\omega ) & {\varepsilon }_{xy}(\omega ) & 0\\ {\varepsilon }_{xy}(\omega ) & {\varepsilon }_{yy}(\omega ) & 0\\ 0 & 0 & {\varepsilon }_{zz}(\omega )\end{array}]$$
(1)

The coordinate systems used to define the response of bGO and aQ are sketched in Fig. 1a, b, respectively. To analyse the properties of HShPs in monoclinic materials, we first rigorously solve Maxwell’s equations (see Methods) to calculate the dispersion relation of the polaritonic modes supported by bGO and — for comparison — aQ, each at two distinct frequencies. Initially, we consider the lossless case, in which the imaginary part of each term in the dielectric tensor is neglected for both bGO and aQ. The solutions for the polariton wavevectors in both materials at two different frequencies are provided in Fig. 1c, d. For aQ, we observe two open hyperboloid surfaces — as expected for uniaxial hyperbolic materials — in which a change in frequency results in a corresponding change in wavevector, while preserving the hyperboloid orientation, that is, the direction of polariton propagation (Fig. 1d). By contrast, as we change the frequency, not only does the bGO polariton wavevector magnitude change but the direction of the hyperboloid also rotates within the monoclinic plane, as can be appreciated by examining the kz = 0 projections (Fig. 1c). This is a direct consequence of the non-trivial relative orientation of the phonon resonances supporting the hyperbolic response6, which results in polariton bands that disperse in azimuth angle as a function of frequency. This feature represents a signature of the reduced symmetry associated with HShPs supported in monoclinic crystals (and is also anticipated in triclinic crystals), in contrast to HPs observed in higher-symmetry lattices.

When we also account for natural material losses resulting from inherent phonon-scattering processes, the polariton propagation in bGO shows a reduced symmetry in comparison with hyperbolic polaritons in aQ, even at individual frequencies, as illustrated in Fig. 1e, f. In these panels, we show the results of full-wave calculations of dipole-launched surface polaritons propagating across the surface of a semi-infinite slab of bGO and y-cut aQ, in which — in both cases — natural material losses were explicitly taken into account. For in-plane hyperbolic materials, these surface waves show a hyperbolic dispersion within the surface plane and are referred to as hyperbolic surface or hyperbolic Dyakonov polaritons26,27, constituting a subset of HPs supported in these materials similar to volume-confined HPs in thin films. For aQ, HPs spread out along one crystal axis of the surface and are symmetric with respect to the crystal axes, as can be confirmed by a Fourier transform of the real-space electric field profile (Fig. 1h). However, for bGO (Fig. 1e), we observe that HPs are rotated with respect to the coordinate system of the monoclinic plane, as anticipated by the isofrequency contours (Fig. 1c). In addition, the wavefronts are tilted with respect to the major propagation direction, with no apparent mirror symmetry. This feature can also be clearly seen by examining the Fourier transform of the real-space profile (Fig. 1g), exhibiting a stronger intensity along one side of the hyperbola. These observations constitute the discovery of HShPs in low-symmetry crystals.

To experimentally demonstrate the effects of reduced symmetry in polariton propagation in bGO in contrast to higher-symmetry materials, we compare the azimuthal dispersion of HShPs in bGO to the one of HPs in aQ using an Otto-type prism-coupling geometry28,29 (sketched in Fig. 2a; for details, see Methods). The experimental azimuthal dispersion of HPs on the surface of aQ is shown in Fig. 2b (see also Extended Data Fig. 3), in excellent agreement with the corresponding simulations (Fig. 2c). The dips in the reflectance spectra show the polariton resonances, which are only observable along specific azimuth angles and are symmetric about the crystal axes, ϕ = 0° (180°) and 90°. By contrast, the experimental azimuthal dispersion of HShPs on monoclinic bGO (Fig. 2d) exhibits no mirror symmetry, again in excellent agreement with the simulated dispersion (Fig. 2e).

To experimentally access the in-plane hyperbolic dispersion of the HShP in bGO observed in Fig. 1e, we mapped out the frequency–momentum dispersion in close spectral proximity of that mode (680–720 cm−1) at many azimuth angles (see Extended Data Fig. 4). The resulting map of polariton resonance frequencies is shown in Fig. 2f, in excellent agreement with the simulated resonance frequencies shown in Fig. 2g. These data allow extraction of single-frequency in-plane dispersion curves shown in Fig. 2h, i from experiment and simulations, respectively, for several selected frequencies, clearly demonstrating a hyperbolic dispersion, in excellent agreement with Fig. 1e. Notably, the base of the hyperbola shifts continuously with frequency, as marked by the symmetry axes for each curve in Fig. 2i, which directly leads to an asymmetric distribution of the group velocity along the hyperbolic dispersion curve, as shown in Fig. 2j (see also Extended Data Fig. 7).

The reduced symmetry observed in the polaritonic dispersion of bGO (Fig. 2d) is a direct consequence of the lack of symmetry in its vibrational structure6. Therefore, the HShPs are not propagating along fixed axes but show a continuous rotation of the HShP propagation direction as the frequency is varied. To describe the nature of this rotation, we diagonalize the real part of the permittivity tensor of bGO $${\rm{Re}}[\overline{\overline{\varepsilon (\omega )}}]$$ individually at each frequency, by rotating the monoclinic plane by the frequency-dependent angle.

$$\gamma (\omega )=\frac{1}{2}\arctan \left(\frac{2{\rm{Re}}({\varepsilon }_{xy})(\omega )}{\,{\rm{Re}}({\varepsilon }_{xx})(\omega )-{\rm{Re}}({\varepsilon }_{yy})(\omega )}\right)$$
(2)

The dispersion of γ(ω) is shown in Fig. 2e (white lines), illustrating that the major polarizability directions within the monoclinic plane, denoted as m and n, vary widely across the range. This frequency-dependent coordinate system enables an easier understanding and classification of the polaritonic response (see Methods and Extended Data Fig. 1 for details). The rotated coordinate axes are shown in Fig. 1e, g (see also Extended Data Fig. 2 for further modes), illustrating their alignment with the hyperbolic dispersion.

Although equation (2) describes the frequency variation of the polariton propagation direction, it does not capture the tilted wavefronts observed in Fig. 1e. This is because, as we choose the rotated coordinate system [mnz], we still retain a purely imaginary off-diagonal permittivity component (see Extended Data Fig. 1). These terms are associated with the non-orthogonal relative orientation of the material resonances, coupling the two crystal axes in the monoclinic plane. As a result, even in the rotated coordinate system [mnz], the dielectric tensor has off-diagonal terms associated with shear phenomena.

To selectively analyse the role of these shear terms, we simulate the polariton propagation in the rotated coordinate system [mnz] at 718 cm−1. In particular, we include a scaling factor for the magnitude of the off-diagonal imaginary component, indicated as ﻿i × fIm(εmn), with ﻿f = 0, 0.5 and 1 (shown in Fig. 3a–c), while retaining the diagonal loss terms. When we remove the off-diagonal component (f = 0), bGO essentially becomes a shear-free biaxial material, akin to α-MoO3 and similar to uniaxial aQ, with polaritons propagating along the optical axes (Fig. 3a). Therefore, whereas polariton propagation in such a fictional form of bGO is anisotropic in specific spectral ranges (similar to polaritons in MoO3 (refs. 1,3)), mode propagation without shear phenomena is symmetric about the (frequency-dependent) major polarizability axes (Fig. 3a). As we gradually increase the magnitude of the off-diagonal imaginary terms back to its natural value (f = 1), the wavefronts become increasingly skewed from the major polarizability axis (Fig. 3b, c). The respective reciprocal space maps in Fig. 3d–f show a strong symmetry breaking in the intensity distribution within the hyperbolic isofrequency curves. This observation provides further evidence that the propagation of polaritons is non-trivial within low-symmetry monoclinic — and, by extension, triclinic — systems, and it cannot be expected in higher-symmetry materials in which polariton propagation patterns are symmetric about the principal crystal axes1,3,4.

To connect the reduced symmetry of the surface subset of HShPs observed here experimentally (Fig. 2) and through our simulations (Fig. 3a–c) to the more general HShPs in the bulk, we now calculate isofrequency surfaces for polariton modes in bGO explicitly including loss, to account for the effect of shear phenomena. To this end, we solve Maxwell’s equations for real momentum values, yielding complex frequency eigenvalues, whose imaginary part accounts for the finite lifetime of the supported modes (see Methods for details). The results of these calculations are shown in Fig. 3g–i. Here the real part of the eigenfrequency is fixed and we find its imaginary part ωi (Fig. 3h), which is proportional to the modal lifetime, and the corresponding value of kz (Fig. 3g) for each pair of km and kn. The calculations are performed in the rotated coordinate system for both f = 0 and f = 1, showing that both the shape of the isofrequency surfaces as well as their lifetimes change greatly with the inclusion of the off-diagonal imaginary components. Notably, these calculations prove that, at individual frequencies and in the major polarizability frame, mirror symmetry of polariton propagation is lost in monoclinic materials as a direct consequence of shear.

To relate the isofrequency contours of HShPs to the surface mode dispersions in Fig. 3d–f, we plot the kz = 0 solution in Fig. 3i, with the colour scale indicating the relative loss ωi of the mode. Two important observations can be made: first, the mirror symmetry of the isofrequency curves is broken for f > 0 and it requires higher-order terms to account for the asymmetric shape; second, the mode losses are redistributed asymmetrically, with losses decreasing in one arm of the hyperbolae but increasing on the other arm. We note that, also in the experimental data (Extended Data Fig. 4), we observe an indication of asymmetric distribution of polariton quality factors along the hyperbolic dispersion curves (see Extended Data Fig. 5).

These observations naturally link HShPs in monoclinic crystals to the rich, emerging area of non-Hermitian and topological photonics. Although loss in orthogonal systems alone can already have interesting consequences for polariton propagation30, the off-diagonal shear terms highlighted here can provide new opportunities for non-Hermitian photonics and for manipulation of topological polaritons in low-symmetry materials. For instance, we foresee asymmetric topological transitions experienced by HShPs, generalizing previous results in orthorhombic systems2 by exploiting the unique non-Hermitian features emerging in low-symmetry materials. In addition, recent studies suggest the connection between Dyakonov surface waves and surface states emerging from one-dimensional band degeneracy (nodal lines) of topological nature of high-symmetry metacrystals31. We anticipate that HShPs may generalize these opportunities to asymmetric topological bands in which non-Hermiticity in the natural materials plays a dominant role.

Here we have demonstrated that low-symmetry crystals can support a new class of hyperbolic polariton modes with broken symmetry due to shear phenomena, which we refer to as HShPs. We introduce bGO as an exemplary material to enable the observation of these phenomena and experimentally demonstrate the symmetry-broken dispersion of the supported surface waves. The non-diagonalizable dielectric permittivity plays a key role in the unique properties of low-symmetry crystals, including monoclinic and triclinic lattices. Our findings are generalizable to engineered photonic systems with at least two non-orthogonal oscillators, including new metasurface designs capturing these physics. Beyond the results provided here for intrinsic, compensation-doped bGO, the presence of free charge carriers in bGO32 may allow for methods for direct steering of the HShP propagation direction (see Extended Data Fig. 6). Finally, exfoliation of thin flakes of single-crystal bGO has also been recently reported33, which will allow to make use of volume-confined HShPs in such bGO thin films or — potentially — even in monolayers34. We anticipate that HShPs may have important implications in the manipulation of phase and directional energy transfer, including radiative heat transport35, ultra-fast asymmetric thermal dissipation in the near field35 and gate-tunability for on-chip all-optical circuitry36. Beyond advances in nanophotonics, infrared polariton propagation has been demonstrated as a means for quantifying crystal strain37, polytypes38, variations in free-carrier density, as well as phononic and electronic properties around defects39, thereby also promising a new metrology tool for characterizing low-symmetry ultra-wide-bandgap semiconductors. We highlight that our results are applicable to any material with non-orthogonal optically active transitions and may therefore be extended to other optical phenomena, such as excitons in triclinic ReSe2 (ref. 40).

## Methods

### Experimental

The insulating (010)-oriented, 5 × 5 × 0.5-mm3 bGO substrate was produced by means of Fe compensation doping and was purchased from Novel Crystal Technology, Inc., Japan. The aQ sample was purchased from MaTeck GmbH, Germany. The absolute azimuth orientation of the samples was extracted from a global fit for each of the datasets of aQ and bGO (plotted in Fig. 2b,d, respectively), resulting in a rotation with respect to the principal x axis of the laboratory coordinate system of ΔϕbGO = 27.95° and ΔϕaQ = 26.96°. The aQ data have been rotated accordingly to shift the crystal axes (mirror planes) onto multiples of 90°. On the other hand, the bGO data are plotted as measured, as there is no principal azimuth angle for alignment because of the broken mirror symmetry. Here the simulation was rotated accordingly to match the data.

The Otto-type prism-coupling experiment measures the spectral dependence of surface waves through sharp absorption peaks observed as dips in the reflectance spectra, by using a prism placed near the material surface28,41. The crystals are oriented such that the monoclinic plane (bGO) and the optical axis (aQ) are parallel to the sample surface. By following the spectral position of the polariton resonances as a function of azimuth angle, we investigate the dispersion of hyperbolic waves at the surface for both bGO and aQ. The Otto geometry effectively selects a specific in-plane momentum component of those surface waves induced by the dipole excitation in Fig. 1e, f, as set by the incidence angle θ and azimuth angle ϕ that define the magnitude28,29 and direction of the selected momentum, respectively.

As an excitation source for the Otto-type prism-coupled experiments, we use a mildly focused mid-infrared free-electron laser (FEL) (spot size ~1 mm2) with small bandwidth (~0.3%) and wide tunability of 3–50 μm, covering the spectral range 350–800 cm−1, in which aQ and bGO support polaritonic modes (details on the FEL have been reported elsewhere42). Although the frequency is scanned by tuning the FEL, different in-plane momenta can be accessed by means of changes in the incidence angle θ by rotating the entire Otto geometry (details on the setup have been reported elsewhere28,43). For the experiments shown in Fig. 2b, d, the incident angle was fixed to 28°, resulting in an in-plane momentum of $${k}_{\parallel }/{k}_{0}\approx 1.10$$ (at 500 cm−1). In contrast to alternative approaches, the Otto geometry features experimental control over the excitation efficiency through tunability of the air gap width dgap. Here the gap was adjusted to a separation in which all excited modes could be observed in the spectra simultaneously, that is, dgap ≈ 8.3 μm for bGO and dgap ≈ 14.4 μm for aQ. Direct readout of dgap with a range of 1–50 μm is realized through white-light interferometry, whereas the contrast of the interference range grants parallel alignment of prism and sample43.

Mapping of the in-plane hyperbolic dispersion (Fig. 2f, h) was performed analogously to the Otto reflectance measurements shown in Fig. 2b, d. However, here we additionally varied the incidence angle θ to map out the frequency–momentum dispersion at each azimuth angle. Reflectance spectra were taken in a narrow frequency range of 670–730 cm−1, at θ = 26°, 28°, 30°, 32° and 34°, corresponding to in-plane momenta of $${k}_{\parallel }/{k}_{0}\approx 1.03,1.11,1.18,1.25\,{\rm{and}}\,1.32$$ (at 700 cm−1), at nine azimuth angles. To allow prism coupling to the polaritons also for larger momenta, these data were taken at a constant air gap dgap ≈ 4.0 μm. The reflectance minima marking the polariton resonance were extracted from these data and are shown in Fig. 2f. The theoretical polariton resonance map, Fig. 2g, was calculated using a transfer matrix formalism44, by extracting the peak positions of Im(rpp) of the air–bGO interface. To extract the single-frequency in-plane hyperbolic dispersion curves (Fig. 2h, i), we interpolated the momentum for a given frequency in the frequency–momentum dispersion for each measured/calculated azimuth angle.

### Theoretical

#### Transfer matrix

The calculations of the optical response shown in Fig. 2c, e and the polariton resonance map in Fig. 2g, as well as the dispersion maps in Extended Data Fig. 2c, d, were performed using a generalized 4 × 4 transfer matrix formalism44. In short, the formalism enables the calculation of reflection and transmission coefficients in any number of stratified media with arbitrary dielectric tensor, which enables to account for the anisotropy of our samples.

#### COMSOL simulations

COMSOL45 version 5.6 was used for simulating point dipole excitation of HShPs on bGO. A point dipole was placed 100 nm above the surface of an infinite slab of bGO, with a dielectric permittivity matching that of ref. 6. The dielectric function of aQ was taken from ref. 46. Perfectly matched impedance boundary conditions were used on the sides of the simulation, which — in principle — absorb all radiation. However, to account for the imperfect behaviour of the boundaries, we ensured that the bGO slab was sufficiently large (250 × 250 × 8 μm), such that the wave is sufficiently damped when it reaches the boundary so as not to influence the results. Reciprocal space maps were generated by 2D Fourier transformation of the real-space electric field profiles.

#### Isofrequency surface with complex frequency

To obtain the isofrequency contour surface of the bulk wave when losses are considered in the materials, we turn to the complex-frequency method and solve the source-free Maxwell equations as follows:

$$(\begin{array}{ccc}{\tilde{\omega }}^{2}{\varepsilon }_{xx}-({\tilde{k}}_{z}^{2}+{\tilde{k}}_{y}^{2}) & {\tilde{\omega }}^{2}{\varepsilon }_{xy}+{\tilde{k}}_{x}{\tilde{k}}_{y} & {\tilde{k}}_{x}{\tilde{k}}_{z}\\ {\tilde{\omega }}^{2}{\varepsilon }_{xy}+{\tilde{k}}_{x}{\tilde{k}}_{y} & {\tilde{\omega }}^{2}{\varepsilon }_{yy}-({\tilde{k}}_{z}^{2}+{\tilde{k}}_{x}^{2}) & {\tilde{k}}_{y}{\tilde{k}}_{z}\\ {\tilde{k}}_{x}{\tilde{k}}_{z} & {\tilde{k}}_{y}{\tilde{k}}_{z} & {\tilde{\omega }}^{2}{\varepsilon }_{zz}-({\tilde{k}}_{y}^{2}+{\tilde{k}}_{x}^{2})\end{array})(\begin{array}{c}{E}_{x}\\ {E}_{y}\\ {E}_{z}\end{array})=0,$$

in which $${\varepsilon }_{nn}={\varepsilon }_{nn}^{r}+{\rm{i}}{\varepsilon }_{nn}^{k},\,nn=xx,yy,zz$$ and $${\varepsilon }_{xy}={\varepsilon }_{xy}^{r}+{\rm{i}}{\varepsilon }_{xy}^{k}$$. We fix the real component k0 of the complex frequency ω and normalize it as $$\tilde{\omega }=\frac{\omega }{{k}_{0}}=1+{\rm{i}}{\tilde{\omega }}_{{\rm{i}}},$$$$\,{\tilde{\omega }}^{2}=1-{\tilde{\omega }}_{{\rm{i}}}^{2}+2{\rm{i}}{\tilde{\omega }}_{{\rm{i}}},$$ in which ωi is an effective inverse mode lifetime, calculated neglecting the effect of the complex frequency on the material dispersion, and normalize the wavevector as $${\tilde{k}}_{x,y,z}=\frac{{k}_{x,y,z}}{{k}_{0}}$$. Note that we choose a negative sign for the time-dependent term e−iωt, so $${\tilde{\omega }}_{{\rm{i}}}$$ must be real and negative to reflect the decaying nature of the wave. The analytic expression $$F({\tilde{\omega }}_{{\rm{i}}},{\tilde{k}}_{x},{\tilde{k}}_{y},{\tilde{k}}_{z})=0$$ is found by means of the secular equation of the above matrix, and two equations are obtained by separating the real and imaginary components of F, namely,

$$\begin{array}{c}{F}_{{\rm{r}}}({\tilde{\omega }}_{{\rm{i}}},{\tilde{k}}_{x},{\tilde{k}}_{y},{\tilde{k}}_{z})=0,\\ {F}_{{\rm{i}}}({\tilde{\omega }}_{{\rm{i}}},{\tilde{k}}_{x},{\tilde{k}}_{y},{\tilde{k}}_{z})=0.\end{array}$$

The isofrequency contour of the bulk wave and the imaginary component $${\tilde{\omega }}_{{\rm{i}}}$$ are evaluated from those two equations. The numerical examples at 718 cm−1 are given in Fig. 3g, h on the basis of the above method (kx,y → km,n). Notice that, in this procedure, we use the permittivity tensor elements calculated at Re(ω). When $${\tilde{k}}_{z}=0$$, the analytic expression for the isofrequency contour of the bulk wave can be written as

$$[{\tilde{\omega }}^{2}({\varepsilon }_{xx}{\varepsilon }_{yy}-{\varepsilon }_{xy}^{2})-({\varepsilon }_{xx}{\tilde{k}}_{x}^{2}+{\varepsilon }_{yy}{\tilde{k}}_{y}^{2}+2{\varepsilon }_{xy}{\tilde{k}}_{x}{\tilde{k}}_{y})]=0,$$

which turns into two equations by separating the real and imaginary parts,

$$\begin{array}{c}(1-{\tilde{\omega }}_{{\rm{i}}}^{2}){F}_{{\rm{r}}}-2{\tilde{\omega }}_{{\rm{i}}}{F}_{k}-({\varepsilon }_{xx}^{r}{\tilde{k}}_{x}^{2}+{\varepsilon }_{yy}^{r}{\tilde{k}}_{y}^{2}+2{\varepsilon }_{xy}^{r}{\tilde{k}}_{x}{\tilde{k}}_{y})=0,\\ 2{\tilde{\omega }}_{{\rm{i}}}{F}_{{\rm{r}}}+(1-{\tilde{\omega }}_{{\rm{i}}}^{2}){F}_{k}-({\varepsilon }_{xx}^{k}{\tilde{k}}_{x}^{2}+{\varepsilon }_{yy}^{k}{\tilde{k}}_{y}^{2}+2{\varepsilon }_{xy}^{k}{\tilde{k}}_{x}{\tilde{k}}_{y})=0;\end{array}$$
$$\begin{array}{c}{F}_{r}={\varepsilon }_{xx}^{r}{\varepsilon }_{yy}^{r}-{\varepsilon }_{xx}^{k}{\varepsilon }_{yy}^{k}-({\varepsilon }_{xy}^{r}{}^{2}-{\varepsilon }_{xy}^{k}{}^{2})\\ {F}_{k}={\varepsilon }_{xx}^{r}{\varepsilon }_{yy}^{k}+{\varepsilon }_{xx}^{k}{\varepsilon }_{yy}^{r}-2{\varepsilon }_{xy}^{r}{\varepsilon }_{xy}^{k}.\end{array}$$

and they can be further simplified into the following equation

$$\begin{array}{l}[({\varepsilon }_{xx}^{r}{F}_{k}-{\varepsilon }_{xx}^{k}{F}_{r}){\tilde{k}}_{x}^{2}+2({\varepsilon }_{xy}^{r}{F}_{k}-{\varepsilon }_{xy}^{k}{F}_{{\rm{r}}}){\tilde{k}}_{x}{\tilde{k}}_{y}+({\varepsilon }_{yy}^{r}{F}_{k}-{\varepsilon }_{yy}^{k}{F}_{{\rm{r}}}){\tilde{k}}_{y}^{2}{]}^{2}\\ +4({F}_{{\rm{r}}}^{2}+{F}_{k}^{2})[({\varepsilon }_{xx}^{r}{F}_{{\rm{r}}}+{\varepsilon }_{xx}^{k}{F}_{k}){\tilde{k}}_{x}^{2}+2({\varepsilon }_{xy}^{r}{F}_{{\rm{r}}}+{\varepsilon }_{xy}^{k}{F}_{k}){\tilde{k}}_{x}{\tilde{k}}_{y}\\ +({\varepsilon }_{yy}^{r}{F}_{{\rm{r}}}+{\varepsilon }_{yy}^{k}{F}_{k}){\tilde{k}}_{y}^{2}-{F}_{{\rm{r}}}^{2}-{F}_{k}^{2}]=0.\end{array}$$

Therefore, the isofrequency curves in the $${\tilde{k}}_{z}=0$$ plane as well as the imaginary component of the bulk complex frequency are obtained from the above expression.

### Characterization of the polariton modes in bGO

#### Assignment of the polariton mode nature

First, we briefly outline how conventional polariton materials are classified. A material supports surface polaritons at frequencies for which the real part of the crystal permittivity fulfils Re(ε) < −1 (ref. 47). In uniaxial crystals, the diagonal permittivity elements can be of different sign, leading to hyperbolic behaviour in which either the real part of one element is negative and the other two are positive (type I) or two are negative and one is positive (type II)9. However, this classification of anisotropic materials relies on the off-diagonal permittivity tensor elements being zero at all frequencies for an appropriate choice of coordinate system. The lower symmetry of bGO requires the emergence of εxy ≠ 0. The dielectric permittivity elements using coordinates as indicated in Fig. 1a are shown in Extended Data Fig. 1a–e, but no coordinate system exists in which Re(εx,y) ≠ 0 at all frequencies. This, as we have demonstrated in the Otto-geometry experiments (Fig. 2d), results in a non-trivial polaritonic response with highly directional modes that propagate along frequency-dictated propagation angles in the a–c plane. Furthermore, because εxy ≠ 0, it is not straightforward to determine whether the modes are elliptical (εxx, εyy, εzz < −1) or hyperbolic in nature (type I or type II), as the propagation angle is typically not aligned with one of the principal axes.

To unambiguously describe the nature of the polariton modes, we switch to a frequency-dispersive coordinate system [mnz], in which the real part of the permittivity tensor is diagonal. This is achieved by rotating the monoclinic plane by the frequency-dependent angle γ(ω) (equation (2)). The dispersion of γ(ω) and the resulting diagonal elements of ε[mnz] are plotted in Extended Data Fig. 1f–j. This new frequency-dispersive coordinate system enables the unique assignment of the supported polariton mode nature, which we have colour-coded in Extended Data Fig. 1g–j. For bGO, we observe the full range of possible combinations of positive and negative real parts of εmm, εnn and εzz, leading to dielectric (white), elliptical (grey) and hyperbolic spectral regimes of type I (in-plane in blue, out-of-plane in red) and type II (in-plane in green, out-of-plane in yellow). By performing such a frequency-dependent rotation of the permittivity tensor, we have simplified the system into a pseudo-biaxial crystal at each frequency. However, as the dielectric tensor of a monoclinic crystal is not diagonalizable20,48,49, the in-plane, off-diagonal element of ε[mnz] retains a non-vanishing imaginary part at all frequencies, that is, Im(εmn) ≠ 0 (plotted in Extended Data Fig. 1i), giving rise to the reduced symmetry of hyperbolic shear polaritons in monoclinic crystals, as discussed in Fig. 3.

The frequency-dependent rotation of the dielectric permittivity tensor is performed in three subsequent steps. First, the in-plane permittivity tensor as shown in Extended Data Fig. 1a–c is rotated about the angle γ(ω) (equation (2)). However, γ(ω) has jumps of 90° at arbitrary frequencies, resulting in abrupt discontinuities in the real parts of εmm and εnn, for which the two curves switch values. By analysing the derivative of εmm, we extract the frequency values ωjump, in which the jumps occur and reassign the permittivity curves, respectively. At the eight in-plane TO frequencies ωTO of bGO, the permittivity curves feature a pole, which is also captured in the analysis of the derivative. Therefore, at this step, the resulting curves are smooth between the TO frequencies, but switch assignment at every ωTO. The switching of the curves at every ωTO is performed in the last step. However, near ωTO, the permittivity features a large imaginary part, which is not accounted for in the rotation angle γ(ω). This leads to poles in γ(ω) at ωTO (see Extended Data Fig. 1f), which — in turn — results in a small avoided crossing of εmm and εnn at the TO frequencies. Therefore, the reassignment of the last step results in discontinuous solutions near ωTO, which is clearly not physical. To resolve this issue, we cut out ±1 cm−1 in both εmm and εnn at all eight ωTO and smooth the curves by interpolation, resulting in the pseudo-biaxial permittivity curves as shown in Extended Data Fig. 1g–h. The rotation about γ(ω) also leads to discontinuities in the off-diagonal imaginary part Im(εmn) at the frequencies ωjump. However, because εmn=εnm, the abrupt rotation about 90° only leads to sign changes at every ωjump. The curve of Im(εmn) shown in Extended Data Fig. 1i is corrected for these sign changes.

#### Polariton behaviour in bGO in the rotated frame

To verify the polaritonic behaviour of bGO in the rotated frame, that is, the pseudo-biaxial crystal, we subsequently analyse the surface polariton dispersion in the rotated coordinate system [mnz] in Extended Data Fig. 2. For electric fields in the mz or nz planes, the analytical expression describing extraordinary surface polaritons in uniaxial crystals can be used50:

$${k}_{{\rm{polariton}}}=\frac{\omega }{c}\sqrt{\frac{{\varepsilon }_{zz}{\varepsilon }_{{\rm{in}}-{\rm{plane}}}-{\varepsilon }_{zz}}{{\varepsilon }_{zz}{\varepsilon }_{{\rm{in}}-{\rm{plane}}}-1}}\,,$$

in which εin-plane = εmm, εnn. In bGO, the solutions yield four polariton branches for each direction, m and n, respectively, plotted as red dotted lines in Extended Data Fig. 2c, d. These analytical results are in perfect agreement with the numerically obtained surface polariton dispersion using a transfer matrix formalism44. To obtain the polariton propagation properties of the system, we calculate the full electric field patterns by placing a point dipole source above the bGO surface at x = y = 0 and simulating the optical response along the bGO–air interface (z = 0) with COMSOL Multiphysics45. The real-space field profiles clearly show the rotation of the major polarizability direction as a function of frequency, demonstrated for six different modes M1–M3 and N1–N3 (See Extended Data Fig. 2e–g, k–m). Frequencies are indicated as black dash-dotted lines in Extended Data Fig. 2a–d. Mode N4 is shown in Fig. 1e, g. The field profiles align with the rotated coordinate system, with basis vectors indicated by the ‘m’ and ‘n’ crosshair in each figure.

To relate the calculated dispersion of the polariton branches to the field profiles, we calculate the momentum–k maps of these modes, as obtained by a 2D Fourier transformation of the respective electric field patterns of Extended Data Fig. 2e–g, k–m in Extended Data Fig. 2h–j, n–p, respectively. At all selected frequency positions, the electric field patterns contain a directional wave of large amplitude with low spatial frequency, as well as a wave with high spatial frequency. The observed in-plane momenta of the low-k modes follow the modal dispersion predicted in Extended Data Fig. 2c, d, along the m and n axes for modes M1–M3 and N1–N3, respectively, as indicated by the black circles in Extended Data Fig. 2h–j, n–p. According to the mode characterization provided in Extended Data Fig. 1, these modes are hyperbolic, either of type I in-plane (M1–M3 and N1) or of type II in-plane (N2, N3). For all HShP modes, field patterns and k-space maps are characterized by twofold rotational symmetry only, in agreement with the 2D plane group 2 (no mirror plane symmetries). Further, the distinct peaks in the k-space maps verify the principal polariton propagation direction, whereas the corresponding radial and azimuthal spreads are representative of their decay length and degree of directionality, respectively. Analogous to the model at 718 cm−1 discussed in the main text, the maxima of the k-space maps shown in Extended Data Fig. 2 do not lie on the major polarizability axes (most prominently for the cases in Extended Data Fig. 2n, o), owing to shear phenomena in monoclinic bGO. This is discussed further in the main text and in Fig. 3.

## Data availability

The data and data analysis scripts that support the findings of this study are available from https://doi.org/10.5281/zenodo.5613335. Additionally, the Source data are provided with this paper.

## Code availability

The code generating Fig. 1c, d and Fig. 2g, i, j can be downloaded from https://doi.org/10.5281/zenodo.5613335. The code used for Fig. 3g–i is available from Andrea Alù (aalu@gc.cuny.edu) on reasonable request.

## References

1. Ma, W. et al. In-plane anisotropic and ultra-low-loss polaritons in a natural van der Waals crystal. Nature 562, 557–562 (2018).

2. Hu, G. et al. Topological polaritons and photonic magic angles in twisted α-MoO3 bilayers. Nature 582, 209–213 (2020).

3. Zheng, Z. et al. A mid-infrared biaxial hyperbolic van der Waals crystal. Sci. Adv. 5, eaav8690 (2019).

4. Taboada-Gutiérrez, J. et al. Broad spectral tuning of ultra-low-loss polaritons in a van der Waals crystal by intercalation. Nat. Mater. 19, 964–968 (2020).

5. Low, T. et al. Polaritons in layered two-dimensional materials. Nat. Mater. 16, 182–194 (2016).

6. Schubert, M. et al. Anisotropy, phonon modes, and free charge carrier parameters in monoclinic β-gallium oxide single crystals. Phys. Rev. B 93, 125209 (2016).

7. Claus, R. Polariton dispersion and crystal optics in monoclinic materials. Phys. Status Solidi 88, 683–688 (1978).

8. Dai, S. et al. Tunable phonon polaritons in atomically thin van der Waals crystals of boron nitride. Science 343, 1125–1129 (2014).

9. Caldwell, J. D. et al. Sub-diffractional volume-confined polaritons in the natural hyperbolic material hexagonal boron nitride. Nat. Commun. 5, 5221 (2014).

10. Armbruster, T. et al. Recommended nomenclature of epidote-group minerals. Eur. J. Mineral. 18, 551–567 (2006).

11. Liu, M. et al. Symmetry breaking and geometric confinement in VO2: results from a three-dimensional infrared nano-imaging. Appl. Phys. Lett. 104, 121905 (2014).

12. Koch, E. E., Otto, A. & Kliewwer, K. L. Reflection spectroscopy on monoclinic crystals. Chem. Phys. 3, 362–369 (1974).

13. Basov, D. N., Averitt, R. D. & Hsieh, D. Towards properties on demand in quantum materials. Nat. Mater. 16, 1077–1088 (2017).

14. Shiozaki, K. & Sato, M. Topology of crystalline insulators and superconductors. Phys. Rev. B 90, 165114 (2014).

15. Caldwell, J. D. et al. Low-loss, infrared and terahertz nanophotonics using surface phonon polaritons. Nanophotonics 4, 44–68 (2015).

16. Poddubny, A., Iorsh, I., Belov, P. & Kivshar, Y. Hyperbolic metamaterials. Nat. Photon. 7, 948–957 (2013).

17. Li, P. et al. Infrared hyperbolic metasurface based on nanostructured van der Waals materials. Science 359, 892–896 (2018).

18. Peck, D. & Ostrander, A. Crystallography: the monoclinic system. Mindat.org https://www.mindat.org/article.php/2787/Crystallography%3A+The+Monoclinic+System (2020).

19. Wu, Y., Zhang, W., Wang, Y., Zou, Z. & Chen, J. Energy dissipation analysis based on velocity gradient tensor decomposition. Phys. Fluids 32, 035114 (2020).

20. Sturm, C. et al. Dipole analysis of the dielectric function of color dispersive materials: application to monoclinic Ga2O3. Phys. Rev. B 94, 035148 (2016).

21. Higashiwaki, M. & Jessen, G. H. Guest Editorial: The dawn of gallium oxide microelectronics. Appl. Phys. Lett. 112, 60401 (2018).

22. Granqvist, C. G. Handbook of Inorganic Electrochromic Materials (Elsevier, 1995).

23. Miyata, T., Nakatani, T. & Minami, T. Gallium oxide as host material for multicolor emitting phosphors. J. Lumin. 87–89, 1183–1185 (2000).

24. Réti, F., Fleischer, M., Meixner, H. & Giber, J. Effect of coadsorption of reducing gases on the conductivity of β-Ga2O3 thin films in the presence of O2. Sens. Actuators B Chem. 19, 573–577 (1994).

25. Winta, C. J., Wolf, M. & Paarmann, A. Low-temperature infrared dielectric function of hyperbolic α-quartz. Phys. Rev. B 99, 144308 (2019).

26. Jacob, Z. & Narimanov, E. E. Optical hyperspace for plasmons: Dyakonov states in metamaterials. Appl. Phys. Lett. 93, 221109 (2008).

27. Álvarez-Pérez, G., Voronin, K. V., Volkov, V. S., Alonso-González, P. & Nikitin, A. Y. Analytical approximations for the dispersion of electromagnetic modes in slabs of biaxial crystals. Phys. Rev. B 100, 235408 (2019).

28. Passler, N. C. et al. Second-harmonic generation from critically coupled surface phonon polaritons. ACS Photonics 4, 1048–1053 (2017).

29. Folland, T. G., Nordin, L., Wasserman, D. & Caldwell, J. D. Probing polaritons in the mid- to far-infrared. J. Appl. Phys. 125, 191102 (2019).

30. Antoniou, P., Suchanek, F., Varner, J. F. & Foley, J. J. Role of cavity losses on nonadiabatic couplings and dynamics in polaritonic chemistry. J. Phys. Chem. Lett. 11, 9063–9069 (2020).

31. Gao, W. et al. Experimental observation of photonic nodal line degeneracies in metacrystals. Nat. Commun. 9, 950 (2018).

32. Schubert, M. et al. Longitudinal phonon plasmon mode coupling in β-Ga2O3. Appl. Phys. Lett. 114, 102102 (2019).

33. Hwang, W. S. et al. High-voltage field effect transistors with wide-bandgap β-Ga2O3 nanomembranes. Appl. Phys. Lett. 104, 203111 (2014).

34. Rivera, N., Christensen, T. & Narang, P. Phonon polaritonics in two-dimensional materials. Nano Lett. 19, 2653–2660 (2019).

35. Salihoglu, H. et al. Energy transport by radiation in hyperbolic material comparable to conduction. Adv. Funct. Mater. 30, 1905830 (2020).

36. Maia, F. C. B. et al. Anisotropic flow control and gate modulation of hybrid phonon-polaritons. Nano Lett. 19, 708–715 (2019).

37. Gigler, A. M. et al. Nanoscale residual stress-field mapping around nanoindents in SiC by IR s-SNOM and confocal Raman microscopy. Opt. Express 17, 22351 (2009).

38. Huber, A., Ocelic, N., Taubner, T. & Hillenbrand, R. Nanoscale resolved infrared probing of crystal structure and of plasmon–phonon coupling. Nano Lett. 6, 774–778 (2006).

39. Hauer, B. et al. Exploiting phonon-resonant near-field interaction for the nanoscale investigation of extended defects. Adv. Funct. Mater. 30, 1907357 (2020).

40. Arora, A. et al. Highly anisotropic in-plane excitons in atomically thin and bulklike 1T′-ReSe2. Nano Lett. 17, 3202–3207 (2017).

41. Passler, N. C. et al. Strong coupling of epsilon-near-zero phonon polaritons in polar dielectric heterostructures. Nano Lett. 18, 4285–4292 (2018).

42. Schöllkopf, W. et al. The new IR and THz FEL facility at the Fritz Haber Institute in Berlin. Proc. SPIE 9512, 95121L (2015).

43. Pufahl, K. et al. Controlling nanoscale air-gaps for critically coupled surface polaritons by means of non-invasive white-light interferometry. Appl. Phys. Lett. 113, 161103 (2018).

44. Passler, N. C. & Paarmann, A. Generalized 4 × 4 matrix formalism for light propagation in anisotropic stratified media: study of surface phonon polaritons in polar dielectric heterostructures. J. Opt. Soc. Am. B 34, 2128–2139 (2017).

45. COMSOL Multiphysics (COMSOL, 2020).

46. Spitzer, W. G. & Kleinman, D. A. Infrared lattice bands of quartz. Phys. Rev. 121, 1324–1335 (1961).

47. Raether, H. Surface Plasmons on Smooth and Rough Surfaces and on Gratings (Springer, 1988).

48. Jellison, G. E. et al. Spectroscopic dielectric tensor of monoclinic crystals: CdWO4. Phys. Rev. B 84, 195439 (2011).

49. Mock, A. et al. Band-to-band transitions, selection rules, effective mass, and excitonic contributions in monoclinic β–Ga2O3. Phys. Rev. B 96, 245205 (2017).

50. Falge, H. J. & Otto, A. Dispersion of phonon-like surface polaritons on α-quartz observed by attenuated total reflection. Phys. Status Solidi 56, 523–534 (1973).

51. Schubert, M. et al. in Gallium Oxide: Materials Properties, Crystal Growth, and Devices (eds. Higashiwaki, M. & Fujita, S.) 501–534 (Springer, 2020).

## Acknowledgements

We thank W. Schöllkopf and S. Gewinner (FHI Berlin) for operating the infrared free-electron laser. N.C.P., G.C., M.W. and A.P. thank R. Ernstorfer (TU Berlin) and S. Mährlein (FHI Berlin) for careful reading of the manuscript. N.C.P. acknowledges support by the International Max Planck Research School for Functional Interfaces in Physics and Chemistry. G.H. acknowledges the support from A*STAR AME Young Individual Research Grants (YIRG, no. A2084c0172). X.N., G.H. and A.A. acknowledge the Office of Naval Research with grant no. N00014-19-1-2011 and the Vannevar Bush Faculty Fellowship. M.S. acknowledges National Science Foundation awards DMR 1808715, DMR 1420645 and OIA-2044049, and Air Force Office of Scientific Research awards FA9550-18-1-0360, FA9550-19-S-0003 and FA9550-21-1-0259, and the Knut and Alice Wallenberg Foundation. J.D.C. was supported by the Office of Naval Research grant no. N00014-18-12107 and J.R.M. by the National Science Foundation, Division of Materials Research grant no. 1904793. T.G.F. was supported by University of Iowa Startup Funding.

## Funding

Open access funding provided by Max Planck Society.

## Author information

Authors

### Contributions

N.C.P. and G.C. performed the experiments. N.C.P. and A.P. performed the transfer matrix simulations. N.C.P., T.G.F. and A.P. performed COMSOL simulations. X.N., G.H. and A.A. performed the analytical derivations and the calculation of isofrequency surfaces. All authors contributed to writing the manuscript. A.A., J.D.C., T.G.F. and A.P. oversaw the project. M.S. and J.D.C. initiated the research.

### Corresponding authors

Correspondence to Andrea Alù, Joshua D. Caldwell, Thomas G. Folland or Alexander Paarmann.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

## Peer review

### Peer review information

Nature thanks Raul Freitas, Prineha Narang and the other, anonymous, reviewer for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Extended data figures and tables

### Extended Data Fig. 1 Frequency-dispersive dielectric permittivity of bGO and surface polariton mode assignment.

ae, Dielectric permittivity tensor elements εxx, εyy, εxy (=εyx) and εzz, respectively, of charge carrier-free bGO at infrared frequencies6. f, Rotation angle γ (equation (2)) as a function of frequency. g, h, Diagonal elements εmm and εnn of the frequency-dispersive in-plane permittivity, featuring four distinct reststrahlen bands each. i, Non-zero imaginary part of the off-diagonal element Im(εmn). j, Unchanged out-of-plane permittivity εzz. Depending on the combination of positive or negative real parts of εmn, εnn and εzz, different types of phonon polariton are supported (colour-shaded), such as elliptical SPhPs and type I and type II, in-plane ($$\parallel$$) and out-of-plane ($$\perp$$) hyperbolic polariton modes, respectively.

### Extended Data Fig. 2 Rotating HShPs on bGO.

a, b, Rotation angle γ as a function of frequency. c, d, Dispersion of HShPs on the surface of a bulk bGO crystal calculated with the frequency-dispersive permittivity tensor along the (rotating) m axis (c) and the n axis (d), obtained using a transfer matrix method44. The four supported polaritons along each axis m and n are clearly distinguishable, in perfect agreement with the theoretically calculated polariton dispersion (red dotted lines)50. Black horizontal dash-dotted lines mark the frequencies M1–M3 and N1–N4 at which the electric field distribution is plotted. N4 is shown in Fig. 1e, g. eg, Real-space electric fields at the bGO surface at frequencies M1–M3, respectively. hj, The respective two-dimensional Fourier transformation. The fields were calculated using COMSOL Multiphysics45 (see Methods for details). km, Real-space electric fields at frequencies N1–N3, respectively. np, The respective Fourier transforms. All maps (ep) were calculated using the non-dispersive permittivity tensor (Extended Data Fig. 1a–e), thus showing rotated field patterns with different orientations, depending on the frequency. The thin black and white crosshairs indicate the principal axes of the respective frequency-dispersive coordinate system, its rotation given by γ at the corresponding frequency. Small black circles in hj and np mark the momentum value of the analytical dispersion in c and d, respectively.

### Extended Data Fig. 3 Experimental datasets for bGO and aQ at different gap sizes.

The gap size dgap in our Otto geometry setup can be tuned and monitored43, enabling control over the excitation efficiency of the polariton modes44. Datasets measured for bGO (ac) and for aQ (df) at three different gap sizes each. For smaller gaps, some modes are overcoupled and their resonance features broadened (such as the mode at 500 cm−1 in bGO), whereas for larger gaps, some modes are undercoupled and their resonance features too weak to be clearly distinguishable (in particular, the mode at 725 cm−1 in bGO). The centre gap sizes compromise between these effects. Note that the gap sizes indicated here are the values monitored with a white-light interferometry setup43. The fits performed for the datasets shown in Fig. 2, that is, the datasets shown in Extended Data Fig. 3b for bGO and Extended Data Fig. 3e for aQ, however, yielded larger gap sizes of 8.3 μm (bGO) and 10.4 μm (aQ). The offset can be attributed to non-perfect parallel alignment between prism and sample and a lateral offset between the polariton excitation site with the FEL and the white-light spot for the gap measurement.

### Extended Data Fig. 4 Mapping of the in-plane hyperbolic dispersion in bGO.

Experimental Otto reflectance spectra for various azimuthal angles (ai), each at five different incidence angles θ = 26°, 28°, 30°, 32° and 34°. All data were acquired at constant Otto air gap of dgap ≈ 4.0 μm. Further data were taken also for dgap ≈ 6.0 μm but showed the same behaviour at generally reduced polariton resonance amplitudes.

### Extended Data Fig. 5 Q-factor asymmetry along the in-plane hyperbolic dispersion of bGO.

The full width at half maximum (FWHM) of the polariton resonance dips in Extended Data Fig. 4 was extracted, indicative of the (inverse) quality factor (Q-factor) of the polariton. We used the MATLAB function ‘findpeaks’ to extract the FWHM of the resonance dips, which works robustly independent of the specific peak shapes. To account for the effect of different optical coupling efficiencies for different momenta in the Otto geometry at constant air gap28, the FWHM should only be compared at the same momentum values. Therefore, we first interpolated the azimuth angle at which the polariton crosses the experimental momenta from Fig. 2f and then interpolated the resonance width extracted for these momenta from Extended Data Fig. 4 to that same azimuth angle. This procedure was performed for each frequency at both sides of the hyperbolic curves (Fig. 2h). Finally, the Q-factor asymmetry is given as the ratio of the FWHM on the left side (smaller azimuth angles) and the right side (larger azimuth angles) of the hyperbolic dispersion for each momentum and frequency. Please note that not all hyperbolic dispersions in Fig. 2h cross all experimental momentum values, which is why the Q-factor asymmetry can only be evaluated at selected momentum values as shown for each frequency. Owing to the larger experimental uncertainty of the FWHM analysis (as compared with the peak position), we refrained from extrapolation during this procedure. Also note that the polariton dip shapes for k/k0 = 1.03 (incidence angle θ = 26°) are strongly asymmetric (see Extended Data Fig. 4), such that the quality factor analysis is not very reliable there, explaining the outlier for which the Q-factor asymmetry is <1. For all other cases, we find values clearly >1, that is, we find consistently that the Q-factor is smaller on the left side than on the right side of the hyperbolic dispersion. Also, the data show a trend of increasing asymmetry with increasing momenta, that is, further out in the dispersion. Both effects are in very good agreement with the theoretical predictions (Fig. 3).

### Extended Data Fig. 6 Active tuning of the propagation direction of HShPs in bGO.

Rotation angle γ (equation (2)) calculated as a function of doping concentration N (in cm−3), assuming a Drude contribution with anisotropic charge carrier mobility, μx = 296 cm2 V−1 s−1, μy = μz = 37 cm2 V−1 s−1. (Literature values of μ in bGO feature a large variance51. We assume here strong anisotropy of the charge carrier mobility to emphasize the rotation mechanism.) Clearly, between the TO frequencies at which the HShPs disperse, the rotation angle γ is strongly dependent on the doping concentration, enabling active tuning of the propagation direction of the supported polariton modes. Note that, for an isotropic Drude contribution, on the other hand, equation (2) predicts no rotation of the propagation direction as a function of doping concentration.

### Extended Data Fig. 7 Group velocity analysis for hyperbolic shear polaritons.

Analytical solutions of isofrequency contours of lossless bGO for kz = 0 (see Fig. 1c) are used to numerically evaluate gradients with respect to frequency along the curves. The resulting group velocity (νg) distribution is shown as arrows along the isofrequency curves. The maximum value near the base of the hyperbola for ω = 718 cm−1 is |νg| ≈ 0.038c0, in which c0 is the speed of light in vacuum. The continuous rotation of the hyperbola axis with frequency (see main text) results in a pronounced asymmetry of the group velocity distribution: the peak of |νg| is shifted off the base of the hyperbola (a) and |νg| is notably different along the two arms of the hyperbola (b). This asymmetric distribution of |νg| certainly contributes greatly to the tilted wavefronts of HShPs observed in the main text. The asymmetric distribution of |νg| around the base of the hyperbola is in excellent agreement with the results shown in Fig. 2j.

## Rights and permissions

Reprints and Permissions

Passler, N.C., Ni, X., Hu, G. et al. Hyperbolic shear polaritons in low-symmetry crystals. Nature 602, 595–600 (2022). https://doi.org/10.1038/s41586-021-04328-y

• Accepted:

• Published:

• Issue Date:

• DOI: https://doi.org/10.1038/s41586-021-04328-y

• ### Floquet metamaterials

• Shixiong Yin
• Emanuele Galiffi
• Andrea Alù

eLight (2022)