Introduction

The abundance of multivalent compounds, a malleable structure and the possibility of easy exfoliation has made molybdenum oxide MoO3−x family to be recognized early for battery applications1. More recently they attracted interest as prospective semiconductors for 2D electronics2,3 stemming from their high-κ dielectric properties and high carrier mobility4. The electronic properties of the metallic MoO 2 and semiconducting MoO3 edge members among the molybdenum oxides are well understood within band theory5, and the band gap in the latter can be manipulated with a simple O reduction procedure4. Remarkably, removing a few MoO 6 octahedra from a unit cell or adding alkali atoms goes far beyond affecting the semiconductor properties but rather produces a wide range of stoichiometric suboxides MonO3n−1, the so-called “Magneli” phases. These phases have surprisingly complex phase diagrams, with richness reminiscent of that found in cuprates and other transitional metal oxides6. The examples include, but are not limited to, the archetypal incommensurate CDW system, “blue bronze”, K0.3MoO3, superconducting Li0.9Mo6O177,8 or the hidden nesting9,10 CDW compounds η−Mo4O11 and KMo6O17. Mo8O23 stands out within the MonO3n−1 series for being a semi-metal with narrow bands, combining a small Fermi energy and low dimensionality that sets the perfect conditions for the emergence of strong correlations.

The complexity of Mo8O23 is seen already in the evolution of its properties with temperature, that cannot be ascribed to a single ordering process11. The materials’ properties are thought to be dominated by the high-temperature structural transitions that are usually associated with the onset of a CDW, but their nature is still far from understood. X-ray diffraction and neutron scattering studies12,13,14 reveal two structural transitions: the second-order-like one to an incommensurate state at \({T}_{CDW}\mathop{ < }\limits_{ \tilde {}}360\), followed by an atypical lock-in transition at TICC = 285 K (qIC = (0.195, 0.5, 0.120) → qC = (0, 0.5, 0)). However, magnetic susceptibility15 is featureless at both transitions and resistivity12 exhibits weakly insulating behaviour below 300 K, hence a gap opening is uncertain and the CDW scenario is not well established. Alternatively, the CDW origin of the observed structural distortion was challenged by the tight-binding ionic treatment of the band structure16, suggesting that the soft mode of octahedral rotation is driving the transition instead, similar to the M3 mode in perovskites. On the other hand, no noticeable softening around the structural transitions in either of the visible phonon modes were observed in the recent combined transient reflectivity spectroscopy and Raman studies11, thus questioning this scenario too.

The lower temperature properties are even less understood, as several probes suggest further changes significantly below any reported structural transitions. At around T*  150 K, the resistivity experiences a pronounced maximum12. At the same time, slightly above T*, at T 160 K, time-domain electronic relaxation dynamics shows the appearance of a relaxation bottleneck11, which was interpreted as an opening of a gap, that is unrelated to the CDW, on at least a part of a Fermi surface below this temperature. Surprisingly, the most pronounced phonon mode also softens in this temperature range – at Tm 200 K. Its higher value, Tm > T*, could indicate competing orders11 and an incipient phase transition. At even lower temperatures, T < 30 K resistivity rapidly increases again, whereupon no changes were found by either structural or optical probes. A wide variety of temperatures at which anomalies are observed by different measurement techniques make the driving energy scale hard to determine and raise further questions whether one or more transitions occur below (TICC).

To unveil the nature of the ground state and the sequence of transformations leading to it, here we explore the evolution of the electronic structure in the phase diagram of Mo8O23 using spectroscopic (STS, ARPES) and magnetotransport measurements on high-quality single crystals, and compare them with the DFT predictions. We find that despite quantitative agreement between DFT calculations and spectroscopic data on the scale of 1 eV, the low-temperature low-energy behaviour of Mo8O23 electronic spectra is that of a strongly correlated multiband material, in contrast with a narrow band semi-metal to semiconductor CDW-like transition model anticipated from DFT. This discrepancy is driven by the previously unresolved purely electronic transition at 70 K with no structural involvement. The present data allows building of the unifying picture behind the phase diagram of Mo8O23.

First, we elucidate the nature of high temperature CDW transitions. For the first time we observe the metal-insulator transition in resistivity at TCDW = 343 K, accompanied by the mean-field-like opening of the partial gap of 2Δ  100 meV in STS spectra. The unusually large, non-thermal smearing shifts the onset of carrier depletion from TCDW to below 200 K, where the latter follows an Arrhenius law with a similar-sized gap. Moreover, the smearing results in a kink in Fermi level density of states at TT*, thus negating the presence of any phase transition at this temperature. We can consistently understand such behaviour through the tendency towards a high-temperature CDW formation due to the quasi-1D bands with the strong electron-hole asymmetry. The size of the gap 2Δ ≈ 100(50) meV predicted by GGA (LDA) DFT is also consistent with the experimental observations down to 100 K. However, this simple picture breaks down at lower temperatures.

Second, we identify the emergence of the new strongly-correlated ground state at low temperatures in the abscence of any reported structural transitions. The full gap of 130 meV, seen by all our probes, gradually develops below 70 K. Below 70 K Hall concentration first decreases and then saturates. The latter, together with the unusual spectral gap shape, suggests the nonuniform ground state with nanoscale phase separation, common for interacting systems. The signatures of interactions in narrow bands below the structural transitions set the stage for electronically driven metastability.

The paper is organized as follows. We first present confirmation that the studied surfaces indeed correspond to the pristine stoichiometric Mo8O23 structure. We then move to presenting first-principle calculations of the band structure and compare them with ARPES and STS spectra at fixed temperatures. We then proceed by describing the transport and tunnelling spectroscopy studies in the full temperature range covering both high- and low-T transitions. Finally, we discuss all the data together in a self-consistent fashion in order to elucidate the unconventional behaviour of this material.

Results

Surface structure

Early STM investigations17 of Mo8O23 revealed two types of surface topographies, which were preliminary associated with the partial removal of an apical oxygen upon cleaving. While oxygen losses in the bulk are highly unlikely at the typical measurement conditions (see Methods), the surface changes might affect our results. To ensure that we study the properties of the pristine Mo8O23 surface, we start with claryfying the surface structure.

X-ray studies12,13,18 show that Mo8O23 consists of layers of MoO6 octahedra perpendicular to [010] axis, in which the Z-shaped clusters of Mo4O14 (shown in red on Fig. 1a) are neighboured by the cross-shaped Mo4O15 clusters (shown in green on Fig. 1a)16. Additionally, the O-Mo-O bonds parallel to [010] axis are alternating in each MoO6 octahedron16, such that all the Mo atoms are always shifted either upwards or downwards from octahedra equatorial plane. The corresponding pattern for the high-temperature structure from12,16 is shown in the Fig. 1a with the Mo atoms shifted upwards denoted by the large empty circles. Because of the opposite sense of Mo shift alternation in ac-plane along [001]c−axis (Fig. 1a,b), the unit cell of the high-temperature structure is (Mo8O23)2 and the space group is non-symmorphic with the c-glide symmetry between (Mo8O23) subunits16. The CDW distortion results in unit cell doubling along b-axis too as corresponding distortion and rotation of MoO6 octahedra has the opposite character in a neigbouring layers; the bilayer unit cell of Mo8O23 after lock-in transition at TICC = 285 K is shown at Fig. 1c.

Figure 1
figure 1

Structure and STM imaging of the Mo8O23 single crystal: (ac) Crystal structure of Mo8O23: (a) the high-T (010) (ac-)plane crystal structure (b) high-T bc-plane structure and (c) low-T bc-plane structure (based on crystallographic data from refs12,13). Blue parallelogram indicates the unit cell, empty circles show the Mo atoms that are shifted upwards from the equatorial oxygen plane of MoO6 octahedron (that is discussed also in refs13,16). (d) Pseudo-topographic STM image (Vtip = +1.3 V; I = 100 pA) of (010) ac− plane at 4.2 K. Thick white parallelogram shows the experimental unit cell. Green and red squares represent the assignment of MoO6 octahedra positions. Only those with Mo atoms shifted upwards (empty circles) from equatorial plane13,16 appear to be visible.

Pseudo-topographic STM imaging (Vtip = +1.3 V and I = 100 pA, T = 4.2 K) of the (010) plane reveals atomically flat surface with highly regular structure and the nm-sized unit cell (Fig. 1d). The STM image appearance depends strongly on the bias, but stays the same in the (0.6, 1.5) V range, suggesting that it is the closest to a topographic one. The images taken at these biases remain the same also at elevated temperatures, at least up to 200 K. The Fourier transform readily reveals the unit cell with the sizes very close to those seen by XRD. The surface structure assignment for this contrast is straightforward: only the octahedra with Mo atoms shifted upwards are seen in the image. They are shown with empty circles in Fig. 1a–c. Unlike the previous studies17, we observe the same surface structure in all the samples tested (15) on the scale of 100 nm, thus ensuring that in ARPES and STS measurements we study the pristine Mo8O23.

DFT electronic structure and comparison to STS and ARPES

The generalized gradient approximation (GGA) DOS and bands for the high-temperature structure is shown in Fig. 2a. The overall DOS shown at Fig. 2c is very similar to MoO3, with filled oxygen bands situated about 2 eV below the empty molybdenum bands. The main difference is the occurrence of the almost split-off lowest two molybdenum bands in the [−1 eV, 0 eV] range. These two bands are (almost) fully occupied by the four electrons (note that the unit cell of Mo8O23 actually contains two Mo8O23 units, so the electronic states are in comparison with semiconducting MoO3 filled with 4 more electrons). These bands demonstrate the characteristic Dirac crossings at Z point due to non-symmorphic structure. The crossings can be also obtained from the group-theoretical treatment (see SI). The local-density approximation (LDA) band-structure (see Fig. S1) is very similar, with the exception that valence bands and conduction bands are closer. The corresponding LDA Fermi surface sheets are displayed in Fig. 4a with quasi-1D hole pockets near top and bottom of the Brillouin zone and ellipsoidal electron pockets in at ky = 0.

Figure 2
figure 2

Comparison of calculated and measured band structures: (a) Calculated band structure for the high-temperature atomic structure (T > TCDW). Γ denotes the Brillouin zone centre, whereas X = (0.5, 0, 0), Y = (0, 0.5, 0) and Z = (0,0,0.5) in the (ka = a*, kb = b*, kc = c*) basis. Different colours correspond to a different bands. Right panel shows the comparison of DFT GGA density of states with the dI/dV tunnelling spectrum at T = 403 K > TICDW (setpoint: 1.3 V/1 nA). (b) Calculated band structure for the low-temperature atomic structure (T < TICC). Right panel shows the comparison of DFT GGA density of states with the dI/dV tunnelling spectrum at T = 4.2 K (setpoint: 1.3 V/1 nA). (c) Comparison of the low-T and high-T calculated density of states in the wide energy range. (d) Γ−Z band structure cut measured with ARPES at T = 20 K.

The GGA results for the low-temperature structure are shown in Fig. 2b. The unit cell is doubled in the b direction, and hence one observes pairs of bands mostly weakly influenced by the low temperature distortion with respect to the high temperature structure result. The main influence of the distortions is to fully open a (direct) gap that however remains small, with the bottom/top of the bands at X point at ±25 meV with respect to the chemical potential at T = 0.

Prior tight-binding band structure calculations16 agree well with the present GGA results: the former have already captured qualitatively the gross features of Mo8O23 such as quasi-one-dimensional bands below the Fermi level and the semimetalic character of room-temperature properties. The GGA results allow now the quantitative assessment and direct comparison to the ARPES and STS data.

Tunnelling DOS in the ±1 eV range is roughly consistent with the DOS found in the GGA calculations. We compare the spectra obtained at T = 4.2 K and at T = 400 K > TCDW above CDW transition, corresponding to the commensurate and undistorted crystal structures respectively12,13. The DOS is strongly asymmetric with much larger weight above EF in the whole temperature range, in accord with the calculations. A notable discrepancy occurs at E = −0.7 eV where the GGA calculations predict a suppression and (for the low-temperature structure) a small gap (see Fig. S2).

The ARPES results reveal quasi-1D valence states dispersing along the c direction, as expected from the crystal structure, and in agreement with the DFT calculations. Figure 2d is an energy-wave vector ARPES intensity map measured at T = 20 K along the \(\overline{\Gamma }-\overline{Z}\) direction of the surface Brillouin zone. The measured dispersions are in overall agreement with the calculated band structure. The main hole-like band feature disperses linearly upwards from the BZ boundary to a maximum at −0.1 eV at the \(\overline{\Gamma }\) point. A second weaker electron-like band with a minimum at −1.1 eV at \(\overline{\Gamma }\) gains intensity after crossing the zone boundary. Neither of these spectral features is as sharp as expected for a simple quasiparticle band. The anomalous energy and momentum width is consistent with the band splitting predicted for the low-temperature phase in Fig. 2b, but the underlying bands cannot be clearly resolved. This suggests that each band is further broadened by electronic correlations and/or electron-phonon interactions. The ARPES data do not provide evidence for the predicted gap at the \(\overline{Z}\) zone boundary around −0.7 eV and, in this respect, are consistent with the DOS measured by STS.

The quasi-1D character of the bands is well illustrated by the ARPES constant energy maps in Fig. 3. The map in Fig. 3a, measured at the valence band maximum, shows parallel sheets aligned along \(\overline{\Gamma }-\overline{X}\), i.e. perpendicular to the c axis (the chain direction). These constant energy contours are split at larger binding energy, as shown in Fig. 3c, corresponding to the double intersections with the hole-like band on both sides of \(\overline{\Gamma }\). The parallel sheets are not perfectly straight. They exhibit wiggles with the periodicity of the BZ, which reflect the small inter-chain coupling. These wiggles are more clearly visible at 20 K (Fig. 3b) where the contours are sharper. Similar wiggles are are also found in LDA equienergy surfaces.

Figure 3
figure 3

ARPES data: (a,b) Constant energy maps extracted at the top of the valence band EV at T = 80 K and at T = 20 K, respectively. The polygonal contour indicates the surface Brillouin zone. (c) Constant energy map taken 450 meV below EV at 80 K. (d) ARPES spectra measured at the Γ point at T = 20 K and at T = 80 K.

Figure 4
figure 4

(a) Fermi surface for the high-T structure (LDA). Red — electron pockets, blue — hole pockets. (b) equienergy surface evaluated at energy 50 meV (c) and at energy 200 meV below the top of the band. (d) Lindhard susceptibility in the kx,0,kz plane (e) Lindhard susceptibility in the kx, π/b, kz plane, evaluated from the contributions of the two bands closest to the chemical potential at T = 350 K, retaining just band-diagonal contributions. Red dots are qICDW and -qICDW and black dot is qCCDW according to12.

Figure 5
figure 5

Transport, Hall effect and magnetoresistance measurements in Mo8O23 single crystal. (a) Temperature dependencies of the in-plane resistivity components ρxx along c and a crystal axes. Arrows indicates the temperature T* ≈ 127 K of the resistivity peak. No apparent feature is found at the transition to the commensurate state, TICC = 285 K. (b) The high-temperature part of ρa resistivity component on heating and cooling. No history dependence or hysteresis is seen beyond the small deviations due to contacts degradation above 335 K. The change of slope /dT from metallic to insulating indicates the CDW onset at TCDW ≈ 343 K. (c) Temperature dependencies of Hall coefficient (black circles, left axis) and coefficient α in magnetoresistance (red squares, right axis). (d) Temperature dependencies of the Hall density (black circles, left axis) and Hall mobility (blue triangles, right axis) (e) Normalized magnetoresistance \(\frac{\Delta R}{R}=\frac{{R}_{xx}(B)-{R}_{xx}(B=\mathrm{0)}}{{R}_{xx}(B=\mathrm{0)}}\) curves for various temperatures (left axis). Dotted curves show Hall resistance at corresponding temperatures (right axis). (f) Arrhenius plot of Hall concentration (blue dots) and the two-band toy model (line). The model is given by nHN0exp(−Δ(T)/kT) + N1 where N0 and N1 correspond respectively to the DOS for the group b#0 that is gapped at TCDW and group b#1 that remains ungapped by that transition.

Figure 3d shows spectra measured near the top of the valence band at \(\overline{\Gamma }\) at 80 K and 20 K. Already at 80 K the spectral weight at the Fermi level EF is very small, consistent with the presence of a gap at this temperature. In the broad line shape one can identify a peak at −0.2 eV and a weaker shoulder at −0.07 eV, indicated by arrows, which can be associated with the energy splitting predicted by theory. We notice again that the energy width of the underlying features is not limited by the experimental resolution. At 20 K the shallower feature grows in intensity and moves away from EF to −0.11 eV, suggesting a corresponding widening of the energy gap.

To understand where the tendency towards ICDW might come from, it is instructive to plot the equienergy surfaces in the momentum space. Figure 4b,c show that the occupied states have strongly 1D character with dominant dispersion along the c*-direction (see also Fig. 2); the character becomes even more pronounced deeper from Fermi level. To investigate the tendencies towards charge ordering instabilities, we evaluated the real part of the Lindhard electronic susceptibility (see Supplementary) that is shown in Fig. 4d,e. One indeed finds that the electronic susceptibilities are large and in particular finds the enhancement of the electronic susceptibility in a strip centred at kz = 0 with slightly higher values at ky = π/b. This is in qualitative agreement with the experimental values, where the incommensurate ordering is found at (0.195, 0.5, 0.120). The calculated susceptibility is large but not maximal at this wave vector (see SI for discussion).

Transport and Hall measurements

The in-plane resistivity temperature dependencies, ρxx(T), of Mo8O23 single crystals are shown in Fig. 5a. The overall temperature dependence is consistent with previously reported data12, which was limited to below 300 K. The in-plane anisotropy is \({\rho }_{a}/{\rho }_{c}\sim 5\) at room temperature, consistent with the one expected from band structure calculations. In the highest-temperature metallic state, just before any transitions, the resistivity along the most conducting c-axis is still rather high for a metal – ρc ≈ 20 mOhm · cm, which, in the absence of significant disorder, points to large scattering and suggests the important role of correlations. It is also close to values expected for a polar metal19,20, which is interesting since Mo8O23 in CDW phase also lacks inversion symmetry13.

Above room temperature, on cooling below TCDW ≈ 343 K, resistivity changes its slope from metallic to insulating (Fig. 5b), indicating the onset of the phase transition associated with the incommensurate charge density wave12. Despite the broad resistivity feature at TCDW, there is apparently no hysteresis at the transition temperature. The prehistory effects are limited to contact degradation which can be annealed. As temperature is lowered, resistivity increases with a minor change of slope at incommensurate to commensurate transition TIC−C = 285 K.

The most pronounced feature is the cusp at T* ≈ 130 K, below which the resistivity starts to drop. Below 30 K the slope changes again to insulating. These features take place in the temperature range where no structural changes were reported. While they are observed in both in-plane components, the T* value is several Kelvin different between the two, emphasizing its link to transport properties rather than to the presence of some phase transition.

In a wide range of temperatures (T > 20 K), the Hall effect is linear-in-B, while the MR is quadratic-in-B at least up to 10 Tesla, see Fig. 5e. This is an indicator that the mobilities of the carriers involved in the transport are much lower than than 1/10 T−1 1000 cm2/Vs. Indeed, maximum low-field Hall mobility estimate yields μ ≈ 280 cm2/Vs at T = 20 K. Hall effect shows n-type carriers over the whole temperature range. As temperature decreases from the room temperature, the resistivity, Hall effect and positive magnetoresistance (PMR) coefficient α (ρxx(B) = ρ0 + αB2) – all increase. We relate this growth with the decrease of number of carriers involved in conductivity with decreasing temperature.

In the 110–140 K interval, the resistivity and the α value have maxima, while the Hall resistance (Fig. 5c) has a plateau. Below 100 K the Hall concentration starts to decrease again, and the positive magnetoresistance drops steeply. The peak in PMR can be understood with the two-liquid model21 as the result of matching conductivities of two group of carriers (see SI). Interestingly, these changes do not manifest themselves in the mobility (triangles in Fig. 5d): the latter grows monotonously as temperature decreases. Such behaviour suggests that only concentration is affected by the low-temperature transformation, whereas scattering mechanisms stay intact.

For the lowest temperatures (below 20 K), the negative magnetoresistance emerges, that becomes sharper as temperature decreases (Fig. 5e). We relate the NMR to the weak localization phenomenon, as it is also accompanied by decrease of mobility.

Low-energy tunnelling spectroscopy

The low-energy ±100 meV tunnelling spectra reveal the complex two-stage gap opening from the initial semi-metal state as the temperature decreases from 400 K to 9 K (Fig. 6a). The curves at 375 K and 400 K, above the CDW transition temperature, are featureless in this energy range and demonstrate almost constant finite tunnelling density of states with a shallow wide minimum at EF. To emphasize the spectral changes upon cooling, we normalize all the curves by the 375 K data, that is above the CDW transition temperature. Density of states starts to decrease in the ±50 meV energy range already below 350 K. This spectral feature grows in width towards 250 K. As temperature decreases, the dip becomes progressively deeper, reaching zero density of states at EF below 30 K. At the same temperature, its width becomes larger (Fig. 6b). This is preceded by the onset of the pronounced asymmetry in the spectra below 60 K (lower panel in Fig. 6a). On the other hand, no large-scale changes are observed around T*, where transport crossover is observed.

Figure 6
figure 6

Tunnelling spectroscopy at low energies: (a) Tunnelling conductance dI/dV curves in the 9–400 K temperature range (normalized to the dI/dV at T = 375 K) and (b) 2D plot of the same dI/dV curves symmetrized for clarity (c) Examples of curves below and above T = 175 K fitted with Dynes formula with constant Γ (black) and energy-dependent Γ(E) (red). (d) Temperature dependence of the gap value 2Δ, energy-independent smearing Γ0 and energy-dependent ΓE, extracted using modified Dynes formula from curves in (a). Red curve shows the Δ(T) fit for T [176, 350] K with the mean-field (BCS) dependence. (e) Blue dots show the evolution of the normalized experimental Fermi level tunnelling density of states (raw dI/dV values) with temperature. Orange and green curves show the theoretical Fermi level density of states evolution due to the thermal smearing of GGA and experimental 4.2 K tunnelling spectra, respectively. Red dashed line shows the Fermi level density of states evolution for spectra obtained with the DF with Δ(T) and Γ0(T) substituted from the fits in (c) and the kink position is shown by an arrow.

Quantitative analysis of the T ≥ 175 K tunnelling spectra with the Dynes formula22(DF) reveals a mean-field-like gradual opening of the gap starting at T = 350 K, which coincides with the MIT observed in resistivity. The DF satisfactorily describes the normalized spectra (Fig. 6c) in the wide energy range, ±0.3 eV, and allows us to extract independently the gap Δ and the phenomenological smearing factor Γ0, shown in Fig. 6d. The gap saturates around 250 K, and further decrease in the density of states is due to decreasing smearing. Neither of the parameters experiences features around the TICC transition. Interestingly, the ratio of the saturation gap value and the transport MIT temperature, Δsat/kTCDW ≈ 1.6, is close to that for the weak coupling, quite unusual for inorganic CDW systems23.

Simple DF analysis breaks down below T 175 K in estimating correctly the smearing, whereas the gap is reliably found to stay almost constant down to 40 K whereupon it starts to grow again, increasing 1.5 times at 9 K. The simple DF fit overestimates density of states at low energies, as shown in Fig. 6c.

To describe consistently the spectra in the full temperature range, we introduce into DF the phenomenological dependence of smearing on energy (see SI), \(\Gamma (E)={\Gamma }_{0}+{\Gamma }_{E}{(\frac{E-{E}_{F}}{\Delta })}^{2}\). The choice of Γ to be energy-dependent is motivated by it’s high sensitivity to changing the fitting range in DF, quite unlike the gap Δ which is more robust. The form of Γ(E) is given by the Taylor series expansion22 respecting the symmetry of the problem. With increasing the energy-dependent contribution ΓE, the quasi-particle peaks at the gap edges are rapidly suppressed, while close to EF the spectrum is less affected, as illustrated for the 70 K curve in Fig. 6c. This corresponds to a significantly different shape of the tunnelling DOS, compared to the simple DF. We will use the ratio of γ = ΓE0 to quantify this difference.

The results of the fit are shown in Fig. 6d. The spectrum starts to change when γ becomes non-zero only below T 150 K, reaching γ ≈ 1 around 70 K and rapidly peaking along with the increase of the gap size below 40 K. In contrast, the usual smearing Γ0 temperature dependence is slightly nonlinear and featureless. At high temperatures, it coincides with that in the conventional DF, reaching Γ ≈ 2.7 kT, and approaches zero at low temperatures, along with zeroing of the density of states at EF.

The observed large non-thermal smearing Γ0 > Δ strongly affects opening of the gap leading to the kink in the temperature dependence of the Fermi level density of states (FLDOS) close to T*. Generally, FLDOS in a small-gap insulator should remain zero for kT << Δ and then smoothly increase with temperature due to thermal smearing. This general behaviour is indeed reproduced (Fig. 6e) if low-temperature DOS is convoluted with the Fermi-Dirac distribution using the Bardeen formula for tunnelling24 and is inconsistent with the data. For the kink to emerge, the temperature in simulations should be increased to kT Δ. The same effect is reproduced also with DF if Γ0 becomes large compared to Δ, as shown in Fig. 6e for the experimental values of Δ and Γ0.

Increase of the gap Δ and change of the DOS shape, γ, below 30 K mark the stabilization of the new state at low temperatures, clearly different from the high-temperature one. With Γ0 → 0 and finite ΓE at low temperatures, the tunnelling spectrum is completely different from the high-temperature DF shape. This change is clearly linked with the increase of the gap, implying that the latter is also unrelated to the order present at high temperatures. Linear extrapolation of Δ(T) gives the onset temperature of 67 K, close to the point where γ ≈ 1, further supporting the close connection between the low-temperature state and change in the spectra shape. Importantly, the tunnelling spectrum at energies larger than the gap, |E − EF| > Δ, also starts to deviate from the high temperature one at 70 K (bottom panel in Fig. 6a), perhaps due to a change in tunnelling conditions or phase separation due to emergence of the new state.

To conclude, low-energy tunnelling spectroscopy provides us with strong evidence for the presence of two phases, one of which is related to a mean-field ordering transition, whereas another has correlated nature. The onset of the latter is clearly linked with the changes of background density of states at energies larger than the gap.

Discussion

High-T CDW transition

For the temperature range 175–400 K, the overall results fit nicely the scenario of the semi-metal to incommensurate CDW phase transition. Indeed, for the first time we observe the metal-insulator like transition in resistivity at TICDW ≈ 343 K, (see Fig. 5b) together with the mean-field gap opening seen in the STS spectra (see Fig. 6a,d). The transition temperature is within the range expected from structural and neutron scattering studies 315 K ≤ TICDW ≤ 360 K12,13,14. The tendency to ICDW formation in Mo8O23 is lucid if one considers quasi-one dimensional character of the bands together with the temperature dependence of the chemical potential, that follows from the strong asymmetry of DOS of high-T band structure. The gap size predicted by DFT is consistent with the observations.

It is still puzzling why the Hall concentration follows the Arrhenius law expected for a gapped CDW only below 200 K, well below TICDW. Similarly, why the soft mode observed in time-resolved reflectivity spectroscopy11 does not behave as expected for a CDW transition at TICDW, and softens only around 200 K. Below we try to address these questions by looking into the spectroscopic signatures.

The combination of a large non-thermal smearing, Γ0/kT 2.7, and a rather small gap/transition temperature ratio, Δ/kTCDW 1.6, observed both in STS and ARPES, leads to the regime of Γ0 > Δ at high temperatures, which makes Mo8O23 system clearly unconventional. For comparison, the ratio Δ/kTCDW is several times larger in similar systems23, and only slight deviation of smearing from thermal was reported for CDW in the same material class system Rb0.3MoO325. There are several contributions to smearing. Impurities and inhomogeneities of the order parameter can produce finite Γ0, but its temperature dependence is weak in that case26. Inelastic phonon scattering22,27 could be the reason for large Γ0 values – phonon peaks in Raman measurements11 are present up to 100 meV, which is even larger than Γ0 values seen in STS. The above mechanism was predicted to result in Γ0T3, showing the tendency to saturation close to TC28. We indeed observe slight nonlinearity in energy-independent Γ0(T) above 30 K suggesting that inelastic phonon scattering contributes substantially to smearing.

The regime of Γ0 > Δ changes the conventional carrier depletion usually observed in magnetic susceptibility or Hall measurements. Delocalized carriers contribute to Pauli and Landau terms in magnetic susceptibility – both are proportional to the Fermi level density of states29. In conventional CDW systems it is expected to decrease abruptly after the transition, as it happens e.g. in Mo4O1115. However, in Mo8O23 magnetic susceptibility is featureless around TCDW15 – same as FLDOS extracted from STS – and does not resolve the CDW transition. Instead, it changes slope at lower temperatures, where Γ0 Δ. Similarly, Hall concentration should follow the activation law after opening the gap, but in Mo8O23 it changes weakly down to T 200 K and only with further cooling the activation behaviour sets in (Fig. 5). We thus conclude that the carrier density sees the gap only below this temperature, when Γ0 becomes comparable to Δ, despite the fact that the actual gap opens at much higher temperatures.

It is thus not surprising that the onset of the gap shape change γ happens in the same temperature range where Γ0 becomes comparable to Δ. For the Γ0 Δ regime, small modifications to FLDOS, if present, are buried under the contribution arising from large smearing due to quasiparticles scattering. Once this contribution becomes small compared to the one produced by additional emerging order, the latter becomes visible in spectroscopic measurements. This is nicely illustrated by the same temperature for onset of γ and Fermi level kink in Fig. 6d,e. The kink in the Fermi level DOS at around 150 K is most likely responsible for the transient reflectivity relaxation bottleneck and the increase of the optically-excited quasiparticles lifetime observed in the vicinity of this temperature11. The extreme sensitivity of transient reflectivity measurements to the delay in carrier depopulation suggests that anomalous softening of the most pronounced coherent mode close to 200 K may also have the same origin. That would signify mode’s polaronic nature, i.e. that of correlated electrons strongly interacting with the lattice. Interestingly, the polaronic nature of quasiparticles coupled to the high-energy phonons would be also consistent with the smearing of the photoemission spectra in other oxides30. Otherwise, the high-temperature bottleneck observed in the transient reflectivity relaxation11 below TCDW agrees well with the present data: 2Δ  0.14 eV from the bottleneck fit is close to 2Δ(T = 200 K)  0.1 eV from STS.

Low-T non-structural transformation

In contrast to the high-T regime, the entire comprehensive set of data below 175 K directly confronts the simple phase diagram of a narrow-gap semiconductor or even that with only one CDW order parameter. Multiple experimental probes reveal that more than one carrier species is involved in the ordering processes which occurs more than 200 K below the reported structural CDW transitions. Therefore, one has to conclude that the DFT band structure can describe well the gross features 1 eV, but the low-energy DFT predictions, <0.1 eV, are inconsistent with the observations. We note that the narrow-gap semiconductor DFT scenario (see SI) still can be applied down to 100 K to explain the resistivity maximum at T*. Alternatively, DFT with a stronger influence from semi-metallic DOS can be considered within an excitonic insulator scenario and also used to explain the gapped behaviour (see SI).

The additional transformation takes place at 70 K and it is not associated with the structural changes. The signatures are observed in multiple probes: (i)ARPES spectral shape changes between 80 and 20 K, (ii) STS spectral shape changes around 90 K, the gap size increases below 40 K and can be extrapolated to 70 K, (iii) Hall concentration decreases at 70 K and then saturates at a different value. Below we will discuss the possible scenarios describing the above set of data.

Magnetotransport measurements yield the most direct evidences of at least two types of carriers: magnetoresistance peaks around 110–140 K and the Hall concentration demonstrates a plateau in the same range of temperatures. This behaviour can be approximated with a simple model nHall2gN0 exp(−Δ(T)/kT) + N1 illustrated in Fig. 5f. Indeed, Hall concentration has a linear section in Arrhenius plot, giving a gap value of 100 meV comparable to that in STS. The ratio of the high-T Hall concentration to that on the plateau is N0/N1 ≈ 400, \({N}_{1}\mathop{ < }\limits_{ \tilde {}}{10}^{17}\) cm−3. Such behaviour of magnetotransport is expected when more than one band contributes to the Fermi surface: part of the Fermi surface is gapped due to CDW onset, whereas the small metallic pocket is left unaffected in a different part of the Brillouin zone.

The low-density group of carriers is further depleted below 70 K in Hall measurements – the same temperature where the correlation gap sets in the STS data. Since we plot the dI/dV curves normalized by the high-temperature spectrum, we do not resolve well the band occupied by N1 carriers, only the change in it below 70 K. The latter is likely related to a small kink around 50 K in FLDOS (Fig. 6e) on the verge of our resolution. This depletion is also consistent with the magnetic susceptibility measurements12, which revealed a small drop in χdeloc (Pauli + Landau contributions of delocalized carriers) below 70 K. Interestingly, no magnetic ordering so far has been observed12, hence indicating that the change in χdeloc is associated with the decrease of the FLDOS29.

Still, it is unusual that the change in a relatively small group of carriers affects the spectral shape on the energy scale of 2Δ(T = 0)  130 meV. Such a contribution to DOS can have two possible origins: spatial electronic inhomogeneity and orbital character (see SI). The data in this study were averaged spatially (see Methods). Therefore, in the case of a real space phase separation, the contribution to the total DOS coming from each phase is weighted by a relative area it occupies. Alternatively, in the case of reciprocal space phase separation, the contributions are weighted then by tunnelling selectivity – the tunnelling matrix element. It is more selective to orbitals sticking out of the sample plane, compared to those in-plane24 (see SI). The changes in tunnelling spectra below 70 K thus allows us to conclude that the relative weight, either spatial or orbital, of the new state increases and is responsible for its enhanced visibility. At the same time, the increase of the gap size starts only below 40 K, demonstrating that the increased gap size is directly related to the emerging low-temperature state, not the CDW.

Do the two states compete or coexist? We note that neither Hall concentration nor resistivity demonstrate activated behaviour at low temperatures, T < 70 K, in clear contrast to a simple band/CDW insulator, thus suggesting inhomogeneity or some additional contribution to transport. Macroscopic separation with large domains of either states can be ruled out because no hysteresis effects were observed in our resistivity measurements. Another scenario involves formation of static or dynamic nanoscale separation, which is quite generic in correlated systems with competing interactions31. The behaviour observed in Mo8O23 is thus reminiscent of that in the vicinity of the Widom line endpoint, characteristic of many Mott insulators. However, in the absence of reports on magnetic structure, dynamic coexistence of insulating and metallic nano-droplets, in the spirit of the Gor’kov-Sokol model32, appears the most favourable. Further real space spectroscopic imaging studies are necessary to address this issue.

To summarize, we have tracked the electronic transformations in the phase diagram of Mo8O23, and found a new correlated-electron ground state, which emerges in the absence of known structural distortion. Using our new results we have proposed the robust unifying picture of various unexplained features previously observed in transport, magnetic and optical measurements. This work also brings a new light on the onset of the CDW in inorganic systems, emphasizing the role of electron lifetimes. The very large electron-hole asymmetry in the band structure implies that photoexcitation may lead to photodoping, and thus to the control of the phase diagram by light. Such perturbations could affect the transient electron occupation that would modify the preferred ordering wave vector. The competition of CDW and correlated state can be exploited for memristor devices or field-effect gating of in-plane resistance.

Methods

The samples were grown from Mo powder (99.9% 1–2 μm), and Mo(VI) oxide taken in molar ratio 1:20 (≈1 g of mixture) thoroughly mixed and grind in agate mortar into a refined powder. Growth was done, similar to previous reports18,33, with chemical vapour transport using iodine as agent. The crystals of Mo8O23 can be easily distinguished from other suboxides by pinkish color and their barrel shape. They can be nicely cleaved in the ac-plane. The monoclinic space group P2/c and unit cell parameters were confirmed at room temperature with the single-crystal X-ray diffraction (XRD) to match the previous reports13. Thermogravimetric analysis demonstrated no change of the sample mass upon heating in Ar atmosphere up to 200 °C.

The resistivity tensor measurements were performed in vacuum on samples cut along corresponding crystal axis of approx. 1 mm × 300 μm × 100 μm with the conventional low-frequency (7–14 Hz) AC four-point technique at the currents of I 10–100 μA, in the linear IV regime. The samples were oriented using XRD. Magnetotransport measurements were done on a cleaved 180 μm thick sample (lateral size 0.6 × 0.7 mm2) in van der Pauw geometry using Cryogenics dry CFMS-16 cryomagnetic system in the temperature range 2–320 K and magnetic fields up to ±10 T. The use of 50 μA current ensured the linear IV regime throughout the whole temperature range. Due to intrinsic anisotropy of the sample and inevitably imperfect geometry, the diagonal component of the resistivity tensor admixtures to the Hall effect. High magnetic field range allowed us to extract magnetoresistance (symmetric in magnetic field) and the Hall signal (antisymmetric in magnetic field) measured simultaneously.

To achieve low contact resistance in (magneto)transport measurements, the pre-patterned contact areas were sputter-covered by the 50 nm Au (40 nm Ti for magnetotransport) layer. Samples were mounted by gluing thin annealed gold wires to the contact pads with the conducting silver epoxy and cured at 110 °C. To avoid strain, samples were left suspended on wires during the temperature dependence measurements. The wires were anchored to the sapphire substrate to ensure proper thermal contact.

All spectroscopic measurements were performed on fresh surfaces cleaved at room temperature in an ultra-high vacuum chamber, at a pressure of 2 × 10−10 mbar. Tunnelling spectra and topography were measured with Pt tips; their spectra were checked on crystalline gold. Data at various temperatures were measured using the heated sample stage weakly coupled to the cryostat kept at either 4.2 K or 77 K. The temperature values shown are taken from the thermometer readings installed on the stage. Tunnelling spectra at each temperature were averaged spatially over several unit cells taken from different sample spots within the area of 200 × 200 nm2. ARPES measurements were performed on samples cooled to 80 K and 20 K, using 21.2 eV photons from of a high-brightness He lamp. The total spectral broadening (experimental and thermal) was lower than 80 meV for the data taken at 80 K and 20 meV for data taken at 20 K. The Fermi level was calibrated on a clean polycrystalline gold surface.

The density-functional theory calculations of electronic structure were done for previously identified crystal structures12,13 using the Wien2k framework. The wave-vector summations were done over 500 and 1000 k-points in the irreducible Brillouin zone for the low-temperature and high-temperature structures, respectively. The energy cut off for valence electrons was set to −6 Rydberg, that is oxygen is described as He + valence electrons and molybdenum as argon + valence electrons. We performed the calculations both within the local-density approximation(LDA) and as well the generalized gradient approximation(GGA). Hubbard U and spin-orbit coupling were not included in the present calculations. In related 4d correlated oxides, such as ruthenates or the rhodates, the spin-orbit coupling does not importantly affect the density of states.