Observation of unusual topological surface states in half-Heusler compounds LnPtBi (Ln=Lu, Y)

Topological quantum materials represent a new class of matter with both exotic physical phenomena and novel application potentials. Many Heusler compounds, which exhibit rich emergent properties such as unusual magnetism, superconductivity and heavy fermion behaviour, have been predicted to host non-trivial topological electronic structures. The coexistence of topological order and other unusual properties makes Heusler materials ideal platform to search for new topological quantum phases (such as quantum anomalous Hall insulator and topological superconductor). By carrying out angle-resolved photoemission spectroscopy and ab initio calculations on rare-earth half-Heusler compounds LnPtBi (Ln=Lu, Y), we directly observe the unusual topological surface states on these materials, establishing them as first members with non-trivial topological electronic structure in this class of materials. Moreover, as LnPtBi compounds are non-centrosymmetric superconductors, our discovery further highlights them as promising candidates of topological superconductors.

presence of the Dirac like feature. I support publication of this manuscript in Nature Communications. I would encourage authors to comment on other aspects of possible topological properties their data. One of the hallmarks of topologically non-trivial states, especially for any topological signatures in transport is presence of odd number of FS crossings. I understand that the feature reported here lies completely below Ef, but authors should examine this aspect of their data and comment in text.
Reviewer #3 (Remarks to the Author): In this manuscript, Z. K. Liu and et.al. reported observation of unusual topological surface states in half-heusler compounds LnPtBi (Ln=Lu or Y), an interesting compound which is non-centrosymmetric and superconducting at low temperature. These properties, if combined with non-trivial topological states, are indeed attractive and have been drawing recent attentions (for example, Weyl semimetal without inversion symmetry or topological superconductors). Half-heusler compounds have been theoretically predicted to be topologically non-trivial, and some of the compounds (Ln=Lu, Dy, Gd) have been measured before (Ref.27), but with no clear conclusions. In this manuscript, however, by careful ARPES measurement and detailed comparison with ab-initio calculations, Liu et.al observed clear surface states as well as a non-trivial Dirac cone well below Ef at gamma point with high resolutions. This is very nice and provide experimental support to the theory and prediction. I should say this is a piece of nice work with beautiful data and careful analysis that are convincing, however, I do not think it is novel enough to justify its publication on Nature Commun. My judgement is based on the following considerations.
(1) The compounds are actually metallic, namely, they are NOT topological insulators. In principle, in the absence of gap, the Z2 topological property is ill-defined (except topological metal, where surface Fermi arcs are expected). In this case, the so-called topological surface states, if any, are not protected.
(2) The surface states observed at Fermi levels are actually all trivial. The only claimed Dirac cone (TSS) is located well below Fermi level. This is again unattractive, because topological surface states are interesting only if they are located within a band-gap and crossing the Fermi level. Dirac cone type surface states away from Fermi level may widely exist in most of materials (if not all).
(3) The above two weak points make the compounds "boring", although their data are beautiful. This fact also weaken the possible connection, as claimed by authors, between the topology and the specially interesting non-centrosymmetry as well as superconductivity in this family of compounds. In general, we expect the superconductivity is related to the Fermi level and Fermi surfaces (which are all trivial in this case as observed).
In summary, I believe the manuscript provides interesting and convincing information, particularly for half-heusler compounds, but its importance may not satisfy the criteria of Nature Commun. I therefore suggest author to expand the paper and submit it to specialized magazines.

REVIEWERS' COMMENTS:
Reviewer #1 (Remarks to the Author): The authors have made substantial revisions in respond to the comments and questions raised by the reviewers. I recommend accepting the revised manuscript to be published in Nature Communications.
Reviewer #2 (Remarks to the Author): I find the response the authors to both reports satisfactory and revised version of the manuscript suitable for publications.
Reviewer #3 (Remarks to the Author): Authors have made comprehensive response to my previous report. I think the response is good and reasonable. As I have explained in my previous report, this paper is nice and provides convincing data to support the non-trivial topological nature of LnPrBi class of materials. On the other hand, however, it is indeed difficult to justify its novelty. First, for topological insulators, the existence of bulk gap is important for real transport measurement. Otherwise, the topological surface states, if any, play minor roles for the global properties. Second, for topological semimetals, the bulk states should play more important roles than the surface fermi arcs. In my opinion, nowadays, there are too many ARPES studies, which confirm theoretical calculations and report the existence of non-trivial surface states. These studies are good, contribute the field, but not as significant as some years ago (when the field just started). Nowadays, the most important issue (from the viewpoint of materials) is to find better materials which can be well manipulated and controlled. I therefore leave the final judgement up to the editor.
Ternary Heusler compounds have attracted great interests due to their rich physical properties and the possible topological nature predicted by calculations. The search for the topological nontrivial surface states in these materials is of great importance. This paper present clear measurements of the electronic structure of LnPtBi, providing convincing evidence for surface states below Fermi energy with "X" shape band dispersion, which was missing in previous ARPES reports. Combined with band structure calculations, the authors suggest that the band is topological nontrivial.
The paper is well written, and the data quality is high. I feel that their findings are sufficiently exciting and suggestive to warrant publication in Nature Communications, as long as the authors can adequately address the following questions/comments: Authors' response: We thank the reviewer for acknowledging the importance of this work and the high quality of our data. We also thank the reviewer for the helpful suggestion. As suggested, we present further analysis below (and in the revised manuscript) to address the questions/comments from the reviewer.
1. The main conclusion of this paper is that the "X" shape band is topologically nontrivial surface state. However, considering that the "X" shape band shows a back bending dispersion at the tip part, if the back bending dispersion is also surface state, the surface state can be eliminated by shifting the chemical potential, which sets doubts on the topological nontrivial nature of the band proposed in this paper. In the part D of the supplementary, the authors assume the back bending dispersion as the hybridization of surface state and bulk states, but did not provide any experimental evidence. At least, the authors should show the kz dispersion of the band bending dispersion to support their assumption.

Authors' response:
Thanks for the reviewer's suggestion to add more discussions and k z dispersion of the back-bending dispersion. We have added the k z dispersion of the "X" shape band at different binding energy, and confirm that the back-bending part indeed shows considerable k z -dispersion (as opposed to the linearly dispersing part which does not), thus providing clear experimental evidence that the back-bending part indeed hybridize and merge into the bulk band. The related discussions have been added as the new Part B in the supplementary information of the revised manuscript.
In Fig. R1 below (we also add this figure into the Supplementary Information as new Fig. S2), we chose two momentum distribution curves (MDCs) that cut through the "X" shape dispersions at different binding energy to demonstrate their different k z dispersions (MDC1 cuts through the back-bending part near the band top (E b =0.41eV) and MDC2 cuts through the linear dispersion part (E b =0.67eV), respectively).
For better illustration, we mark the MDC peaks related to back-bending part (blue circles, see  Fig.S7 show poor data quality, making it hard to identify different bands, in contrast to the claimed "clear circular dichroism". To get convincing conclusion on the chiral texture for band "TSS" and "SS3", MDCs measured using "CR","CL" and "CR+CL" photons should be compared. Moreover, as the spin resolved ARPES measurements have not been performed at the current stage, the CD-ARPES measurements which can hint on the spin configuration is very important, and should be presented in the main text.

Authors' response:
We thank the reviewer for the suggestions on making better comparisons on the CD-ARPES data (as these measurements were carried out at low energy (20eV), the photoemission cross section of LuPtBi under such energy photons caused relatively worse statistics compared to other measurements in the manuscript).
In Fig. R2 (also the new Fig.5 added in the main text of the revised manuscript) we show more detailed CD analysis including the fitting to four MDCs cutting through the TSS and SS3 bands under circular right (CR) and circular left (CL) photons, as well as their difference spectrum (CR-CL). The circular dichroism of both SS3 and TSS is more apparent in the CR-CL spectrum (see Fig. R2(a)(iii)). The fitting of the four MDCs in Fig. R2(c) further proves the chiral texture for TSS and SS3 bands. In MDC1 that cuts through the band top of SS3 (0.14eV below E F ), we observed a negative peak (labeled by purple mark) and a positive peak (labeled by the blue mark), corresponding to the two branches of the SS3 which has stronger intensity with "CR" and "CL" photons, respectively. Going towards higher binding energy, the single peak of each branch evolves into two, as seen in the MDC2 at 0.31eV below E F : purple peak splits into purple and yellow while blue peak splits into blue and cyan (the fitted peaks from the same branch show the same sign of amplitude).
As for the TSS, the fittings to the MDC3 at E-E F =-0.44eV and MDC4 at E-E F =-0.71 shows one branch of the 'X' shape is stronger with "CR" photons (green marks) thus appears as a positive peak in the "CR-CL" spectrum; while the other branch is stronger with "CL" photons (orange peaks) and appears as a negative peak.
Therefore, our ARPES measurement indeed shows clear circular dichroism signal from both SS3 and TSS band: each branch of the TSS and SS3 is suppressed with one circularly polarized photons and enhanced with the other.
We have added Fig. R2 as the new Fig. 5 into the main text of the revised manuscript together with the corresponding discussions. 3. The authors of Ref.27 observed a band contributed by surface state near Gamma with weak intensity, and they suggest that further ARPES experiments are required to explore whether this band is topological non-trivial or not. In the manuscript, the weak band is also clear in Fig.3c and Figs.4(a)(b)(f)(g); however, this band is not even mentioned. Could this band be explained by a Kramers pair of bands? The author should compare their results with the previous literatures and discuss on this open issue.
Authors' response: We thank the referee for the careful reading and pointing that out. Indeed there is a weak band near Γ next to the SS2 band (see Fig. R3(a) below, indicated by the green arrow). The position and shape of this band is similar to the band reported in Ref. 27 (currently Ref. 29 in the revised manuscript), which was suggested in Ref. 27 as a Rashba type surface state. However, the fact that this band only shows sharp single branch dispersion make it very different from other Kramers pair in this materials, such as SS2 and SS3 nearby that both show clear (and large) spin-splitting.
To investigate the origin of this band, we first carry out photon energies dependent measurement and study its k z -dispersion ( Fig. R3(b)(c)). As clearly shown in Fig.  R3(c), the position of the Fermi surface crossing and shape does not show much variation along k z (Fig. R3(c)), showing its surface origin. However, this band also shows considerable difference from other Rashba type surface states nearby, such as adjacent SS2 and SS3 bands in that: (1) this band does not come in pairs and (2) the dispersion has much weaker intensity comparing to SS3 and SS2 in ALL measurements. These differences indicate that this band is not typical Rashba split surface state (such as SS2 and SS3).
In order to understand the nature of this band, we carry out ab-initio calculations under different conditions. From Fig. R4 R4(c)(ii)). Such enhancement of spectral weight with thicker slabs, together with its surface nature we demonstrated in Fig. R3, suggest that it is a surface resonant state (a mixture of surface and bulk states), which satisfactorily explains the absence of the k z dependence and its different features from Rashba type surface states (such as SS2 and SS3).
The discussions above have also been added into the Supplementary Information of the revised manuscript.

Response to Reviewer's comments:
Reviewer #2: The authors present study of electronic structure of rare earths half Hausler compounds. In addition to previously observed set of surface states, they found evidence for Dirac-like feature at the center of the Brillouin zone at -0.45 eV. Topological quantum materials are very active and exciting and still relatively new area of condensed matter physics. Rare earths half Hustler materials are particularly promising as they combine topological structure, superconductivity and heavy fermion behavior. The topic of this manuscript is therefore timely and interesting.
The data is of high quality and presence of Dirac-like feature at the center of the zone is established beyond doubt. Extensive photon energy dependent studies validate surface character of this feature.The manuscript is well written with clear motivation and nice data demonstrating clearly the presence of the Dirac like feature. I support publication of this manuscript in Nature Communications. I would encourage authors to comment on other aspects of possible topological properties their data. One of the hallmarks of topologically non-trivial states, especially for any topological signatures in transport is presence of odd number of FS crossings. I understand that the feature reported here lies completely below Ef, but authors should examine this aspect of their data and comment in text.

Authors' response:
We thank the reviewer for acknowledging the importance of this work and the high quality of our data. We also thank the reviewer for the helpful suggestions.
Regarding the non-trivial topological surface state (TSS) we report in the manuscript, although it is located ~500meV below E F and does not contribute directly to the transport properties, its existence directly proves the non-trivial topological invariant in the half-Heusler alloy -this could help to explain the many interesting/exotic properties discovered in this compound (and possibly also other half-Heulser compounds), such as the high mobility electrons and large magnetoresistance (Z. P. Hou et al., Phys. Rev. B 92, 235134(2015)), chiral anomaly and Weyl fermions under magnetic field (M. Hirschberger et al., arXiv:1602.07219 andC. Shekhar et al. arXiv:1604.01641) and unconventional superconductivity which hosts nodal gap structure (H. Kim et al., arXiv:1603.03375v2) and possible spintriplet pairing mechanism (Y. Nakajima et al., Science Advances 1, e1500242 (2015)).
Moreover, the TSS could become more relevant after proper manipulation of the doping level and the band structure in this compound: By tuning of the Fermi level, one would be able to move the E F close to TSS and have the topological surface electrons contribute to the superconductivity pairing, and thus achieve one possible scenario of topological superconductivity. In addition, there have been theoretical works demonstrating that half-Heusler compounds will become a topological insulator with the TSS sitting in the bandgap under uniaxial strain (D. Xiao et al., Phys. Rev. Lett. 105, 096404(2010)); or become Weyl semimetals under compressive strain (J. W. Ruan et al., Nat. Commun. 7:11136 (2016)). Therefore, our identification of the non-trivial topology in half-Heusler compounds is an important milestone which will encourage the exploration for these new topological phases. Besides, as the TSS and its evolution would be important in these new topological phases (TIs or Weyl semimetals), our discovery of the TSS thus provides an important indicator for the identification, bandstructure engineering and the observation of topological phase transitions.
As suggested by the reviewer, we have also added the discussions into the manuscript (page 7, paragraph 2).

Response to Reviewer's comments:
Reviewer #3: In this manuscript, Z. K. Liu and et.al. reported observation of unusual topological surface states in half-heusler compounds LnPtBi (Ln=Lu or Y), an interesting compound which is non-centrosymmetric and superconducting at low temperature. These properties, if combined with non-trivial topological states, are indeed attractive and have been drawing recent attentions (for example, Weyl semimetal without inversion symmetry or topological superconductors). Half-heusler compounds have been theoretically predicted to be topologically non-trivial, and some of the compounds (Ln=Lu, Dy, Gd) have been measured before (Ref.27), but with no clear conclusions. In this manuscript, however, by careful ARPES measurement and detailed comparison with ab-initio calculations, Liu et.al observed clear surface states as well as a non-trivial Dirac cone well below Ef at gamma point with high resolutions. This is very nice and provide experimental support to the theory and prediction. I should say this is a piece of nice work with beautiful data and careful analysis that are convincing, however, I do not think it is novel enough to justify its publication on Nature Commun. My judgement is based on the following considerations.
Authors' response: We first thank the reviewer for acknowledging the quality and value of our work. However, with due respect, we do not agree with the reviewer's judgment that the work is not "novel enough". Please allow us to elaborate below the significance of Heulser topological materials below, why they are different from topological insulators, and their influence/implication to many important and exciting scientific problems -all of these make the discovery of the non-trivial topology in these compounds really important and exciting, thus worth the dissemination to broad audience of Nature Communications: (1) Half-Heusler alloys represent a big family compounds that interest physicists, chemists and material scientists with many intriguing properties (including unusual magnetism and superconductivity) which need to be understood. In particular, they have drawn considerable research attentions recently due to the discovery of various exotic properties, such as: a Many of these new properties strongly indicate or require that the topological electronic structure to plays an important role in these compounds, yet a direct evidence of their non-trivial topological nature is still lacking. Moreover, the fact that Ref 27 (currently Ref 29 in the revised manuscript) gave a negative conclusion (due to the data quality) makes the situation more confusing, thus a clarification of is urged. Our work, for the first time, clearly demonstrated the topological surface states in LnPtBi, thus unambiguously proves the non-trivial topological nature of these half-Heusler compounds.
(2) In addition to the observation of the topological surface states, the significance of our work goes beyond. As these topological surface states are unusual: Naively one would assume that the topological surface states must appear inside the bulk gap (this was part of the reason why they were overlooked in previous experimental and theoretical study), nevertheless our results clearly show that they can exist inside the inverted band gap -which is not necessarily the normal gap between conduction and valence bands -and can (surprisingly!) survive the hybridization with the bulk states, thus the topological surface states are not as vulnerable as one previously thought and must live in the normal band energy gap.
(3) Besides the topological surface states, our work confirms the inverted bulk band structure, -which is the prerequisite to understand the unusual superconductivity in these compounds, for instance, to interpret the unconventional pairing mechanism.
(4) The revelation of the electronic structure and topological nature of the half-Heusler compounds can further provide important guidance for tailoring the band structures and engineering novel topological material/phases. For example, by properly tuning the Fermi level, we could move E F to intersect the topological surface states and have the topological surface electrons contribute to the superconductivity pairing, thus achieve one possible scenario of topological superconductivity. In addition, there have been theoretical works demonstrating that half-Heusler compounds will become a topological insulator with the TSS sitting in the bandgap under uniaxial strain (D. Xiao et al., Phys. Rev. Lett. 105, 096404(2010)); or become Weyl semimetals under compressive strain (J. W. Ruan et al., Nat. Commun. 7:11136 (2016)).
Therefore, our identification of the non-trivial topology in half-Heusler compounds, for the first time, is an important milestone which will provide a solid foundation and further encourage the exploration for novel topological phases in this exciting big family of materials.
Below we would like to respond to the reviewer's technical comments in details: (1) The compounds are actually metallic, namely, they are NOT topological insulators. In principle, in the absence of gap, the Z2 topological property is ill-defined (except topological metal, where surface Fermi arcs are expected). In this case, the so-called topological surface states, if any, are not protected.
Authors' response: Although Heusler compounds (and their sister compound HgTe) are indeed gapless in the bulk band structure, they have been demonstrated to be topologically equivalent to topological insulators, as is commonly accepted. As an example, there have been theoretical work (e.g. PRL 105, 096404 (2010)] that showed the Z2 index can be well defined as long as an infinitesimal gap opens due to weakly symmetry breaking. Therefore, topological surface states are indeed protected by the Z2-type inverted bulk band structure.
(2) The surface states observed at Fermi levels are actually all trivial. The only claimed Dirac cone (TSS) is located well below Fermi level. This is again unattractive, because topological surface states are interesting only if they are located within a band-gap and crossing the Fermi level. Dirac cone type surface states away from Fermi level may widely exist in most of materials (if not all).

Authors' response:
We do not agree with the reviewer on this point, and guess that the reviewer's comment is about the topological insulators -if the reviewer considers that the topological surface states are only interesting when they are in the bulk gap -then he/she is dismissing all the importance and exciting recent researches on topological Dirac and Weyl semimetals discovered recently which do not have a bulk gap.
Indeed, this work is about the topological semimetals, and that the TSSs are below the Fermi energy does not necessarily mean they are unattractive or uninteresting. As we have stated before, the existence of TSSs is originated and protected by the nontrivial topology, thus holding all the important properties (e.g., spin-momentum locking), similar to the TSSs that locate inside the band-gap of topological insulators. Moreover, in addition to the TSSs, the non-TSS surface states also exhibit rather interesting Rashba-type spin splitting, which is appealing for spintronics applications. In fact, both TSSs and non-TSS surface states in these half-Heusler compounds demonstrate attractive physics and exotic properties, making topological Heulser compounds a rich platform to test fundamental theory and explore future applications.
Finally, we would like to stress that the main message of the manuscript is not only limited to the discussion on the fascinating properties of either TSS or other nontopological surface states. Rather, by showing the existence of TSS, we established the topological non-trivial nature of the half-Heusler compounds we measured, which is essential for the understanding of many novel properties recently discovered (e.g., superconductivity with unconventional pairing and nodal gap structure, chiral anomaly and large magnetoresistance). Also, the complete band structures obtained in this work can also serve as a guide for tailoring the bandstructure for better properties (e.g. our measurement suggest that if the sample is p-doped, one could shift E F to intersect the TSS, etc.).
(3) The above two weak points make the compounds "boring", although their data are beautiful. This fact also weaken the possible connection, as claimed by authors, between the topology and the specially interesting non-centrosymmetry as well as superconductivity in this family of compounds. In general, we expect the superconductivity is related to the Fermi level and Fermi surfaces (which are all trivial in this case as observed).