Introduction

In the newly discovered high superconducting transition temperature (T c ) iron-based family, the FeSe superconductors possess the simplest crystalline structure but attract much attention owing to multifarious physical properties.1,2,3,4 The T c of bulk FeSe crystal is as low as 8 K, while it can be considerably enhanced to above 37 K under high-pressure,5 electric field gating,6 or insetting the intercalation layer.7 Particularly, in the monolayer FeSe on SrTiO3 a dramatically high T c above the liquid point of nitrogen was observed,8 which offers the possibility of breaking the record as those of cuprate family. The origin for the enhancement of T c is still an open question, while a common consensus has been proposed as an accompaniment to the modification of the Fermi surface. Therefore, studying on the electronic state of the FeSe system provides a perfect arena to understand the high-T c mechanism.

Different from the conventional superconductors, competing electronic orders such as unidirectional charge density wave and nematic order exist in both cuprate and iron-based superconductors. Among these, the nematic electronic order demonstrates a spontaneous symmetry breaking from C4 to C2 symmetry (the order parameters remain invariant under the inversion, the D4h group can be viewed as C4), which has been generally considered as a strong correlation with the fundamental unsolved electronic issue in Fe-based superconductors, especially in recent work on the FeSe system.9,10 For the FeSe bulk crystals, the structural transition from tetragonal to orthorhombic occurs at T s  = 90 K, while the anisotropy of the electronic structure is not a consequence of the lattice distortion, but a result of the microscopic mechanism such as spin fluctuation or orbital ordering. Researches on the nematic order in iron-based superconductors have generally supported the spin-fluctuation origin. However, because of the absence of long-range magnetic order in the FeSe system, orbital ordering is probably the origin for the electronic transition. Moreover, recent angle-resolved photoemission spectroscopy (ARPES) results showed the emergence of the nonequivalent energy shifts of xz/yz orbital bands below T s ,10,11 implying the orbital origin of the structural transition.

Furthermore, similar to the nematic order, the superconducting pairing symmetry strongly relates to detailed electron–electron interaction. To be specific, the orbital order with inter-orbital electron–electron interactions would favor a sign-preserving s-wave pairing, while spin fluctuation with intra-orbital interaction for a sign-changing s±-wave or d-wave. Recent ARPES and scanning tunneling microscopy (STM) results suggested the sign-changing pairing symmetry such as s±- wave or d-wave in FeSe, implying that the magnetic fluctuations may still assist the superconducting pairing.12,13 With the high-resolution STM measurement, the elliptical vortices have been directly observed at superconducting state on the FeSe bulk samples,14 for which the extremely weak structure distortion (~0.5%) can hardly induce such pronounced anisotropy, while the nematic order and superconducting order parameters are expected to play the important roles. Ginzburg–Landau (GL) theory firstly can offer a phenomenological way to investigate the vortices in superconductors with the s-wave symmetry. The GL theory itself can be derived exactly from the microscopic BCS theory.15 By means of the Gorkov’s derivation and symmetry analysis, the GL theory has been generalized into several pairing symmetries such as s + id,16,17,18,19,20 p-wave,21 and so on. Among these symmetry models the s + id, where the extended s±-wave competes with the d-wave pairing order, is generally used to investigate the iron-based superconductors.22,23,24

In this work, we construct the GL type free energy which contains the nematic order, s-wave and d-wave superconducting orders with up to 4th order interactions. The time-dependent GL (TDGL) equation is derived from the free energy to describe the FeSe system. By implementing the open boundary condition, our simulation reveals the configuration and dynamics of the elliptical vortex and the nematic order. With the periodical boundary condition, the oblique vortex lattice rather than a triangular one is found. Our simulation results have a good agreement with the previous experiment in the configuration of the single vortex and the vortex lattice.14 The presence of the nematic order can break the symmetry from C4 to C2 and enhance the superconductivity. The symmetry allowed trilinear term will enhance the anisotropy of the superconducting order and induce the nearly degeneracy of s-wave and d-wave.25,26

Results

High-resolution STM and scanning tunneling spectroscopy experiments provide the possibility for further investigation on the single vortex, for which the vortices configuration can be reconstructed as well studied in various superconductors.27,28,29,30,31,32 In the previous work by Song et al.,14 the vortices and vortex lattice in the FeSe superconductors were directly observed, and the vortex core was found in an elliptical shape, where the stretched direction is along one of the Fe–Fe bonds.

With the open boundary condition, the interplay between the anisotropic vortices and finite geometric region are investigated in the present work. Although one can hardly observe the evolution and dynamics of the vortices and nematic order in realistic experiments, the real-time simulation results can provide an approach to understand the motion of the vortices. By solving the TDGL equations, the results show that nematic order breaks the symmetry from C4 to C2 during the evolution.

By using the periodic boundary condition, the vortex lattice is also investigated. Based on the simulation, the vortices favor an oblique lattice rather than the triangular lattice; this is due to the trade-off between the two-fold symmetry of the repulsive interaction and the closet packing. The simulation results are consistent with the experimental data.

Finite region and vortex configuration

Previous works suggest that the pairing symmetry is probably s± wave or d wave, they both can be described by the addition of the isotropic and the anisotropic superconducting order. The isotropic order parameter is coupled to the anisotropic order parameter by the interaction,16,18,33

$$F_{{\mathrm{int}}} = \frac{\gamma }{2}\left( {{\mathrm{{\Pi}}}_x\psi _s{\mathrm{{\Pi}}}_x^ \ast \psi _d^ \ast - {\mathrm{{\Pi}}}_y\psi _s{\mathrm{{\Pi}}}_y^ \ast \psi _d^ \ast + c.c.} \right),$$
(1)

where the two complex fields ψ s , ψ d stand for the s-wave component and d-wave component in the mixed superconducting order, and γ is the coupling constant. Π = (−iħ∇ − e*A) is the gauge invariant derivative, ∇ is the del operator, A is the magnetic vector potential, and e* = 2e is the charge of the superconducting charge-carriers, where e is the electron charge. This term is invariant under rotation of π/2, when taking the integration by parts,

$$\psi _s\left( {{\mathrm{{\Pi}}}_x^2 - {\mathrm{{\Pi}}}_y^2} \right)\psi _d^ \ast + c.c.$$
(2)

and thus,

$$\begin{array}{*{20}{l}} {\psi _s\left( {{\mathrm{{\Pi}}}_x^2 - {\mathrm{{\Pi}}}_y^2} \right)\psi _d^ \ast + c.c. \to } \hfill \\ {\psi _s\left[ { - \left( {{\mathrm{{\Pi}}}_x^2 - {\mathrm{{\Pi}}}_y^2} \right)} \right]\left( { - \psi _d^ \ast } \right) + c.c. \to \psi _s\left( {{\mathrm{{\Pi}}}_x^2 - {\mathrm{{\Pi}}}_y^2} \right)\psi _d^ \ast + c.c.} \hfill \end{array}$$
(3)

The above type of anisotropic interaction is used in our model. Besides the mixed superconducting order, a real field ϕ stands for the nematicity order, which competes with the mixed superconducting order. Because the higher order terms are negligible, the free energy which is up to 4th order can be described as

$$\begin{array}{*{20}{l}} {f = f_s + f_d + f_\phi + f_{{\mathrm{int}}}} \hfill \\ {f_{s,d} = - \alpha _{s,d}\left| {\psi _{s,d}} \right|^2 + \frac{{\beta _{s,d}}}{2}\left| {\psi _{s,d}} \right|^4 + \frac{1}{{2m_{s,d}^ \ast }}\left| {{\mathrm{{\Pi}}}\psi _{s,d}} \right|^2} \hfill \\ {f_\phi = - \alpha _\phi \phi ^2 + \frac{{\beta _\phi }}{2}\phi ^4 + \frac{1}{{2m_\phi }}\left| {\nabla \phi } \right|^2} \hfill \\ {f_{{\mathrm{int}}} = \gamma _1\left| {\psi _s} \right|^2\left| {\psi _d} \right|^2 + \gamma _2\left( {\psi _s^{ \ast 2}\psi _d^2 + c.c.} \right)} \hfill \\ { + \gamma _3\left( {{\mathrm{{\Pi}}}_x\psi _s{\mathrm{{\Pi}}}_x^ \ast \psi _d^ \ast - {\mathrm{{\Pi}}}_y\psi _s{\mathrm{{\Pi}}}_y^ \ast \psi _d^ \ast + c.c.} \right)} \hfill \\ { + \lambda _1\phi \left( {\psi _s^ \ast \psi _d + c.c.} \right) + \lambda _2\phi ^2\left| {\psi _s} \right|^2 + \lambda _3\phi ^2\left| {\psi _d} \right|^2} \hfill \end{array}$$
(4)

where α i , β i are the parameters describing the Landau phase transition and i = s, d and ϕ. Considering T s  < T d  < T ϕ , the α i submits to α s  < α d  < α ϕ . γ j (j = 1, 2, 3) is the coupling constant between s-wave and d-wave components, and λ k (k = 1–3) is the coupling constant between the superconducting order parameters and the nematic order. \(m_i^ \ast\) (i = s, d) is the mass of the superconducting charge-carriers, and the microscopic electron pairing theory of superconductivity implies that \(m_i^ \ast\) = 2 m i , where m i is the electron mass. m ϕ represents the effective mass of the nematic order.

Instead of directly making the isotropic order parameter coupled to the nematic order, our model (Eq. (4)) suggests that the nematic order triggers off the mixture of s-wave and d-wave components, and the anisotropic interaction between s-wave and d-wave components causes larger anisotropy. Meanwhile, the trilinear term \(\lambda _1\phi \left( {\psi _s^ \ast \psi _d + c.c.} \right)\) will enhance the anisotropy. Besides, according to the previous mean-field analysis, this term may induce nearly degeneracy of s-wave and d-wave, which was also supported by the spin-fluctuation model.25,26 This is different from the p-type 122-system, where the degeneracy of s-wave and d-wave is due to the close critical temperatures.34

In the following simulation, the phase transition parameters in the Eq. (4) are set to be α s  = 1.0, α d  = 1.5, α ϕ  = 2.0, β i  = 1, i = s, d, and ϕ, which are based on the superconductivity and nematicity transition temperature.9,12,35 The coupling constant of the anisotropic interaction is set as γ3 = 0.2, while other coupling constants are set to be 0.4 based on the consideration of convergency. λ2, λ3 are negative. The effective masses are set to be m s  = 1, m d  = 2, m ϕ  = 4.

The initial condition is quite important for convergency of the non-linear partial differential equations, though different initial conditions will arrive at the same stable state in this type partial differential equations. The initial states of the complex order parameters are set to be proportional to \(\psi _0 = \frac{{x - x_0 + i\left( {y - y_0} \right)}}{{\sqrt {\left( {x - x_0} \right)^2 + \left( {y - y_0} \right)^2} }}\). For the s-wave component, the initial state is ψ0, while it is relatively small for the d-wave component. The initial state of the nematic order is set to be 0.5, and all the vector potentials are set to be 0.

Evolution of the vortices and the nematic order

The vortex core can be visualized by atomically resolved STM measurements. Moreover, by applying the specific periodical magnetic field, it is possible to view the motion of the vortex.36 However, it is still difficult to conduct real-time observation on the evolution and dynamics of the vortices. Time-dependent simulation can be applied to simulate how the vortices generate from the boundary and how they move and interact with each other. Through the real-time simulation, deep understanding of the vortex dynamics can be achieved and further applications can be simulated, such as modification of the sample to enhance the critical current.

Different from the other type-II superconductors, the elliptical vortices in FeSe sample is related to the C4 → C2 symmetry breaking, where the nematic order plays an important role on enhancing the symmetry breaking in the superconducting state.37 With the real-time simulation, the transition from C4 symmetry to C2 symmetry is revealed (see Supplementary Videos).

The real-time evolution of the elliptical vortices is shown in Fig. 1. At the beginning, the pattern of the s-wave component shown in Fig. 1a.1 is quite similar to that in typical type-II superconductors,38,39,40,41 the magnetic field penetrates into the sample from the edge. Due to the Bean–Livingston barrier,42,43 the vortices cannot immediately get into the sample. The interaction of intra-vortices is qualitatively repulsive and the vortices could only locate along the edges.

Fig. 1
figure 1

Evolution of superconducting orders and nematic order in a 1.5λ × 1.5λ region (see Supplementary Videos). a 1–7 is s-wave components \(\left| {\psi _s} \right|^2\), b 1–7 is d-wave components \(\left| {\psi _d} \right|^2\), and c 1–7 is the nematic order ϕ2. The legend shows the magnitudes of the order parameters. Each 1–7 corresponds to t = 0.1, t = 0.7, t = 2, t = 5, t = 9, t = 30, respectively

Full size image

The enhanced magnetic field will force the vortices to penetrate into the sample. As a result, Fig. 1a2 demonstrates the arrangement of four vortices. The system will gradually achieve its minimum free energy by rearranging the vortices. However, with the nematic order competing with the mixed superconducting order, the situation is quite different. Figure 1a3–6 depicts the intermediate stage, where the C4 → C2 symmetry breaking happens, because the nematic order mixes the two superconducting order and enhances the anisotropic interaction. Meanwhile, the interaction terms λ1 and γ1,2 compete with each other, λ1 favors a large separation between s-wave and d-wave, while the other one favors small separation. Figure 1a7 demonstrates the equilibrium state of this system, the two vortices are both elliptical and repulse with each other in short range.

Different from the s-wave component, the initial state of the d-wave component is relatively small and then induced to be anisotropic by the nematic order. The vortices of d-wave component are slightly less eccentric due to its four-fold tendency. Apart from the superconducting order, the nematic order shown in Fig. 1c1–7 exhibits strong C2 symmetry at first. It then has four-fold symmetry due to the interaction with s-wave component. The final state shown in Fig. 1c.7 has C2 symmetry and is less eccentric than the superconducting order.

Vortex lattice

Vortices arrange into the vortex lattice in an infinite region. Though it is impossible to simulate the complicated TDGL equations in an infinite region, simulation on the single unit cell with periodic boundary condition can make the investigation of the vortex lattice possible.44,45,46 By extending the unit cell according to the periodicity, the vortex lattice can be recovered.

Defining the ratio of the side lengths of the rectangular unit cell as r, \(r = \sqrt 3\) corresponds to triangular lattice which is typical for most type-II superconductors, as shown in Fig. 2a. Early study reported that the oblique vortex lattice,16,44,45,47 \(r < \sqrt 3\), for s + id model costs less energy than the triangular one which is ascribed to the four-fold symmetry of the system. To be specific, the system tends to preserve the four-fold symmetry while the closet packing between the vortices leads the vortex lattice to be triangular. As a result, the vortex lattice favors the oblique one by making the trade-off between preserving the four-fold symmetry and the triangular lattice. While in our case, the nematic order breaks the C4 symmetry to C2, and thus the system finds the balance between the two-fold symmetry and the triangular lattice. Therefore, it favors the lattice with \(r > \sqrt 3\).

Fig. 2
figure 2

The vortex lattice (a) is s-wave component \(\left| {\psi _s} \right|^2\), b is d-wave component \(\left| {\psi _d} \right|^2\) and c is the nematic order ϕ2. d–f are the vortices in FeSe sample under different applied magnetic fields in experiment (adapted from ref. 14). The ratio r is defined as the separation of two vortices along the x direction divided by that along the y direction

Full size image

In the simulation, the normalized magnetic field is set as 4π, which allows two vortices in the one unit cell. The area of the unit cell is set to be 16λ2. By varying the ratio r, the minimum energy density f = F/L x /L y is achieved at r = 2.81, as shown in Fig. 2. It is hard to realize the real-time detection in experiments and compare the vortex dynamics with the simulation, but the equilibrium state where the vortices form a stable vortex lattice can be compared with the simulation results. Based on the parameters provided above, the simulated oblique vortex lattice r = 2.81 is in agreement with the previous result r ~ 2.80 in the experiment where many vortices are observed in FeSe under applied magnetic field of 8 T.14

Discussion

Analytical treatment of the anisotropy

According to the previous simulation results, the anisotropy of the interaction serves to the elliptical shape of the vortices. Since the London penetration depth is considerably large than the coherence length \(\lambda \gg \xi\), the coupling to the electromagnetic field can be neglected. Given that the variation of the nematic order is small, and the vortices experience a uniform nematic order away from the origin, thus nematic order is set to be a stationary field, ϕ = ϕ0, and satisfy ϕ0 → −ϕ0 under the rotation of π/2. The equations of the superconducting order are,

$$\begin{array}{*{20}{l}} \begin{array}{l}\left( {\lambda _2\phi _0^2 - \alpha _s} \right)\psi _s + \beta \left| {\psi _s} \right|^2\psi _s + \left| {\psi _d} \right|^2\left( {\gamma _1\psi _s + 2\gamma _2\psi _s^ \ast } \right)\\ + \gamma _s\left( {\partial _x^2 + \partial _y^2} \right)\psi _s + \gamma _3\left( {\partial _x^2 - \partial _y^2} \right)\psi _d + \lambda _1\phi _0\psi _d = 0\end{array} \hfill \\ \begin{array}{l}\left( {\lambda _3\phi _0^2 - \alpha _d} \right)\psi _d + \beta \left| {\psi _d} \right|^2\psi _d + \left| {\psi _s} \right|^2\left( {\gamma _1\psi _d + 2\gamma _2\psi _d^ \ast } \right)\\ + \gamma _d\left( {\partial _x^2 + \partial _y^2} \right)\psi _d + \gamma _3\left( {\partial _x^2 - \partial _y^2} \right)\psi _s + \lambda _1\phi _0\psi _s = 0\end{array} \hfill \end{array}$$
(5)

Where \(\gamma _{s,d} = \frac{{\hbar ^2}}{{2m_{s,d}}}\). A direct observation on Eq. (5) reveals that λ2, λ3 and \(\phi _0^2\) will change the critical point of the phase transition in this system, as the minimum point of the potential is at ψs,min = ±\(\sqrt {\left( {\alpha _s - \frac{{\lambda _2}}{2}\phi _0^2} \right){\mathrm{/}}\beta _s}\), ψd,min = ±\(\sqrt {\left( {\alpha _d - \frac{{\lambda _3}}{2}\phi _0^2} \right){\mathrm{/}}\beta _d}\). The coefficients of the term ψ s and ψ d are non-zero and different for most situations. Because λ2, λ3 are negative, the presence of the nematic order will enhance the superconductivity.48

The interesting question is that the λ1 term will break the rotation symmetry. When rotating π/2, the λ1 term picks up a different sign compared with other terms. Such term enhances the anisotropy of the vortices.

Polynomial terms in Eq. (5) do not contribute to the anisotropy but the gradient terms and the nematic order will. Without losing universality, we set γ s  = γ d  = γ. Eq. (5) can be reformulated as,

$$\left( {\gamma \left( {\partial _x^2 + \partial _y^2} \right) + \gamma _3\left( {\partial _x^2 - \partial _y^2} \right)} \right)\psi + \lambda _1\phi _0\psi + P\left( {\psi _s,\psi _d} \right) = 0$$
(6)

where ψ = ψ s  + ψ d , P is the polynomial of ψ s and ψ d . Ignoring the nematic order ϕ0, the solution is actually a elliptical vortex, which can be seen by transforming the gradient terms to a Laplacian under the coordinate transformation,

$$x^{\prime} \to \frac{1}{{\sqrt {1 + \gamma _3{\mathrm{/}}\gamma } }}x,y^{\prime} \to \frac{1}{{\sqrt {1 - \gamma _3{\mathrm{/}}\gamma } }}y$$
(7)

Due to Eq. (7), the coordinate is elongated along y direction and contracted along x direction as shown in Fig. 3. Therefore, the symmetry is broken into C2 once returning to the original coordinate.

Fig. 3
figure 3

The coordinate is elongated along y direction under the coordinate transformation

Full size image

However, the nematic order obeys ϕ0 → −ϕ0 under rotation of π/2, it is impossible to view x-direction and y-direction equivalently, because the nematic order offers a angle dependent term. To capture the feature, ϕ0 is set to be k(x2 − y2), where k is a constant. Figure 4 shows that turning on the nematic order makes the elliptical vortex more anisotropic.

Fig. 4
figure 4

Nematic order will turn on the λ1 interaction to enhance the anisotropy of the vortices. The profile of vortex a without and b turn on the λ1 interaction

Full size image

Comparison with other systems

The anisotropic electronic structure is normal in some high-T c superconductors. For instance, in the cuprate family, YBa2Cu3O7−δ (YBCO) has a tetragonal to orthorhombic phase transition at the under-doped level of oxygen, resulting in a symmetry breaking from D4h to D2h along the Cu–O chains.49 Here, the structure transition temperature is considerably higher than the superconducting transition temperature T c . The previous works on the YBCO have constructed the GL free energy which obeys D2h symmetry and found the elliptical vortex along the b-axis.50,51 In the present FeSe system, orthorhombic phase happens after a structural phase transition at 90 K. However, it is argued that the small crystalline distortion (~0.5%) itself cannot lead to such large anisotropy in electronic structure.14,52 Therefore, the nematic order may originate from the strongly interacting fermion system, where the small symmetry breaking term will be promoted to finite magnitude when lowering the energy scale, say, temperature. Thus, different from the explicit symmetry breaking treatment in YBCO, the symmetry breaking nematic order in FeSe is temperature dependent and should be treated as a competing order which competes with the D4h superconducting order.24 On the other hand, the elliptical vortex core is along the Fe–Fe bonds, being considerably different from that along the b-axis in YBCO.

In some iron-based superconductors, the superconducting nematic transition temperature was also found below the superconducting transition temperature, such as p-type 122-system iron pnictide superconductors.34 The nematic superconducting state in this superconductor is induced by the small symmetry breaking term in the normal phase, which qualitatively differs from the ordinary nematicity observed in the orthorhombic structural phase. The superconducting state nematicity occurs just after the onset of superconducting transition and reveals an anisotropy shifted by π/4 from those of the normal nematic state.34 Meanwhile, the superconducting transition temperature resembles a low energy scale, hereby, the lower nematic transition temperature may correspond to an infrared phenomenon of the strongly interacting fermion system.

However, one can hardly identify which order occurs first. When the superconducting transition temperature T c is lower than the nematic transition temperature T n , it can be expected that the nematic order still exists in superconducting state and strongly interacts with the superconducting orders, which are shown in Fig. 5a. For the case of T c  > T n , there is an intermediate region T c  > T > T n , where the nematic order is absent in the superconducting state, and the profile of the vortex core is shown in Fig. 5b. Without the nematic order, the mixture of s-wave and d-wave makes the vortex slightly anisotropic. However, with the nematic order occurring below T n , the anisotropy of the superconducting vortex is enhanced. In the simulation, the high order correction term is added into the free energy.

Fig. 5
figure 5

a, b are the profiles of superconducting order (ψ s ) with and without nematic order

Full size image

Methods

We begin by constructing the GL type free energy. Intuitively, the nematic order competes with the mixed superconducting order parameters, where isotropic s-wave and anisotropic d-wave components are considered. We mainly focused on the 2-dimensional (2D) geometry due to the quasi-2D feature for the Fe-based superconductors, and the 2D GL free energy can mostly capture the ingredients in the high-T c superconductors. By taking the variation of the free energy, the TDGL equations are derived. The TDGL was constructed to investigate the dynamics of the vortices in the dirty limit where the penetration depth λ is greatly larger than the coherence length ξ.53 For the FeSe case, due to λ ~ 500 nm and ξ ~ 5 nm,54,55,56 the dirty limit construction of TDGL is valid. Numerical results of the TDGL equations with the open boundary condition, given by the finite element method, can reveal the shape, configuration and dynamic properties of the vortices. The numerical solutions with periodic boundary condition provide the vortex lattice in the equilibrium state. Our theoretical calculation results show the elliptical vortices and oblique vortex lattice, which are in agreement with the experiment.14 In the following formalism, the a-axis and b-axis, or x-axisand y-axis are defined along either of the Fe–Fe bond directions as shown in Fig. 6, for which the directions of the Fe–Fe bonds keep the symmetry of the nematicity.

Fig. 6
figure 6

The crystal structure of FeSe, where the x-axis and y-directions are defined as along the Fe–Fe bonds

Full size image

Free energy

Competing order such as nematicity can strongly interact with the superconducting order parameters. Chowdhury et al. investigated the anisotropic interplay between the competing order and the single isotropic superconducting order,23 and they argued that the different effective masses which are induced by the anisotropic interaction could lead to the anisotropic vortex. However, the pairing symmetry in FeSe is sign-changing s±-wave or d-wave suggested by the recent experiments.12,13 Meanwhile, the s + id model could suitably describe the iron-based superconductors.22 Here, the mixed superconducting order together with the anisotropic interaction can also cause an anisotropic vortex core, where the isotropic s-wave order parameter interacts with the anisotropic d-wave order parameter. The existence of nematic order will mixed the two superconducting order and significantly enhance the anisotropy. Thus, the nematic order can enhance the small anisotropic interaction between the superconducting orders to form an extremely elliptical vortex.

Previous work investigated the special trilinear term thoroughly. It turns out if the coupling constant λ1 = 0 or very small, the system may favor s + id symmetry, but for large nematic fluctuation, the intermediate state has s + d symmetry26 character. The free energy (Eq. 4), including both self-energy and interaction energy, remains invariant under the mirror reflection and the rotation of π/2.

$$x \to y,y \to - x,\psi _s \to \psi _s,\psi _d \to - \psi _d,\phi \to - \phi$$
(8)

However, the γ3 term in the interaction causes different effective masses along the two directions, and consequently, results in the anisotropic vortex cores and affects the arrangement of the vortices, namely, the vortex lattice. Based on the consideration of symmetry, up to 4th order, it is impossible to turn on the direct interaction between the nematic order and the anisotropic gradient term. Nevertheless, the nematic order can tune the stationary part of the superconducting orders in the free energy and let them mixed, thus the anisotropic interaction γ3 enhanced. The λ1 term will also enhance the anisotropy as explained in Section Analytical Treatment of the Anisotropy. Because the λ2, λ3 are negative, the presence of the nematic order will enhance the superconductivity, such as increasing the transition temperature.

Time-dependent ginzburg–landau equation

The TDGL equation can be obtained by taking the variation of the free energy as follow,

$$\begin{array}{*{20}{l}} {\frac{{\hbar ^2}}{{2m_iD_i}}\left( {\frac{\partial }{{\partial t}} + i\frac{e}{\hbar }{\mathrm{\Phi }}} \right)\psi _i = - \frac{{\delta f}}{{\delta \psi _i^ \ast }}} \hfill \\ {\frac{{\hbar ^2}}{{2m_\phi D_\phi }}\frac{{\partial \phi }}{{\partial t}} = - \frac{{\delta f}}{{\delta \phi }}} \hfill \\ {\sigma \left( {\frac{{\partial {\bf{A}}}}{{\partial t}} + \nabla {\mathrm{\Phi }}} \right) = - \frac{{\delta f}}{{\delta A}} - \frac{1}{{4\pi }}\nabla \times \nabla \times {\bf{A}}} \hfill \end{array}$$
(9)

where D i (i = s, d, ϕ) is the phenomenological diffusion coefficients, and Φ is the scalar potential of the electromagnetic field.

In the open boundary conditions, the superconductivity-vacuum boundary can be obtained directly from the variation of the free energy as

$$\begin{array}{*{20}{l}} {\left( {\frac{\hbar }{i}\left( {l_{x,i}\frac{\partial }{{\partial x}}\widehat {\bf{x}} + l_{y,i}\frac{\partial }{{\partial y}}\widehat {\bf{y}}} \right) - q{\bf{A}}} \right)\psi _i \cdot {\bf{n}} = 0} \hfill \\ {\nabla \phi \cdot {\bf{n}} = 0} \hfill \\ {\nabla \times {\bf{A}} = {\bf{B}}_{\bf{a}}} \hfill \\ {\left( {\frac{{\partial {\bf{A}}}}{{\partial t}} + \nabla {\mathrm{\Phi }}} \right) \cdot {\bf{n}} = 0,} \hfill \end{array}$$
(10)

where \(\widehat {\bf{x}},\widehat {\bf{y}}\) are the unit vector along x and y directions, respectively, and \(l_{x,i} = 1 + \frac{{m_i}}{{m_{sd}}}\) and \(l_{y,i} = 1 - \frac{{m_i}}{{m_{sd}}}\) correspond to the different effective mass along x and y directions, respectively. Thus, the dynamics property of the superconducting order is different along the x and y axis.

To solve the TDGL equations numerically, the complicated TDGL equations are normalized by introducing,

$$\begin{array}{*{20}{l}} {\left( {x,y,z,t} \right) \to \left( {\lambda x^{\prime},\lambda y^{\prime},\lambda z^{\prime},\frac{{\xi ^2}}{D}t^{\prime}} \right),{\bf{A}} = \frac{\hbar }{{e\xi }}{\bf{A}}^{\prime}} \hfill \\ {\psi _i = \sqrt {\frac{{\alpha _i}}{{\beta _i}}} \psi _i^\prime \left( {i = s,d} \right),\sigma = \frac{1}{{\mu _0D\kappa ^2}}\sigma ^{\prime}} \hfill \end{array}$$
(11)

where the new quantities are labeled by prime, the spatial and temporal coordinates are scaled according to the λ and the \(\xi = \frac{\hbar }{{\sqrt {2m_s\alpha _s} }}\), the GL parameter is defined as κ = λ/ξ and \(\kappa \gg 1\) for the FeSe system.54,55,56 The dimensionless form contains the gauge invariant derivative, \({\mathrm{{\Pi}}} = - \frac{i}{\kappa }\nabla - {\bf{A}}\).

The TDGL is invariant under the gauge transformation. Given an arbitrary function χ(x, y, z, t), and introduce the gauge transformation as,

$$\tilde \psi _i = \psi e^{i\kappa \chi },\widetilde {\bf{A}} = {\bf{A}} + \nabla \chi ,{\tilde{\mathrm \Phi }} = {\mathrm{\Phi }} - \frac{{\partial \chi }}{{\partial t}}.$$
(12)

Because of the extra degree of freedom, the gauge should be fixed to achieve the definite equations. For the sake of the simplicity, the scalar potential Φ can be eliminated by choosing the London gauge, let,

$$\frac{{\partial \chi }}{{\partial t}} = {\mathrm{\Phi }}$$
(13)

thus the TDGL equations are no longer dependent on the scalar potential Φ, and the electric field is \({\bf{E}} = - \frac{{\partial {\bf{A}}}}{{\partial t}}\).

The former procedure derives the TDGL equations and corresponding open boundary conditions based on the free energy. To implement the finite element method, the complex order parameters are decomposed into the real and imaginary part; the vector potentials are decomposed into x, y components. For the consequences of the non-linear feature, the mesh of the region is adaptively refined to achieve high accuracy. Solving TDGL is minimizing the total energy of the system, and the stable state will be reached after hundreds to thousands of the normalized time.

Periodic boundary condition

The open boundary condition describes the superconductor–vacuum boundary straightforward, therefore, it is useful to investigate the finite size solution. When coming to the infinite size, the periodic boundary condition should be introduced on each unit cell. The complex order parameter and the vector potential should be modified from one unit cell to another, which can ensure the gauge invariance.57 Two lattice vectors are used to characterize the lattice, namely, t1 and t1. The complex order parameters will pick up a phase while additional term should be added to the vector potential from one unit cell to another, namely,

$$\begin{array}{*{20}{l}} {\psi _i\left( {{\bf{x}} + {\bf{t}}_k} \right) = \psi _i\left( {\bf{x}} \right)e^{i\kappa g_k}} \hfill \\ {{\bf{A}}\left( {{\bf{x}} + {\bf{t}}_k} \right) = {\bf{A}}\left( {\bf{x}} \right) + \nabla g_k} \hfill \end{array}$$
(14)

where, \(g_k = - \frac{1}{2}\left( {{\bf{t}}_k \times B{\bf{k}}_3} \right) \cdot {\bf{x}}\), k = 1 and 2, and B is quantized by \(B = \frac{{2\pi n}}{{\kappa \left| {\mathrm{\Omega }} \right|}}\), in which \(\left| {\mathrm{\Omega }} \right|\) is the area of the unit cell and n is integer.

The explicit form for a rectangular unit cell is,

$$\begin{array}{*{20}{l}} {\psi _i\left( {L_x,y} \right) = \psi _i\left( {0,y} \right)e^{\frac{{i\phi y}}{{2L_y}}}} \hfill \\ {A_x\left( {L_x,y} \right) = A_x\left( {0,y} \right)} \hfill \\ {A_y\left( {L_x,y} \right) = A_y\left( {0,y} \right) + \frac{\phi }{{2\kappa L_y}}} \hfill \\ {\psi _i\left( {x,L_y} \right) = \psi _i\left( {x,0} \right)e^{\frac{{ - i\phi x}}{{2L_x}}}} \hfill \\ {A_x\left( {x,L_y} \right) = A_x\left( {x,0} \right) - \frac{\phi }{{2\kappa L_x}}} \hfill \\ {A_y\left( {x,L_y} \right) = A_y\left( {x,0} \right)} \hfill \end{array}$$
(15)

where, i = s and d, ϕ = 2nπ is the reduced vortex flux, L x and L y characterize the size of the unit cell. The variation of the vector potential is neglected, due to \(\kappa \gg 1\).

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.