Learning aerodynamics with neural network

We propose a neural network (NN) architecture, the Element Spatial Convolution Neural Network (ESCNN), towards the airfoil lift coefficient prediction task. The ESCNN outperforms existing state-of-the-art NNs in terms of prediction accuracy, with two orders of less parameters. We further investigate and explain how the ESCNN succeeds in making accurate predictions with standard convolution layers. We discover that the ESCNN has the ability to extract physical patterns that emerge from aerodynamics, and such patterns are clearly reflected within a layer of the network. We show that the ESCNN is capable of learning the physical laws and equation of aerodynamics from simulation data.


Predicting lift coefficient with neural networks
Calculating the airfoil lift coefficient is one of the most critical tasks in aerodynamics. It is generally achieved by using traditional Computational Fluid Dynamics (CFD) methods, which are often known for being computational expensive. We propose a neural network based model: Element Spatial Convolution Neural Network (ESCNN), to efficiently predict the airfoil lift coefficient 29 .
ESCNN is an end-to-end neural network that takes the airfoil coordinates x j , y j and angle of attack α as input, and gives the lift coefficient C l as output. The ESCNN architecture is straightforward. As shown in Fig. 1, it consists of two standard convolution layers followed by non-linear activation function, and a fully connected layer before the prediction 29 . The two convolution layers use 200 convolution filters, where the filter size of first layer is 5 × 1 and the second layer is 1 × 1 respectively. The last layer, fully connected layer, contains 159 neurons. Note that since the ESCNN takes sequential airfoil coordinates as input, and only convolutions and activations are involved in the architecture, therefore all the hidden layers are not permutation invariant.
The airfoil data samples are taken from the database of UIUC Applied Aerodynamic Group 30 , which covers a wide range of airfoil types from real-world designs. Each airfoil is represented by 160 points in x, y format. For the flow conditions, the Mach number Mach and Reynolds number Re are fixed to 0.3 and 1.3 × 10 7 respectively, and the angle of attack α varies from −2 to 10 • to avoid flow separation (laminar flow). The ground truth of C l is computed with the CFD solver Xfoil 31 , for each angle of attack at the specified flow condition for each airfoil geometry. In total 15678 samples are generated to create the final dataset. The whole dataset is divided into training set and validation set at the ratio of 8:2. Note that in laminar flow where the dataset is generated, the lift coefficient C l varies linearly with the angle of attack α , however the relationship between C l and the airfoil coordinates is highly-nonlinear 32 . Table 1 shows the performance benchmarks of different models, where the regression models are provided as baselines, and three types of NNs are compared. Experiments are implemented on the Pytorch and MindSpore open-source deep learning frameworks. The Multilayer Perceptron (MLP) network model has three hidden layers, where each layer contains 256,128,128 neurons respectively. The AeroCNN is a recent NN model that achieves state-of-the-art prediction accuracy 33 . In the AeroCNN framework, the airfoils are processed as images, such that AeroCNN can adopt the typical convolution neural network architecture for image recognition 33 . The error ε is the relative error defined by Eq. (1), where Ĉ l denotes the predicted lift coefficient and C l denotes the ground truth lift coefficient.
It is noted that even the baseline regression models can bring the error down below 10%. However, further reducing the prediction error is a difficult task: the model's learning capacity must be large enough to accurately approximate the non-linear relationship between C l and airfoil coordinates. Generally, the error can be further  www.nature.com/scientificreports/ reduced at the cost of adding more parameters to increase the model's complexity 34 , provided sufficient training data. However, the cost becomes more significant as the error gets smaller: the parameters are doubled to reduce the error from 5.58% (Multilayer Perceptron) to 3.46% (AeroCNN). In contrast, the ESCNN achieves 0.97% error with two orders of parameters less than AeroCNN. Figure 2 shows examples of the ESCNN prediction performance on the validation set, where the predictions are accurate and the C l − α linearity is well captured. With such few parameters and limited capacity, how can ESCNN perform so well in a high-dimensional and highly-nonlinear aerodynamic system?

Learning the Kutta condition
While investigating how ESCNN learns to make prediction, the fully connected layer, as shown in Fig. 1, draws our attention, since it is the last hidden layer before prediction.
We track the neuron values of the fully connected layer with a test airfoil at 3 • angle of attack during the training. The test airfoil NACA 2412 contains 160 coordinates. Variations of the neuron values [f 1 , f 2 , . . . , f n ] at the end of different training stages are shown in Fig. 4, where the x-axis denote the sequential index number [1, 2, . . . , n] of the neurons, and the y-axis denote corresponding neuron values [f 1 , f 2 , . . . , f n ] . Since the ESCNN model takes airfoil coordinates that are sequentially formatted as input, the order of coordinates is reflected in the neurons of learned hidden layers. The epoch numbers represent different training iterations, the network learns and evolves with the increasing of training epoch numbers. From the figure, it is obvious that as the training progresses, the fully connected layer is converging to a sine-shaped pattern, and this pattern begins to stabilize at around 400 epochs as shown in Fig. 4d. This result is also consistent with the learning curve that is shown in Fig. 3, as in this figure, after 400 epochs, the fitted Mean Squarer Error (MSE) converges towards a constant.
We notice an interesting phenomena, as shown in Fig. 4, the first and the last neurons values are always the same ( f 1 = f n ) even at the very beginning of training. It seems like the ESCNN has learned, early in the training process, of the condition f 1 = f n , and followed this rule strictly during the training progress. This f 1 = f n pattern reminds us the fundamental principle in aerodynamics: Kutta condition.
Recall the Kutta condition 32 : in fluid flow around a body with a sharp corner, the flow pattern in which fluid approaches the corner from both directions meets at the corner before flows smoothly go away from the body,  Apparently, ESCNN manages to figure out the importance of the Kutta condition by itself-it prioritizes to keep the value of the first neuron and the last neuron the same during the entire learning process.

Learning the vortex strength distribution pattern
To further investigate the meaning of the sine-shaped pattern, we compute the vortex strength distribution over airfoil using Vortex Panel Method (VPM). VPM is an engineering numerical method to compute the vortex strength distribution over airfoil. It replaces the airfoil surface with a series of vortex panels. Figure 6 shows the scaled vortex strength distribution calculated by VPM over the test case NACA 2412 airfoil at 3 • angle of attack. As shown in this figure, when fewer panels are used in the calculation, the distributions of vortex strength , γ , are not smooth. The oscillations from one panel to another is a well-known flaw of VPM which is triggered by the numerical inaccuracy 32 . With the larger number of panels used during the calculation, the oscillations fades away, the results get more accurate, and the sine-shaped pattern of vortex distribution gradually emerges. Figure 7 compares the computed 160 panels vortex strength distribution with the converged layer pattern at the same scale, both curves show a similar sine-shaped pattern, except they are symmetric in relation to the horizontal axis. Furthermore, if we down-sample the number of input airfoil coordinates to fewer dimensions, the converged sine-shaped pattern are still kept by the trained network, as shown in Fig. 8, not subject to the numerical inaccuracy of VPM.
Both the converged layer pattern and the computed VPM vortex strength distribution exhibit a similar sineshaped wave. One possible hypothesis is that the network has learned another physical concept that is a function of the vortex strength. Although there is no aerodynamics law at the moment that theoretically addresses this   www.nature.com/scientificreports/ sine-shaped pattern, we surmise that the distribution of vortex strength over airfoil follows a certain underlying pattern.

Learning with ReLU activation
The initial activation function we use to train the ESCNN model is the LeakyReLU, a common choice for neural networks 35 . More importantly, the output domain of LeakyReLU activation can be both positive and negative values, so is the value of vortex strength. This feature enables ESCNN to learn the vortex related physical quantity.   www.nature.com/scientificreports/ However, what if the network is trained with a ReLU activation function, can it still learn the vortex related physical quantity? In this section, we intentionally limit the value range of the fully connected layer by implementing another activation-ReLU function 35 . Equations (3) and (4) describes the LeakyReLU and ReLU activation function respectively, where k = 0.5 is the negative slope coefficient. For any real-valued input, the LeakyReLU outputs both positive and negative values, whereas ReLU outputs non-negative values only 35 . Figure 9 shows that the ESCNN still learns the Kutta condition and vortex distribution pattern, although the pattern is constrained within the non-negative range. More interestingly, the fully connected layer is forcing itself to evolve into a symmetric pattern as the training goes on, as marked in the dotted rectangular in Fig. 9. Despite that there is no constraint placed in the positive domain, the evolution to such symmetric pattern is quite clear in both Figs. 4 and 9.

Learning the lift coefficient equation
The ESCNN contains standard convolution layers, how can it outperform existing neural networks with significantly less parameters? The key reason of success is that the ESCNN makes prediction by learning the lift coefficient equation of aerodynamics at the last layer. For an airfoil at a fixed angle of attack, the lift coefficient C l is given by Eq. We calculate the correlation coefficient between the neuron values [f 1 , f 2 , . . . , f n ] and the vortex strength over airfoil [γ 1 , γ 2 , . . . , γ n ] , for each sample in the testing dataset. The correlation coefficient cov is defined by Eq. (7), where x and y are both vectors. Results show that the neuron values and vortex strength of all testing samples are highly correlated: the average of correlation coefficient is −0.975 , with a standard deviation of 6.4 × 10 −3 , an example is shown in Fig. 7.    www.nature.com/scientificreports/ We also calculate the correlation coefficient between the length of each panels [l 1 , l 2 , . . . , l n ] and the learned weights [w 1 , w 2 , . . . , w n ] . Note that the learned weights are fixed after training, however, the length of each panels [l 1 , l 2 , . . . , l n ] varies across different airfoils. The panels of an airfoil refer to the segments between two sequential points, as shown in Fig. 10. The panels are serially numbered from 1 to n according to their locations on the airfoil surface (starting from the trailing edge, along the upper surface to the leading edge and back around the lower surface to trailing edge). The length of each panel l i is the spatial distance between two sequential coordinates as defined by Eq. (8). Figure 11 shows the randomly sampled airfoils, and their corresponding panel length distribution. Figure 12 compares the learned weights and the panel length distribution of example airfoils at the same scale. Despite that panel length distributions are different across airfoils, they are close to symmetric. This symmetric pattern explains why the fully connected layer evolves for symmetry during training as shown in Figs. 4 and 9.
The learned weights are not as highly correlated with the panel length distribution, as compared with the correlation between neuron values and vortex strength. However, the symmetric pattern of panel length distribution has been captured by the learned weights. Moreover, we notice from Table 2 that the correlation between the panel length and learned weights affects the prediction performance: higher correlation leads to lower prediction error. Different airfoils have different panel length distributions, whereas the learned weights are fixed, failing to match the weights for corresponding airfoil leads to the increase of prediction error.
How does ESCNN manages to learn the physical quantity of vortex strength and the panel length? The reason is that ESCNN adopts a structured input that incorporates prior physical knowledge.
In the vortex panel method, the vortex strength γ 1 to γ n are obtained by solving a linear system of n equations, and solving each equation requires the angle of attack α and the coordinates of corresponding panel 32 . Inspired by the vortex panel method, we combine the panel coordinates x i , y i , x i+1 , y i+1 and the angle of attack α as an element unit, and then concatenate all the element units into a single vector, as shown in Fig. 1. The first layer, CONV1, performs 1D convolution over each element unit, and each element unit contains sufficient information to solve the vortex strength and panel length.
The physics-informed structured input allows the convolution layer to pick up the vortex strength γ i and panel length l i , and further allows the fully connected layer to learn the lift coefficient equation.

Conclusions
In this work, we propose a neural network model that outperforms existing state-of-the-art NNs in the airfoil lift coefficient prediction task, with two orders of less parameters. We further investigate how the ESCNN makes accurate predictions.
The ESCNN network learns to constrain the first and last neurons to be equal, during the entire learning process. This is the evidence that it self-learns the fundamental aerodynamics principle, the Kutta condition. Moreover, the fully connected layer converges to a sine-shaped wave pattern that is highly correlated to the vortex strength distribution over airfoil, which demonstrates that the network learns the vortex related physical quantity. In addition, we explore ESCNN's learning ability with constrained activation by replacing LeakyReLU to ReLU function. The results show that even with a limited value range of the neurons, ESCNN can still learn the critical physics of Kutta condition and the vortex distribution pattern. In the end, we show that the network learns lift coefficient equation at the last layer.     www.nature.com/scientificreports/