Prediction of a typhoon track using a generative adversarial network and satellite images

Tracks of typhoons are predicted using a generative adversarial network (GAN) with satellite images as inputs. Time series of satellite images of typhoons which occurred in the Korea Peninsula in the past are used to train the neural network. The trained GAN is employed to produce a 6-hour-advance track of a typhoon for which the GAN was not trained. The predicted track image of a typhoon favorably identifies the future location of the typhoon center as well as the deformed cloud structures. Errors between predicted and real typhoon centers are measured quantitatively in kilometers. An averaged error of 95.6 km is achieved for tested 10 typhoons. Predicting sudden changes of the track in westward or northward directions is identified as a challenging task, while the prediction is significantly improved, when velocity fields are employed along with satellite images.

prediction of a typhoon track using a generative adversarial network and satellite images Mario Rüttgers , sangseung Lee , soohwan Jeon & Donghyun You tracks of typhoons are predicted using a generative adversarial network (GAN) with satellite images as inputs. time series of satellite images of typhoons which occurred in the Korea peninsula in the past are used to train the neural network. The trained GAN is employed to produce a 6-hour-advance track of a typhoon for which the GAN was not trained. the predicted track image of a typhoon favorably identifies the future location of the typhoon center as well as the deformed cloud structures. Errors between predicted and real typhoon centers are measured quantitatively in kilometers. An averaged error of 95.6 km is achieved for tested 10 typhoons. Predicting sudden changes of the track in westward or northward directions is identified as a challenging task, while the prediction is significantly improved, when velocity fields are employed along with satellite images.
Every year tropical cyclones cause death and damage in many places around the world. Cyclones are formed when water at the sea surface becomes warm, evaporates, and rises in a form of clouds, and while it cools down, the condensation releases strong energy in a form of winds. Rotation of the earth gives a cyclone its spinning motion. At the center usually a hole forms, which is called an eye of the cyclone. At the eye, the pressure is low and energetic clouds and winds get attracted. Warm air can rise through the hole and, favored by high altitude winds that can create a suction effect, increase the energy of the system. Crosswinds, vertical wind shear or dry air, on the other hand, can block this mechanism 1 . Depending on water conditions and surrounding winds tropical cyclones and their associated storm surges can be a great danger when they find their way to a populated land.
The destructive force of a tropical cyclone comes from the wind speed of the rotating air and the rainfall that can cause disastrous flooding 2 . The highest wind speeds are usually found near the typhoon center. Strong rain, on the other hand, can be experienced near dense cloud structures. Therefore, the track of a cyclone is characterized by both the coordinate of the typhoon center and the shape or the distribution of clouds. Sun et al. 3 found that tropical cyclones are becoming stronger, larger, and more destructive in the context of global warming. Kossin 4 mentions in his study that translation speeds of tropical-cyclones have decreased globally by 10% and in Asia by 30% over the period 1949-2016. The lower the velocity of the cyclone as a whole system, the more rain falls on one location and the higher the risk for flooding. Thus, slower translation speeds can cause even more destructive floods.
The present work concentrates on tracks of cyclones that form in the north-western Pacific Ocean, also known as typhoons. The geographical focus lies on the Korean peninsula. In the past, Korea has suffered from numerous typhoons of different sizes. Typhoon Sarah was the strongest one. The cyclone hit the island Jeju on September 16th in 1959 during the traditional Thanksgiving festival leaving 655 dead, 259 missing, and more than 750,000 homeless 5 . In the younger history of the country, especially typhoons Rusa (August 2002) and Maemi (September 2003) have caused fear and death among the Korean population. Together they have been responsible for 376 casualties and a damage of 11.5 billion USD 6,7 . Xu et al. 8 investigated paths of super typhoons approaching China over the past 50 years. They identified a trend of strong typhoons getting attracted by the moist environment in southeastern China. This trend also affects damages on the Korean peninsula, since the size of a super typhoon can be similar to the spatial distance between the Chinese east coast and the Korean west coast. To save lives and reduce such damages in the future, accurate forecast methods need to be established.
Accuracy is only one criterion when talking about the prediction of natural disasters. Speed and flexibility play a significant role as well. Forecasts should be done quickly and forecast tools should be able to react immediately on sudden changes. Finally they should be inexpensive. Current predictions in South Korea are done by conducting numerical simulations on a Cray XC40 supercomputer with 139,329 CPUs. This method consumes a www.nature.com/scientificreports www.nature.com/scientificreports/ chronologically ordered input images, the ground truth image, and the generated image. A sequence is named with the date and time (universal time coordinated -UTC) of the corresponding ground truth image, for example: "1993072718" stands for 27th of July in 1993 at 6 pm (UTC). prediction of the typhoon center. The accuracy in the prediction of the typhoon center coordinate is measured by comparing the labeled red square of the ground truth image with the predicted red square of the generated image, rather than coordinates in the Cartesian space. This is because convolutional neural networks have difficulty in learning a mapping between the coordinate in the Cartesian space and the coordinate in the one-hot pixel image space 18 . The red square in the predicted image is detected by applying a color filter on the generated image and a convolution with the size of the red square on the filtered image. The pixel with the highest value in the convolved image gives the location of the predicted typhoon center. In most of the cases images with a clearly identifiable red square are generated, as shown in Fig. 1(a). In some cases, however, the GAN provides several alternatives, as visualized in Fig. 1(b). Each possible predicted typhoon center has its own color strength. The color intensities can be interpreted as probabilities; a strong red square shows a confident prediction, a weak mark stands for a possible predicted typhoon center with a low probability. In cases like in Fig. 1(b), the location with the strongest color intensity is chosen. After identifying the (x, y)-pixel coordinates of the predicted typhoon center, they are transformed back to latitudinal (φ) and longitudinal (λ) coordinates via georeferencing.
Results for ten test typhoons are presented in Figs 2-5. Images in the left column of the figures are predicted by using only satellite images as input data, and are referred to Case A, where locations of the typhoon center are marked with yellow squares. Images in the right column of the figures are predicted by using satellite images along with velocity fields at 10 m height as input data, and are referred to Case B, where locations of the typhoon center are marked with blue squares. Results in both cases are contrasted to locations of the real typhoon center, represented by red squares. Tables in the figures contain absolute errors, relative errors, and errors of forecasts that have been conducted by the Joint Typhoon Warning Center (JTWC) for typhoons Faye, Violet, Oliwa, Saomai, Rammasun, and Maemi [19][20][21][22][23][24] , and by the Regional Specialized Meteorological Center (RSMC) Tokyo -Typhoon Center for typhoons Usagi, Muifa, Neoguri, and Malakas [25][26][27][28] .
Focusing firstly on Case A, it is noticeable how errors increase, when typhoons experience sudden northward or westward course changes. It is observed for all ten test typhoons. Figure 2 shows increased errors when typhoon Faye turns northward at sequences 10, 11, and 13, and similarly for Violet at sequences 8-11 and Oliwa at sequences 13-15. In Fig. 3, similar observations are made. When typhoon Saomai changes its path westward at the sequence 15 and northward around the sequence 21 errors increase noticeably. Typhoons Rammasun and Maemi also show difficulty in the prediction when they get deflected northward at sequences 10 and 8-10, respectively. Figures 4 and 5 show results for typhoons Usagi, Muifa, Neoguri, and Malakas. The dragon shaped typhoon Muifa underwent sudden course changes at sequences 16-18 or 24-26. The youngest typhoon among the ten test cases, Malakas, shows the highest error at the sequence 13, when its course has a sudden change to the north.
The path of a typhoon is highly influenced by steering flow that is controlled by storms' ambient environment. Sometimes, when a storm is surrounded by multiple systems whose circulations compete to each other, the steering flow is difficult to predict. As a results, the storms' track is hard to predict. Wu et al. 29 mention that typhoons in the Northwestern Pacific usually experience sudden northward or westward course changes. In the north-turning case, winds are enhanced on the southeast side of tropical cyclones, in west-turning cases, north-easterly winds are enhanced on the west side. One example is illustrated in Fig. 6, which shows satellite images and velocity fields for typhoon Neoguri at sequences 5 ( Fig. 6(a)) and 11 ( Fig. 6(b)). At the sequence 11, when the typhoon experiences a northward deflection, strong winds are noticed at the southeast side of the center, compared to the velocity field at the sequence 5. It may be difficult for the GAN to learn such phenomena only from satellite images.
Results in Case B indicate that combining satellite images with images of surface velocity fields has a positive impact on typhoon predictions. Although surface winds cannot represent the complete steering flow, they contain enough information to reduce the average error for all test typhoons, except for typhoon Malakas. Especially, the previously mentioned northward or westward course changes are predicted much more accurately in Case B. Except for results for typhoons Faye and Rammasun, standard deviations of the error are also reduced for all test typhoons. Therefore, adding information of the velocity field to the network seems to be beneficial to improving the predictive capability of the present GAN-based method.
Several findings are gained by analyzing errors in predictions conducted by the JTWC and the RSMC. Firstly, it is observed that for typhoons Faye, Oliwa, Saomai, Rammasun, Maemi, and Muifa, errors in Case B are lower www.nature.com/scientificreports www.nature.com/scientificreports/ than errors of 12-hour predictions conducted by the JTWC and 24-hour forecasts of the RSMC. Secondly, six of the ten test typhoons show lower standard deviations for results in Case B compared to predictions reported by the JTWC and the RSMC. This may be partly due to the shorter prediction interval (6-hours) in the present study than those of the JTWC and the RSMC. Thirdly, whereas sudden northward or westward deflections are the challenging regime in Case A, forecasts conducted by the JTWC and RSMC show no increased errors when sudden course changes occur. However, their challenging regime seems to be interactions between storms and land, since for all ten typhoons errors increase before landfall. Difficulties in predictions before landfall are noticeable in Case A for typhoons Faye, Saomai, Rammasun, Maemi, Muifa, and especially Neoguri. In Case B, these difficulties are handled much better by the GAN.      www.nature.com/scientificreports www.nature.com/scientificreports/ prediction of the cloud shape. As an example for the generation of images that show the cloud shape, Fig. 7 illustrates the prediction for the sequence 6 of typhoon Maemi. Generated images do not have the same sharpness than the ground truth image, instead they are blurry. Blurriness is a well known challenge in video prediction tasks, which is tried to be overcome by using a gradient difference loss function in the present study. However, although generated images suffer from a certain degree of blurriness, the main structure of clouds is still visible. Furthermore, in the generated image in Case B the spinning motion of the typhoon seems to be reproduced more realistically than in the generated image in Case A. This is shown clearer in Fig. 8, where generated cloud images in Cases A and B of four randomly chosen sequences are presented. Generated images in Case A seem to be static and do not represent any dynamics. In the sequence 11 of typhoon Violet, for example, the image in Case A cannot reproduce the spinning motion of the typhoon (see Fig. 8(a)). The image in Case B, on the contrary, does not only illustrate the typhoon more www.nature.com/scientificreports www.nature.com/scientificreports/ realistically, but also cloud patterns in the remaining parts of the image. In the sequence 6 of typhoon Rammasun, the prediction in Case B again resolves the spinning motion of the typhoon much better than the prediction in Case A (see Fig. 8(b)). Details, like the cloud structure east or north of the typhoon center, are generated much more reliably. In the sequence 15 of typhoon Maemi the cyclone has almost reached the Korean peninsula. Parts of the cloud structure start to dissipate, but a significant part follows a strong westerly wind at a high altitude, called jet stream. In Fig. 8(c) it can be noticed how the result in Case B reproduces the suction effect of the jet stream in a smoother way than the predicted image in Case A. Finally, in Fig. 8(d) the image in Case B highlights much better how surrounding clouds move into the eyewall than the image in Case A.

Discussion
The application of a deep learning method for typhoon track prediction in forms of typhoon center coordinates and cloud structures has been explored. Learning only from satellite images, 42.4% of all typhoon center predictions have absolute errors of less than 80 km, 32.1% lie within a range of 80-120 km and the remaining 25.5% have an accuracy above 120 km. The averaged error lies at 95.6 km. In general, errors increase when typhoons undergo sudden northward or westward course changes. Predictions in this challenging regime get much more accurate, when satellite images are combined with data of the velocity field at 10 m height. In that case, 64.8% of all predictions are below 80 km, 27.7% lie within a range of 80-120 km and only 7.5% remain with an accuracy above 120 km. The averaged error could be reduced by 27.7% to 69.1 km. Furthermore, it has been shown that the GAN is able to generate images that reproduce cloud appearance.
From a user's perspective, it is helpful to make predictions in a quicker and cheaper manner. Current predictions in many countries rely on highly expensive numerical simulations using the state-of-the-art supercomputers, of which acquisition is limited by few advanced countries. Furthermore, it is expected that the present deep-learning-based method can be utilized in combination with other conventional techniques, especially, for www.nature.com/scientificreports www.nature.com/scientificreports/ non-user-biased analysis and decision making processes. Although, in the present study, predictions are conducted for a 6-hour interval, extension of the prediction time-interval is straightforward once the acquisition of more satellite images is possible. To improve the prediction accuracy, the next step will be adding physical information to the input data, like the sea surface temperature, the surface pressure, and velocity fields at various heights. Just learning from satellite images that show the cloud structure and the typhoon center is a good starting point but not sufficient for learning whole complex phenomena that are responsible for the creation and motion of typhoons. Methods satellite images. Input satellite images, which were captured by satellites at the altitude of 35,786 km, have been provided by the Korean Meteorological Administration (KMA) 30 . They contain 76 typhoons from 1993 till 2017 that hit or were about to hit the Korean peninsula. Whereas Hong et al. 11 used satellite images directly, in this study pre-processing of the images is inevitable. During the 25 years of capture time, different satellites have been operating. Thus, raw images have different perspectives on the Korean peninsula. Three different types of image perspectives and pixel sizes are provided (see Fig. 9). The network takes images with the same pixel size and learns better if all images have the same or a similar perspective on the north-western Pacific Ocean.   Table 1. Frequency distribution for the prediction of all sequences for Case A (Only satellite images) and Case B (Satellite images combined with data of the velocity field at 10 m height).
www.nature.com/scientificreports www.nature.com/scientificreports/ visible and infrared spectrum mappings of the earth and its cloud cover. The upper black bar, containing the satellite name, the date and the time written in yellow, as well as the black frame are unnecessary information for the learning phase. Furthermore, the pixel size causes memory issues in the test phase. Thus, images have been cropped and resized to 250 × 238 pixels (see Fig. 9(d)).
The perspective of images captured between 2011 and 2014 is presented in Fig. 9(b). The image with the pixel size of 512 × 412 shows typhoon Neoguri that formed in July 2014. It has been captured by Korea's first multi-purpose geostationary meteorological satellite, namely Communication, Ocean and Meteorological Satellite (COMS). As in the previous case, the upper black bar is cropped and the images are resized to 250 × 238 pixels (see Fig. 9(e)). The view on the earth in Fig. 9(e) is similar to the view in Fig. 9(d).
An example for raw images of the remaining data is shown in Fig. 9(c). The operating satellite is the same like in the previous case. The 512 × 433 pixel shot shows typhoon Goni from August 2015. Except for a different thickness of the upper black bar, the pre-processing steps match with the previous case and lead to images like in Fig. 9(f).
All images have the 'png' format with three color channels (R(red)G(green)B(blue)). To improve the visibility of the country boarders, their color has been changed from yellow to blue. In total 1,628 images are stored, with a time step size of 6 hours between images. There are two accuracy criteria for the typhoon track prediction, a quantitative and a qualitative criteria. In the quantitative criterion, the difference between the coordinate of the predicted typhoon center and that of the ground truth is taken into consideration. In the qualitative criterion, the predicted shape of clouds and the shape of the ground truth cloud are compared.
Every satellite image has been labeled with a red square at the typhoon center. The latitudinal (φ) and the longitudinal (λ) coordinates of the typhoon centers are provided by the Japan Meteorological Agency (JMA) 32 . In order to label each satellite image with its red square, φ and λ coordinates have to be transfered to (x, y)-pixel coordinates in the image. This process is known as georeferencing and is illustrated in Fig. 10. Images are split into training and test data. The training data contain 1,389 images of 66 typhoons, the test data are 239 images of 10 typhoons.
Velocity fields. In order to help the GAN to better learn the movement of typhoons, the public dataset ERA-interim 33 is employed. ERA-interim is a global atmospheric reanalysis starting from 1979 that has continuously been updated until today. It uses a fixed version of a numerical weather prediction system (IFS -Cy31r2) to produce reanalyzed data. The available data have time steps of 6 hours and their dates fit to the satellite images described in the previous section. Although ERA-interim data cover the whole globe, for this work only the area around the Korean peninsula is selected. Raw data with grid resolution of 0.75 degrees are refined to 0.125 degrees by applying linear interpolation, which leads to a resolution of nearly 13.8 km 34 . Images from reanalysis data are generated with the software Panoply and the map type Lambert Conformal Conic to match the view of the satellite images 35 . In the present study, reanalysis data are used by the GAN to learn information about the surface velocity field at 10 m height. Figure 11 gives an example for typhoon Maemi. The velocity field in Fig. 11(b) is www.nature.com/scientificreports www.nature.com/scientificreports/ contrasted with the satellite image in Fig. 11(a). Color changes in the zonal and meridional velocity components located in the white circles indicate a spinning motion of the typhoon.
Deep learning methodology. The deep learning methodology is different for the training and the testing steps. Figure 12(a) shows the concept of adversarial training. For simplicity, a case is shown where only satellite images function as input. As mentioned previously in this work, each full scale satellite image consists of three color channels (RGB). They are represented throughout Fig. 12 by three overlapped layers. Before using the training data as input images to the GAN they get cropped to a total number of 5,000,000 clips with a pixel size of 32 × 32. One clip contains a set of m consecutively cropped satellite images from the past (input, I) and one ground truth image.   www.nature.com/scientificreports www.nature.com/scientificreports/ (D), where 0 stands for a generated image and 1 means the ground truth image. Testing steps are done with full scale test data, containing images with 250 × 238 pixels. They are not cropped. As visualized in Fig. 12(b), input images are taken and normalized by the generator network which generates a full scale image at a future occasion and denormalizes it.
The pixel size of 32 × 32 is selected for efficient usage of GPU memory. Even the network is trained with small size clips (32 × 32), it can eventually learn the same weights of the network that are learned with larger size clips.  www.nature.com/scientificreports www.nature.com/scientificreports/ This is because kernel sizes for convolutions are under 7 × 7, so the network learns spatial characteristics under the space size of 7 × 7 pixels. However, during tests, typhoon satellite images on larger size (250 × 238) patches are predicted. This is possible because of the fully convolutional architecture of the generator.
The configuration of the deep learning network is summarized in Table 2 (see 16 for detailed algorithms). An open source code 36 is employed with modifications in input channels so that it is capable of changing the number of prior sequences and of accounting for additional variables such as velocity fields. The number of channels (ch) depends on the type of input data. If the GAN takes satellite images only as input data, three channels are needed. If images of zonal and meridional velocity fields are added to the input set, ch increases to 5. The network is trained from scratch.
The generator model is trained to minimize a combination of loss functions as follows: where λ l2 = 1, λ gdl = 1, and λ adv = 0.05. Let G k (I) be the predicted image from a convolutional neural network of G k and I ( ) k  be a 1 2 k resized image from the provided ground truth image. The L k 2 loss function evaluates the explicit difference between the predicted and provided images as Generative techniques for video modeling are known to suffer from the blurriness 16,37,38 . To reduce the blurriness, Mathieu et al. 16 proposed a gradient difference loss function (GDL) and reported that a GAN with the GDL improves the blurriness compared to other methods including a recurrent neural network based method proposed by Ranzato et al. 38    www.nature.com/scientificreports www.nature.com/scientificreports/ This supports the network not to be deluded by the generator model by extracting important features of typhoons in an unsupervised manner.
For each testing, the generator model is fed with m consecutive full scale satellite images from the past and generates a full scale typhoon satellite image at future occasion (see Fig. 12(b)). errors in the prediction of typhoon centers. In the current study two different errors are investigated.
The first one describes the distance between the predicted coordinate (φ pred , λ pred ) and the real coordinate (φ real , λ real ), named as an absolute error (E). E is calculated in kilometers (km) by applying the haversine formula 39 , with the earth radius R taken at the location of the real coordinate: However, the absolute error does not give sufficient information about the prediction quality. Tracks of slowly moving typhoons, for example, are more difficult to predict than tracks of fast ones. It is therefore necessary to introduce a relative error (E rel ), that is calculated as follows: