Unsupervised real-world knowledge extraction via disentangled variational autoencoders for photon diagnostics

We present real-world data processing on measured electron time-of-flight data via neural networks. Specifically, the use of disentangled variational autoencoders on data from a diagnostic instrument for online wavelength monitoring at the free electron laser FLASH in Hamburg. Without a-priori knowledge the network is able to find representations of single-shot FEL spectra, which have a low signal-to-noise ratio. This reveals, in a directly human-interpretable way, crucial information about the photon properties. The central photon energy and the intensity as well as very detector-specific features are identified. The network is also capable of data cleaning, i.e. denoising, as well as the removal of artefacts. In the reconstruction, this allows for identification of signatures with very low intensity which are hardly recognisable in the raw data. In this particular case, the network enhances the quality of the diagnostic analysis at FLASH. However, this unsupervised method also has the potential to improve the analysis of other similar types of spectroscopy data.

: Dependency map showing 3x3 components of z for different values of beta. While complicated dependencies arise for a vanishing β, higher values create gaussian-like 2Ddistributions. The disentanglement loss is evaluated in the shown dependencies (for all 12 components of z).
of Figure S2: Encoding of the TOF-position vs. the latent space components for multiple values of β.
In addition to evaluating based purely on the loss, the compressed state is also evaluated via a comparison with the handcrafted labels, described in the main text and shown in Fig. 2. For the TOF-position, the influence of β is shown in Fig. S2. Ultimately, a value of 0.034 was chosen for β (and is used for all the plots in the main article). This value creates a situation where the 'position' is encoded in just two components of z and other key features of interest are encoded in such a way that they are easily accessible (see Fig. 2  The component z4 encodes both I3 (intensity) and B2 (baseline). In order to remove the baseline disturbance via latent space manipulation and still maintain a high reconstruction quality, one has to follow the procedure shown in Fig. S4. If the disturbance is on, I3 is encoded in the interval [0,0.25] as a linear dependency. The disturbance can be removed, without changing the peak intensity, by adjusting the z4 value to match the corresponding value on the linear intensity dependence in the interval [0.4,1]. This is visualised by the vertical arrow in Fig. S4. Figure S4: Procedure for baseline disturbance removal.

Phase correction
The information regarding the photoline position in the time-of-flight spectra is encoded in z0 and z1, which show a circle-like dependency as plotted in Fig. S5a, while φ along that circle is the position of the 2p photoelectrons. Fig. S5b shows the use of only high quality data, in which all 4 eTOFs contain the same information for the wavelength. Here, φ is compared with the leastsquare-fit position. φ is corrected by an additional neural network to compensate for the wiggles presented in Fig. S5b as described in the method section.

Magnetic bottle comparison
Although the data of the magnetic bottle also has low statistics (see Fig. S6), it is affected by SASE-fluctuations in the same way as OPIS. In the magnetic bottle experiment, the center of mass of the 2p photoline of sulfur from 2-Thiouracil is calculated in TOF-channels and compared with the predicted wavelength (WL) from the OPIS experiment. This was evaluated using multiple measures and shows a good agreement. The dependency curve (plotted in red in Fig. S7) is a) assumed to be linear in an easy and robust way or b) fitted with a polynomial approach. The agreement of the fit function vs. the predicted WL is evaluated in absolute and quadratic distance. In all defined errors (fit functions and distance definitions), the agreement of the predicted WL is higher than the agreement with the set wavelength λFEL.
of Figure   of Figure S7: Comparison of the predicted wavelength from the neural network with the center of mass of the magnetic bottle (blue dots). In this example the dependency is assumed to be linear (red line). of 5 5