Deep learning enhanced individual nuclear-spin detection

The detection of nuclear spins using individual electron spins has enabled diverse opportunities in quantum sensing and quantum information processing. Proof-of-principle experiments have demonstrated atomic-scale imaging of nuclear-spin samples and controlled multi-qubit registers. However, to image more complex samples and to realize larger-scale quantum processors, computerized methods that efficiently and automatically characterize spin systems are required. Here, we realize a deep learning model for automatic identification of nuclear spins using the electron spin of single nitrogen-vacancy (NV) centers in diamond as a sensor. Based on neural network algorithms, we develop noise recovery procedures and training sequences for highly non-linear spectra. We apply these methods to experimentally demonstrate the fast identification of 31 nuclear spins around a single NV center and accurately determine the hyperfine parameters. Our methods can be extended to larger spin systems and are applicable to a wide range of electron-nuclear interaction strengths. These results pave the way towards efficient imaging of complex spin samples and automatic characterization of large spin-qubit registers.

center magnetic dipole field (blue oval curves) and 13  nuclear spins (green circles) interacting with the NV center via hyperfine interaction.Bz is the external magnetic field, L  is the Larmor frequency,

Results
Theoretical model and overall procedure.
Figure 1a shows a schematic of the electron-nuclear spin complex considered in this work.
The NV center, an impurity in the diamond crystal lattice, acts as a sensitive probe for the surrounding nuclear-spin environment.The ground state electron spin of the NV center can be initialized and measured using spin-dependent fluorescence and can be manipulated by microwaves 37 .In typical dynamical decoupling spectroscopy, for example based on Carr-Purcell-Meiboom-Gill (CPMG) pulse sequence 22 shown in Fig 1b, the interaction of the electron with its nuclear spin environment leads to sudden and periodic losses of coherence at specific pulse timings.The magnitude and position of the dip in coherence depends on the longitudinal (transverse) hyperfine coupling parameter A (B).The CPMG signal is given by the probability Px that the NV center's spin state is preserved.In the absence of nuclear-nuclear interactions this can be described as 3 , (1 cos )(1 cos ) 1 sin 1 cos cos sin sin 2 , cos cos cos sin sin , where , ( ) ,  is half of the delay between  pulses, k indicates k th nuclear spin, n is the total number of nuclear spins, L  is the Larmor frequency, and N is the repetition number of the unit CPMG pulse (see Fig. 1b).The CPMG signal is given by the multiplication of all the Mk's for n nuclear spins as depicted in Eqn ( 1) and (2).
The main task of our deep learning model is to efficiently encode the features of each k th nuclear spin in Eqn.(1).Once successfully trained, the models can determine Ak and Bk of each nuclear spin from the experimental spectroscopy data (see the bottom panel of Fig. 1b for an example).Figure 1c shows the overall procedure to achieve this task.First, the measurements of CPMG signals and the implementations for generating datasets and training deep learning models are conducted simultaneously.Generating datasets for both HPC models and denoising models is performed using the theoretical model in Eqn.(1).Second, via the generated training datasets, denoising models are trained to reduce noise and HPC models are trained to identify whether specific hyperfine parameters exist in the data or not.Third, to enhance signal-to-noise ratio, the raw noisy CPMG signal is pre-processed by the trained denoising model and decoherence recovery process and is fed into the trained HPC models.
Fourth, using the outputs of the HPC models, an additional deep learning-based regression model is adapted to further restrict possible hyperfine parameter combinations.Lastly, in the auto fine-tuning phase, the prediction of the regression model is used as initial values of the hyperfine parameters, and automatic numerical fitting is performed.Example predictions of the HPC model depending on hyperfine coupling strength (first to third panels) and proximity to similar period (fourth and fifth panels).For all cases, the HPC model predicts correct spin signatures corresponding to input signals showing good consistency between the predicted vectors and the output vectors.(for example, in the first panel, the predicted vector is (0.01 0.99 0) and the output vector is (0 1 0)).The color scale bar in all 2D images ranges from 0 to 1.

Data representation for nuclear spin detection.
The qualitative features of a typical dynamical decoupling signal are as follows.First, the coherence dip of k th nuclear spin is periodic with approximate periodicity 3 (local period) 2 / ( ) ( ) and Ak and Bk are hyperfine parameters of the 'k th target' local period (see Supplementary Fig. 1 for the range of A and B values used in the proposed model).Second, the envelop of the coherence dip amplitudes as a function of  is periodic essentially showing periodic quantum entanglement evolution with the resonant nuclear spin 8 (global period).Third, each coherence dip can show additional fringes depending on the hyperfine interaction strength.In the strong coupling regime, for example when B > 100 kHz, the CPMG signal can exhibit multiple and large fringe oscillations 3,6 (Eqn.2,3).While conventional numerical peak detection or Fourier transform analysis is inefficient in the presence of these oscillating signals, below we show that the deep learning approach offers an excellent alternative route to solve the problem.

Deep learning model for classification.
Focusing on the local period of a specific target spin signal, we develop a set of deep learning models, coined HPC, each of which classifies the existence of a specific period of hyperfine-induced coherence dips in the data.Figure 2c  Panels in Fig. 2d show the classification results using our HPC model.The first panel is for the typical case that a single target period exists without strong disturbance from other spins nor spin bath signal, and the model successfully outputs a vector close to (0, 1, 0).The second panel shows the performance of the model for a strongly coupled single target spin where the parameters for spin number TPD21 in Supplementary Table 1, taken from existing density functional theory (DFT) calculations 37 , is used as an example.As mentioned above, although the spin signal is superposed with wide fringe patterns and oscillations, the model successfully identifies the signature of the target period with the output vector reaching (0.002, 0.99, 0).The third panel shows that the model also successfully classifies the target period even in the presence of other superposed strongly coupled spin signals.Furthermore, the fourth and the fifth panels give an example of the performance for input datasets with a single spin, (A2, B2)/ 2 =(7.8, 20) (kHz) (fourth panel) and with two spins of similar local period, (A1, B1)/ 2 =(7.9, 10), (A2, B2)/ 2 =(7.8, 20) (kHz), (fifth panel).The model successfully distinguishes each case, showing high selectivity of the nuclear spins.Therefore, these results show that our deep learning model provides a promising approach to detect individual nuclear spins with high precision, high selectivity, and for a wide range of hyperfine strengths.

Noise removal and decoherence effect recovery.
Before evaluating the experimental CPMG signal by trained HPC models, we first pre-process the raw experimental data by a denoising model.Figure 3a shows the overall procedure.For the noise removal process, Gaussian noise with the standard deviation 0.05

 
reflecting the experimental noise is added to the training datasets (see Supplementary Fig. 3).The decoherence effect is modelled by the approximate equation 3 , , where T accounts for dephasing of the electron spin, n is an exponential power obtained by fitting the experimental data and τ is half of the inter-pulse delay.We use an autoencoder structure 55,56 , which is an established structure to learn the representations of input data, to encode the features of the noisy input data and generate the denoised data.A one-dimensional convolution (1D CNN) layer 57 , which is widely used to capture the features of one-dimensional data such as time-series signal, is employed for building the denoising neural network.
As shown in Fig. 3b, the signal recovery model effectively removes the noises while retaining nuclear spin signatures of the experimental data.This is highlighted with the capability of recovering After denoising the raw experimental data, the decoherence effect is recovered by applying Eqn.(5) to the denoised data.We find that the confidence scores from HPC evaluating denoised experimental data are, in general, a few percent higher than evaluating raw data (compare (0.02, 0.91, 0) vs. (0, 0.99, 0) in Fig. 3c) and in some cases false predictions of raw data are corrected in denoised data (compare (0.17 0.71 0.1) vs (0.65 0.36 0.01) in Fig. 3c), successfully showing the efficiency of our pre-processing model.2. c.Confirmation of detected hyperfine parameters for spins with large number of interfering signatures (1 st panel), similar target periods (2 nd and 3 rd panels), weak local period signature (4 th panel), small transverse hyperfine coupling (5 th panel), and small longitudinal hyperfine coupling (6 th panel).We compare the obtained values to the results reported in Ref. [19] (bottom row, see main text).The panels also show examples of spins with small A that were not detected in Ref. [19] (3rd, 5th, and 6th panels).The uncertainty in the last digit is given in parentheses.The color scale bar in all 2D images ranges from 0 to 1.We demonstrate the performance of the developed procedures with two experimental datasets with N = 32 and N = 256.This new set of data was collected following the methods described in the Ref. 22 and in Supplementary Fig. 4. Figure 4b shows the comparisons of the experimental data to the reproduced CPMG signal using predicted hyperfine parameters by our deep learning protocol.Panels in Fig. 4c show example cases of predicted spins along with corresponding raw experimental data.The first panel highlights the case where the model can capture the nuclear spin signal and determine (A, B)/ 2 = (-213.19(5), 4.2( 9)) (kHz) even with overlapping signals stemming from other spins.The second and third panel show that the model can accurately distinguish spins with similar periods and automatic fine tuning successfully identifies individual (A, B) pairs matching the experiments.

Regression
Our analysis returns a total of 48 nuclear spins that together accurately describe the data.
However, several of these spins yield near-identical hyperfine parameters.It cannot be excluded that those signals originate from a single spin with a broadened signal due to dephasing and nuclear-nuclear spin interactions, which are not included in the model used here (see Supplementary Note 1 and Supplementary Fig. 5-7 for details).We anticipate that improved selectivity in this regime is possible by using other pulse sequences, for example non-equally spaced dynamical decoupling sequence 36,58 or by taking nuclear-nuclear interactions into account.Here, we chose to count groups of spins with nearly identical parameters as a single spin.In that way, we identify 31 nuclear spins.We summarize the full list of detected nuclear spins and the confidence levels in Supplementary Table 2.

Discussion
We compare our result with those obtained by other methods on the same sample.A manual analysis on a similar data set, taken with the same measurement procedure, identified 7 spins with parameters that match closely to 7 of the 31 spins identified here 22 .The large improvement in number of identified spins from equivalent experimental data highlights the advantage of our deep learning approach.Additionally, we compare the results to a recent multi-dimensional spectroscopy characterization 19 , a more demanding experimental technique that accesses nuclear-nuclear interactions.
For 23 of the 31 spins a good match is observed (Supplementary Table 2).The other 8 spins were not previously identified and are in a spectral range that was not accessed in previous experiments.We corroborate the identification of these spins through additional experiments with a different number of decoupling pulses N = 96 and N = 128 (see Supplementary Fig. 8).These results show the capability of our deep learning protocol to automatically and accurately identify nuclear spins in complex spin systems and characterize the coupling parameters from dynamical decoupling spectroscopy.
We estimate a total computational time of ~3 hours from generating training datasets and training the HPC models to complete the analysis on one set of experimental CPMG data (see details in Supplementary Fig. 2).Once trained, each HPC model can identify the most probable local periods of nuclear spins from the experimental CPMG data almost instantaneously (< 1 second) and obtain the final fitted hyperfine parameters within ~50 seconds per spin (detailed specifications of the computational power used is given in Supplementary Note 1).This fast data analysis highlights the potential of deep learning approaches to efficiently scale up the sensing and characterization of large spin systems.
We find that examining dynamical decoupling spectroscopy signals for various numbers of pulses N is important for the following reasons.First, large N makes spins with small B values visible and this is in general reflected in an increased number of detected spins as shown, for example, in the fifth panel of Fig. 4c.Second, we find that some spins near the Larmor frequency with relatively high B values (> 10kHz) are detectable only in N = 32 since for larger N too many spin signals are overlapped as illustrated in the sixth panel of Fig. 4c.The current protocol does not take nuclear -nuclear spin interactions into account.Therefore, our model fails to detect some of the interacting nuclear spins for N = 256, as for large N and long total evolution times, nuclear-nuclear spin interactions are nonnegligible and lead to a deviating period in the signal (see Supplementary Fig. 9 and note that part of the nuclear-nuclear interactions are known from ref. 19).For N = 32, the data approximately follows a simple electron-nuclear interaction model and nuclear-nuclear interactions can be neglected.In that case, our protocol successfully detects these spins as shown in the fourth panel of Fig. 4c.We envision future improvement of the deep learning protocol by taking into account the nuclear -nuclear interactions and building a single unified model which covers all ranges of hyperfine parameters and various N pulse sequences.At this current stage, discrepancies between experimental data for different N, for example between N = 32 and N = 256, can be used as a signature of nuclear -nuclear interaction.
In conclusion, we have proposed and demonstrated a deep learning approach to automatically detect and characterize individual nuclear spins based on dynamical decoupling spectroscopy with a single electron spin sensor.We have tested the method on a single NV center in diamond and have identified 31 individual 13 C nuclear spins with a wide range of hyperfine parameters.Our method is able to distinguish spins with strong couplings to the NV center which are difficult to handle for conventional peak detection algorithms 36,59 .Our methodology retains the general benefits of deep learning models; it is easy to modify the training sequence or neural network structures for other types of experimental data such as spectroscopy data of other defect centers, including in diamond.
Additionally, these results highlight the capacity of deep learning algorithms to efficiently analyze the complex nonlinear signatures in nano-scale and single-spin magnetic resonance and its robustness against realistic distortions, such as experimental decoherence and noise.Therefore, our results address one of the main challenges for quantum sensing experiments on more complex spin structures and for larger quantum registers and quantum networks based on spin qubits.

Sample and setup
Our experiments are performed on a single, naturally occurring, NV centre in a high-purity chemical-vapor-deposition homoexpitaxially grown diamond (type IIa) with a natural abundance of 13 C (1.1 %) and a 111 crystal orientation.To improve the photon-collection efficiency, we fabricate a solid immersion lens on top of the NV center and we use an aluminum-oxide anti-reflection coating layer (grown by atomic-layer-deposition) 60 .We use on-chip lithographically-defined striplines to apply microwave fields for fast driving of the electron spin transitions (Rabi frequency ≈ 14 MHz).
We apply a static magnetic field, Bz ≈ 403 G, along the NV-axis using a permanent roomtemperature neodymium magnet.We stabilize the magnet field strength to < 3 mG [19] and the magnet is aligned to the NV-axis with uncertainty of 0.07° using thermal echo sequences (see ref. 19 for details of the alignment procedure).
Our experiments are performed at a temperature of 3.7 K in a commercial closed cycle cryostat (Montana Cryostation).This enables us to readout the NV electron spin state in a single shot with high fidelity (94.5%), through spin-selective resonant excitation 60 (see detailed pulse sequence in Supplementary Fig. 4).The electron spin relaxation time is T1 > 1 hour 22 , the natural dephasing time is * 2 4.9(2) μs T  , the spin-echo coherence time is T2 = 1.182(5) ms, and the multipulse dynamical decoupling coherence time is T2 DD > 1 s, for an optimized inter-pulse delay 2 22 .

Training datasets and hyperparameters of the HPC and regression model
For preparing training datasets of HPC and regression models, the spins of target period are chosen from (A, B) candidates separately grouped by target periods (see Supplementary Fig. 1 and Supplementary Note 1).The rest of the spin candidates are randomly selected from a previously published list calculated by DFT 60 and from a range -50 kHz < A < 50 kHz and B < 80 kHz for N32, and 2 kHz < B < 15 kHz for N256 signal (see Supplementary Fig. 2 for more details).The reason to choose this range is that the spins with larger values are already calculated in the DFT list.A range of target periods covered by one model is 250 Hz in N32 and 150 Hz in N256 with an evaluation step size of 50 Hz.(see more detailed parameter usage and performance results in Supplementary Note 1) Although a two-dimensional convolution layer is generally employed for image recognition, to boost computational speed while retaining the accuracy and validation loss, we use Dense layers for HPC and regression models.The LeakyRelu activation shows slightly better performance for the convergence to the lower validation loss than using ReLU activation.Batch Normalization layer 54 with epsilon=1e-05, momentum=0.1 (default values in Pytorch 1.3.1)shows faster convergence to the minimum loss than Dropout regularization.For the last layer, Sigmoid layer generally converges to the higher accuracy than the Softmax layer for our datasets.
Based on the prediction results by the HPC models, the regression model is built and trained.
The input data are generated from randomly selected (A, B) values of predicted periods by HPC models and the output data, in this case, are those spin values themselves (Fig. 4a).Then, the regression model is trained to predict (A, B) values of the input data.The evaluated (A, B) lists for the experimental data are used as initial guess for auto fine-tuning model.

Training datasets and hyperparameters of the denoising model
Unlike the HPC and regression models, training datasets for denoising models are generated in one dimensional data.The spins are selected in the same way as generating datasets for HPC models.
We introduce the auto-encoder structure (Fig. 3a) which is an established structure to encode the distribution of the input data and generate the targeted data.For both encoder and decoder parts, 1D CNN layer and 1D transposed CNN layer are employed rather than RNN layers such as LSTM, GRU layers, which show lower validation errors and faster convergence to the minimum loss.All the kernel size for both CNN layers is 4. In the encoder part, Maxpooling1D layer with kernel size of 2 is used after every single 1D CNN layer.A Batch Normalization layer with default parameters is used between all CNN layers.In all models, AdaBound 62 is employed for the optimizer and the initial learning rate is 0.00015 decayed at each epoch with customized rate (0.5 ~ 0.25).For loss functions, BinaryCrossEntropy loss is used for HPC model and MeanSquareError loss is used for regression and denoising models.

Fine-tuning algorithm
For the auto fine-tuning, the objective functions are defined as, where g(t) is the generated CPMG signal with all predicted (A, B) values, p(t) is the experimental data, Ti is a set of data points near dips of i th spin in the generated data and Tbath is a set of data points where spin baths are formed.For number of predicted spins M, the optimization process runs over the 2M-dimensional space (2M variables; M for A and M for B independently).
Since the loss function is not convex, instead of adopting algorithms such as a gradient descent, particle swarm optimization (PSO) 63 is employed for the optimization process.The number of needed particles increases exponentially as search dimensions increase, yielding a large computational cost.To reduce the cost, the optimization process is conducted sequentially only on one pair of (A, B) at once and iterated for every single spin until the total loss is converged because, as in Eqn. ( 6), the loss function can be split into each term of i th spin candidate.(see pseudo code in Supplementary Note 2)

Usage of trained models and management of total computational time
The denoising model can be reused for other samples if the number of unit CPMG pulse sequence (N) and measurement time resolution are kept the same.The classifier model can be reused if the external magnetic field, N, measurement time resolution, and total measurement time length remain the same.
All HPC and denoising models can be trained separately and generating datasets can also be processed independently.Therefore, for example to reduce total computational time to one-third, three computers can be used independently by dividing the training regions of all TPk into three regions.More detailed parameter usage and whole procedures for implementations of each model and processing experimental data are described in Supplementary Fig. 2 and Supplementary Note 1. Supplementary Table 2.The predicted hyperfine parameters (A, B) and corresponding confidence scores of the HPC models.The full list of detected nuclear spins and comparison with an independent previous experiment based on multidimensional spectroscopy (a different experimental method) [3] and a previous manual analysis of DD spectroscopy (the same experimental method) [6] on the same NV center.Hyperfine parameters (A, B) calculated by DFT are denoted as TPD#.Confidence scores using raw (denoised) CPMG data is given.The C# with an asterisk indicates the spins identified to be strongly coupled with nuclear-nuclear interactions in Ref. [3].C# † indicates a group of spins which appear as a single but broad dip in experimental CPMG data.The values (parentheses values) in the second and third columns corresponds to the confidence scores of the raw experimental data (the denoised data).Estimated errors of (A, B) values of the deep learning results are defined by standard deviation of the results with 50 iterations of fine-tuning method.Unlike in reference 3, spins with B  0 cannot be detected by the experimental sequence used in this work.
The complete list of results

Figure. 1
Figure. 1 General procedure for identifying hyperfine parameters of 13 C nuclear spins.a. Schematic diagram showing the configuration of an electron spin within the nitrogen-vacancy (NV) A (B) is the longitudinal (transverse) hyperfine interaction parameter.b.Typical dynamical decoupling pulse sequence (Carr-Purcell-Meiboom-Gill, CPMG) used for experimental nuclear spectroscopy.The bottom panel shows an example of experimental CPMG data from which electron-nuclear hyperfine interaction is analyzed.c.Pseudoalgorithm for training and hyperfine-parameters-prediction sequences including hyperfine parameter classifier (HPC), denoise and signal recovery, regression-based fitting, and fine-tuning models.The flow of experimental processes (computational processes) is on the red (gray) arrows.

Figure. 2 .
Figure. 2. Individual spin signature identification by hyperfine parameter classifier (HPC) deep learning model.a. Simulated CPMG signal with three spins of different (A, B) values showing general illustrates a structure of the HPC model and training datasets by exemplifying a case of classifying three different classes (see a detailed implementation of generating training datasets in Supplementary Note 1).The input training data is prepared along with three output classes as shown in Fig.2c.Class 1 corresponds to data that does not contain a spin with the target period, class 2 is for one spin with the target period existing in the data, and class 3 for two spins with slightly dissimilar target periods in the data.The output data is denoted in one-hot vector form; (1, 0, 0), (0, 1, 0), and (0, 0, 1) corresponding to no, single, and double target periods, respectively.The model is trained to estimate the confidence score of each element of the three-dimensional vector according to the input image.The model consists of stacked layers of Dense layer, Batch Normalization layer54 and LeakyRelu activation function as shown in Fig.2c.The detailed procedure of the neural network development is described in the methods section, Supplementary Fig.2, and Supplementary Note 1.

Figure 3 .
Figure 3. Denoising and decoherence effect recovery procedure.a. Architecture of the signal detailed oscillatory features of the data where the amplitudes of signals are almost equivalent to the fluctuations due to noise.Panels in Fig. 3c compares the visibility of the spin signal of the raw (left panel) and the processed (right panel) data showing effective removal of experimental noise and enhancement of signal-to-noise ratio, leading to higher performance of prediction by the HPC model.

Figure 4 .
Figure 4. Multiple nuclear spin detection from experimental data.a. Procedures for tuning -based fitting, auto fine-tuning model, and application to experimental data.We now discuss the final stage of the deep learning protocol and the application of the overall procedure to experimental dynamical-decoupling spectroscopy signals as shown in Fig.4a.After application of denoising and HPC models to predict possible local periods, we further apply a deep learning-based regression model to restrict the candidate hyperfine parameters for a subsequent finetuning process.Since the period information from HPC model only provides one functional relation between A and B given as Eqn (4), the purpose of the regression model is to find specific (A, B) values that best explain the shape of the coherence dips as a function of  .We set a search region for the value B ranging from 10 to 70 kHz and find the best fitted (A, B) pairs repetitively for all predicted periods.Since these values are obtained by fitting coherence dips stemming from individual nuclear spins, we use the deep learning-based fit results as an initial guess and iteratively tune (A, B) pairs again in the final step to automatically search a collective list of best fitted (A, B) pairs.We describe a pseudocode of the fine-tuning method in the Supplementary Note 2.