Markerless Measurement and Evaluation of General Movements in Infants

General movements (GMs), a type of spontaneous movement, have been used for the early diagnosis of infant disorders. In clinical practice, GMs are visually assessed by qualified licensees; however, this presents a difficulty in terms of quantitative evaluation. Various measurement systems for the quantitative evaluation of GMs track target markers attached to infants; however, these markers may disturb infants’ spontaneous movements. This paper proposes a markerless movement measurement and evaluation system for GMs in infants. The proposed system calculates 25 indices related to GMs, including the magnitude and rhythm of movements, by video analysis, that is, by calculating background subtractions and frame differences. Movement classification is performed based on the clinical definition of GMs by using an artificial neural network with a stochastic structure. This supports the assessment of GMs and early diagnoses of disabilities in infants. In a series of experiments, the proposed system is applied to movement evaluation and classification in full-term infants and low-birth-weight infants. The experimental results confirm that the average agreement between four GMs classified by the proposed system and those identified by a licensee reaches up to 83.1 ± 1.84%. In addition, the classification accuracy of normal and abnormal movements reaches 90.2 ± 0.94%.

objective evaluation of infant movements. To enable the objective evaluation and measurement of infant movements, the early detection of disorders in infants has been extensively studied. For example, Algra et al. measured myoelectric potentials by attaching electrodes to infants' limbs and evaluating the muscular activity and duration of movement during normal GMs 13,14 . Several studies have used position sensors and acceleration sensors to evaluate the spontaneous movement of limbs during GMs [15][16][17][18][19][20] . Specifically, Heinze et al. 18 used acceleration sensors attached to the four limbs to analyse the periodicity of velocity and acceleration in spontaneous movements, and they reported that the extracted features can be used to diagnose movement disorders. However, the above studies required sensors or markers to be attached to the infants, which can disturb the spontaneous emergence of their movements.
As an alternative, markerless measurement methods for infant movements have also been developed [21][22][23] . For example, our group proposed a video analysis method to analyse infant movements and enable objective evaluation 21 . Adde et al. extracted several features including the magnitude of whole-body movements from video images to classify two types of GMs 22 . This study successfully detected two types of GMs and enabled prognosis predictions of cerebral palsy.
As described above, previous studies have demonstrated the effectiveness of objective and automatic GM evaluation in clinical practice. However, the evaluable movements represent only a portion of all GMs. To classify various GMs, it is necessary to analyse the movements of different regions of the body from various aspects. The next section describes the proposed system, which enables such analysis.
proposed System Figure 1 shows the proposed markerless movement measurement and evaluation system for GMs. The proposed system consists of five parts: movement measurement, feature extraction, movement analysis, GM-based movement classification, and output/storage of results. This section describes the configuration of the proposed system. Movement measurement module. The movement of infants is measured using a video camera fixed directly above and parallel to the crib surface, which is covered by a unicolor fabric spread, as shown in the left-hand side of Fig. 1. The infant is assumed to be lying supine at the centre of the crib, and the height of the video camera is adjusted so that the whole of the infant's body can be captured within the video frame. The video images are recorded and stored on a personal computer with a frame rate of f s Hz. feature extraction. The feature extraction follows the method adopted by Shima et al. 21 . Figure 2 shows the feature extraction images. The coordination of the image is defined as follows: the origin is defined as the top-left www.nature.com/scientificreports www.nature.com/scientificreports/ pixel, W represents the maximum pixel number along the x axis, and H represents the maximum pixel number along the y axis. That is, the x axis is defined in the range x w ( =  w W 1, 2, , ) and the y axis, in the range y h , ). The proposed system converts the measured images to grayscale images, and it calculates the background subtraction to generate binary images using a brightness threshold T, where black (0) represents the background area and white (1) represents the infant's body (Fig. 2a). At the same time, interframe difference images are calculated between time-adjacent frames to generate additional binary images using a brightness threshold T, where white (1) represents a pixel in which infant movement has been detected (Fig. 2b). To extract the movement of all four limbs, the whole-body area is determined and divided into four areas using two line segments, as shown in Fig. 2c,d. This process uses the following algorithm. First, ellipse approximation based on the least-squares method is applied to the background difference image to generate an ellipse surrounding the outline of the body. This allows the area B representing the circumscribed rectangle of the ellipse to be determined (Fig. 2c). By letting α and β represent the pixel numbers of the long and short axes of the rectangle, respectively, the analysis area A is determined as shown in Fig. 2c by adding the margins α = a t a 1 1 pixels and α = a t a 2 2 pixels to the ends of the long axis and β = a t a 3 3 pixels to both ends of the short axis, where t a 1 , t a 2 , and t a 3 are adjustable variables. The analysis area A is then divided into four areas by two line segments as shown in Fig. 2d: the line segment UU′ divides A into the upper and lower body areas with ratios of γ γ − : (1 ), and the line segment VV′ divides A into the left and right body areas with ratios of δ δ − : (1 ). The divided areas are represented by the vari-  www.nature.com/scientificreports www.nature.com/scientificreports/ body; and A 4 , the right lower body. The combination of these four areas generates A k ( = ... k 5, , 9), where A 5 represents the upper body; A 6 , the lower body; A 7 , the left body; A 8 , the right body; and A 9 , the whole body.
To extract features from infant movements, the changes in body posture P A l ( ) k and those in body movement M A l ( ) k are extracted from each analysis area A k ( = ... k 1, , 9). The velocity of the body centre G v (G l x , v , G l y , v ) and the fluctuation of the body centre G d in a frame window (G l x , d , G l y , d ) are also calculated from the area A 9 . Here, l ( = ... l L 1, 2, , ) represents the frame number of the video image, and L is the total number of frames in each analysis interval.
A change in body posture P A l ( ) k is defined by a change in the whole body area, and a change in movement M A l ( ) k is defined as the magnitude of movement normalized by the whole body area using the following equations: where O x y ( , ) represents the binary number of the pixel at coordinates x y ( , ) w h of the background difference image within analysis area A k . Note that . P A ( ) ave 9 is the average of the maximum value of P . G y ave ) over L frames using the following equations: , ave ( ) ave 9 9 where the body centre coordinates G l,x , G l,y are calculated from the background difference image using the following equations: Movement analysis. By using the extracted features, movement analysis is performed based on clinical insights to calculate objective evaluation indices. Specifically, the proposed system generates J indices (Table 2) from four different aspects: (I) movement magnitude, including duration and magnitude of the movement; (II) movement balance, described by the ratio and correlation of movements between analysis areas; (III) movement rhythm, representing the periodicity of movement; and (IV) movement of the body centre. The details of calculation for each index are presented in the Appendix.

Motion classification based on GMs.
First, each index is reduced to its canonical form based on the average μ j (j = 1, 2, …, J) and the standard deviation σ j of the corresponding index collected from the standard subject group, whose motions were clinically preclassified as the normal GMs.
j j j j where z j represents each canonicalized index. As a result of canonicalization, the indices of the target motion can be evaluated in terms of the variation from the indices of the standard group, as the indices are expected to follow the normal distribution (0, 1) N . A feedforward-type neural network called the log-linearized Gaussian mixture network (LLGMN) 24 is used as the classifier for the movements. LLGMN includes Gaussian mixture models in a log-linearized form, enabling the estimation of the probabilistic distribution of a given sample dataset. First, the parameters of the LLGMN are adjusted using learning samples of the C classes of GMs , : c c is the number of learning samples for each GM). After this learning process, the posterior , ] C 1 2 T corresponding to each class can be obtained by inputting a new set of canonicalized indices z obtained from the motion of the infant. Finally, the posterior probabilities calculated for every analysis interval (L frames) are averaged over the entire analysis interval, and the movement type of the infant is classified as the GM class with the highest posterior probability. This method enables the automatic classification of motion types corresponding to the predetermined C types of GM. In addition, to avoid misclassification caused by ambiguous input, the entropy S is employed. This is defined as If S exceeds the threshold value S th , the input motion is classified as Type 0 (i.e., does not belong to any GM motion types). When the movement frequency of the upper or lower body is equal to 0, indicating no movement was detected, it is also classified as Type 0. Figure  The user interface (e) allows users to set the threshold value T for binarization and the head position of the infant. This permits the analysis and classification configuration to be adjusted according to the video image quality. The display system also allows users to visually capture movement features and reconfirm the movements by rewinding the video image and analysis results in the event of abnormalities.

Experimental Configuration
Movement analysis and GM classification experiments were conducted using the proposed system. The experiments were in full compliance with the Declaration of Helsinki, and they were performed under the approval of the Research Ethics Committee of the Prefectural University of Hiroshima and the ethics committee of the National Cerebral and Cardiovascular Center. Informed consent was obtained from the parents of the subjects, and the experimental purpose and methods were fully explained. Informed consent for publication of identifying information/images in an online open-access publication was also obtained from the parents of the subjects whose images appear in this manuscript.
Nineteen infants, including those with LBW, were studied. Subjects A-C were full-term infants, and subjects D-S were LBW infants. The video images were taken under doctor supervision to minimize any burden on the infants and avoid potential dangers. Video images of two additional subjects (T and U) were obtained from a DVD, "Spontaneous Motor Activity as a Diagnostic Tool, " by Prechtl 25 . Therefore, video images of 21 subjects were used in the experiments. The detailed information of the subjects is shown in Supplementary Table S1. The recording and analysis followed the clinical standards of Prechtl's GM assessment, and periods of crying and sleeping were excluded from the analysis. The average video length was 442.17 ± 284.15 s per subject. The measurement and analysis parameters were set as follows: f s = 30 Hz, = .
t 0 02 , W = 720 pixels, H = 480 pixels, E = 10, M th = 0.05, f max = 5 Hz, L = 900 frames, and L f = 300 frames. As the videos were taken under different situations, the ratio parameters for area division γ and δ were manually adjusted to ensure that the Total magnitude of G d www.nature.com/scientificreports www.nature.com/scientificreports/ motion of infants was within the analysis area, and the threshold parameter T for binarization was manually adjusted to enable appropriate background difference images to be generated.
A second-order Butterworth filter was applied to smooth the changes in motion and velocity of the body centre, thus eliminating noise from the video images. The cut-off frequency of the filter applied to the change in motion was = f 10 cut m Hz, and that to the change in body centre was = f 5 cut g Hz. The fast Fourier transform was applied under a window width of 128 data points and an overlap of 127 data points. The normalization window width for the cross-correlation function (see Appendix) was 300 data points, and the overlap was set to 299 data points. The analysis focused on the upper body and lower body (k = 5, 6) and generated J = 25 indices. The indices were calculated every S frames using data in each analysis interval of length L. For subject A-S, we set S = L, meaning the analysis interval was not overlapped. For subject T and U, we set S = 1, meaning the analysis interval was overlapped by L − 1 frames. This is because their videos were considerably shorter than those of the other subjects.
In the experiment, motion images were first classified into four groups by physical therapists (PTs) licensed for GM evaluation. The PTs assessed GMs from the video recordings. This classification task was performed for every 30 s (900 frames) of video, and the classification result of each interval was determined as the most frequently appearing type of GM. In accordance with the official GM assessment guideline, the classification was conducted by a single PT; videos that were difficult to judge were assessed by multiple PTs. Analysis intervals where GMs can be clearly seen were also selected by PTs, and the assessment of GMs was conducted on these intervals. The number of times that each type of GM was classified was as follows: WMs: 193; FMs: 279; CS: 31; and PR: 66. Other types of GMs (e.g. abnormal FMs) were not observed in the video used in the experiments.
The proposed system automatically classified the input motion images of infants during GMs into Types 1-4 (C = 4), corresponding to WM, FM, CS, and PR, respectively. The entropy threshold used to prevent misclassifications caused by ambiguous input was set to = S 1 th . Thirty-one samples of each GM type were selected at random for leave-one-out cross-validation, and the classified results were tested by calculating the average of the classification results given by the PTs. The classification accuracies were averaged over five sets of cross-validation. The indices were standardized using the average values and standard deviations of normal GMs (i.e., WMs and FMs) used for the parameter adjustment of LLGMN ( = N 31 c , total 62 samples). The partial Kullback-Leibler (KL) information measure 26 was used to reduce the number of indices and to evaluate the contribution of each index to the classification. Let K(Q, C) represent the KL information between the probabilistic distribution of input indices and that of classes, and [ ] be the KL information with the www.nature.com/scientificreports www.nature.com/scientificreports/ i-th input index reduced. The partial KL information is then defined by If the reduced i-th index does not contribute to the classification, the partial KL information becomes G [i] = 1. By using the partial KL information, the input indices can be sequentially reduced to extract the most effective ones without evaluating all combinations of indices. In this experiment, 16 samples were randomly extracted for each type (total of 64 samples), and the classification accuracies were calculated with the input indices reduced one-by-one based on the partial KL information. Figure 4a shows examples of the radar charts produced by analysing the video images corresponding to each GM. The axes of these charts correspond to the input indices. To confirm the differences in the indices from the viewpoint of normality/abnormality of GMs, averaged radar charts for the normal GM group (WMs and FMs) and the abnormal GM group (CS and PR) were calculated for all subjects (Fig. 4b). The figure includes the statistical test results obtained by comparing the average value of each index of the normal GMs and the abnormal GMs based on the unpaired t-test (significance level: 5%). Statistically significant differences were observed in one of the indices of movement balance, I 14 . In addition, Fig. 4c shows the changes in body movement of the upper and lower limbs for the CS type. Figure 5 shows an example of the temporal change in the posterior probabilities of subjects A and J. The arithmetic mean of the posterior probabilities over all time intervals (except those used to learn the LLGMN parameters and those containing movements judged to be irrelevant by the PTs) are shown on the left-hand side of the figure. The motion with the highest posterior probability was Type 1 (WMs) for subject A and Type 4 (PR) for subject J. Figure 6a presents the average confusion matrix of the classification results across five trials. The rows and columns correspond to the PT assessments and the classification results of the system, respectively. The average precision 27 of four classes over a total of five trials was 83.1 ± 1.84%. In addition, Fig. 6b shows the average successful classification accuracy of abnormal and normal motions. Here, Types 1 (WMs) and 2 (FMs) are defined as normal motion, and Types 3 (CS) and 4 (PR) are defined as abnormal motion based on the definition of GMs. Figure 6c shows the average classification accuracy as the input indices are reduced one-by-one using partial KL information 26 . The average classification accuracies were calculated from the 64 samples used to calculate the partial KL information, and each error bar corresponds to the standard deviation of the average classification accuracy of the respective GMs. Figure 4a shows that each index becomes approximately 0 for normal GMs (i.e., WMs and FMs). This is because the canonicalization of the indices was carried out using the normal GMs. In contrast, parts of the indices become large for the abnormal GMs (i.e., CS and PR), resulting in distorted radar charts. Specifically, the CS demonstrates that the ratio of the movement frequency between the upper and lower limbs I (5,6) 4 increases, indicating that the motion frequency of the lower body is much smaller than that of the upper body. As shown in Fig. 4c, the change in movement M A ( ) k reinforces this fact, where the movement of the lower body is much smaller than that of the upper body in most time intervals. This can be caused by body rigidity, which is one of the characteristics of CS, in the lower body.

Discussion
The differences between normal and abnormal GMs can also be confirmed from the averaged radar chart (Fig. 4b). For the indices relating to the movements of the body centre, the abnormal GMs showed a larger value than the normal GMs. This indicates that abnormal GMs usually involve quick and rough movements of the body centre (infants' centre of mass estimated from images). Meanwhile, for the rhythm of the body centre and the balance of upper-limb and lower-limb, abnormal GMs showed smaller values than normal GMs. In general, short-lasting tremulous movement is a common feature of GM abnormalities 28 and may lead to an increase in speed and fluctuation of the body centre and unbalanced non-rhythmical movement. Such movements are usually described as monotonic movements lacking fluency, and they are the main characteristics of abnormal GMs. These results confirm that the proposed system can effectively capture differences between normal and abnormal movements from the radar chart described by the proposed indices, and its findings are consistent with the definition of GMs. Shown in Fig. 5, the assessment results given by the PTs indicate that typical and successive WMs were observed in subject A, and PR was continually observed in subject J. As the posterior probabilities calculated by the proposed system agree with these assessment results, the system is clearly useful as a diagnosis support tool.
The data presented in Fig. 6a, which shows that the average precision of four classes is 83.1 ± 1.84%, suggests that the proposed system is effective in accurately classifying four GM types. Figure 6b demonstrates that if the system is only required to distinguish between normal and abnormal motion, its performance improves to a classification accuracy of 90.2 ± 0.94%. Figure 6c shows that the classification accuracy does not significantly decrease until the input indices have been reduced to six types. These six indices are the magnitudes of the fluctuation of the body centre www.nature.com/scientificreports www.nature.com/scientificreports/ (1) Subject J "The infant monotonously and repeatedly raised the right hand, and the left arm shows very little movement and is seemingly paralyzed". (2) Subject B "Fidgety movement can be observed from both shoulders and both wrists". (3) Subject C "A chain of movements, which is a feature of fidgety movement, was observed. For example, the movement started from the right arm, followed by the neck, left arm, left leg, body trunk, and, finally, right leg". Indices related to the upper body could be related to the first two comments, and those related to the body centre may be related to the third comment. Importantly, this analysis allowed us to identify effective indices to evaluate GMs.
In conclusion, for the early diagnosis of infant disorders, this paper has proposed a markerless movement measurement and evaluation system for GMs. The proposed system measures infant motion using a video camera, and it extracts motion indices based on clinical definitions through video image analysis. Based on the extracted indices, a probabilistic neural network is then used to classify motion types corresponding to GMs. Our The results of the GM evaluator are "WMs" for subject A and "PR" for subject J. Note that each example has a different time scale on the horizontal axis, as the video length differs depending on the subjects (see Supplementary Table S1). experimental results show that the proposed system can capture the characteristics of the infants' movements and accurately classify the motion types. Specifically, the classification accuracy of normal and abnormal motions reached 90.2 ± 0.94%, indicating that the proposed system may support early diagnoses.
We plan to test this system by performing long-term measurements in a neonatal intensive care unit (NICU) for infants with LBW who are considered to be at high risk of various disorders. To achieve this, it is necessary to automatically extract sections where the infant is performing GMs from the recorded video. Therefore, new algorithms for detecting sleep and crying intervals based on objective criteria should be introduced into the proposed system in the future. In addition, three-dimensional motion analysis may be required to increase the number of classifiable GMs and describe motions such as kicking that cannot be fully assessed by indices extracted from two-dimensional video images. To improve the classification accuracy, further analyses will include the definition of indices describing the motion characteristics in detail, such as those related to elegance of motion, stability of body centre, and temporal changes in GMs.
In this paper, the authors focused on automating the visual classification of GMs, whereas the proposed system can be applied not only to GM evaluation but also to a wide range of spontaneous movement analysis of infants. In the future, the authors would like to analyse infants with various risks (e.g. those who have experienced intraventricular haemorrhage), thereby allowing wider diagnosis support for PTs and doctors. This may require a detailed analysis between movement features evaluated via the proposed system and biological variables of infants. It is also important to evaluate the infant's movements during sleep and crying, which were excluded from the analysis in this paper. Such segments other than GMs are known to have functional value in the neural development of infants 29,30 . Moreover, recent evidence suggests the importance of head movements in the study of atypical development 31 . However, the aim of this paper is to classify infant movement types based on Prechtl's GM assessment, and the Prechtl method does not consider the head movements of infants. In the future, we plan to improve the system so that more detailed motion characteristics, including head movements, can be evaluated by introducing additional algorithms such as skeleton extraction or pose estimation. is defined as the q-th occurrence of movement, enabling the movement count to be calculated using the following equation: Movement balance. The movement balance is evaluated by using the following three indices.

Data availability
The datasets generated and/or analysed in the current study are available from the corresponding author upon reasonable request.