Longitudinal assessment of U-shaped and inverted U-shaped developmental changes in the spontaneous movements of infants via markerless video analysis

Various attempts have been made to elucidate the development patterns in the spontaneous movements of infants through longitudinal evaluations. Movement complexity has been found to demonstrate u-shaped changes in the measurements focusing on limb movements. However, researchers have not yet clarified how other characteristics, besides movement complexity, change over time. This paper presents a longitudinal evaluation of spontaneous movements in infants using evaluation indices calculated through markerless video analysis. Nine infants with corrected ages from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-1$$\end{document}-1 to 15 weeks participated in the experiments. We confirmed the change in indices over time using statistical methods. Index changes can be classified as positively correlated, u-shaped, inverted u-shaped, and uncorrelated. We also confirmed that the u-shaped and inverted u-shaped indices are negatively correlated. Furthermore, the principal component analysis revealed that the first principal component had the inverted u-shaped changes with the corrected age. These results suggest that it is important to synchronize the inverted u-shaped variations in the movement and velocity with the u-shaped changes in the movement complexity for infant development.


participnats. Nine infants (Participants A-I) participated in this study and markerless measurements were
taken of their spontaneous movements from −1 to 15 weeks of their corrected age. All participants were fullterm infants (born between a gestation period of 37 and 41 weeks). The birth weights of all participants ranged between 2500 and 4000 g, and there were no complications during the perinatal period. The experiments were conducted in accordance with the Declaration of Helsinki and were approved by the Ethics Committee of Hiroshima University (Registration Number: E-1150-1). Informed consent was obtained from the parents of each infant, and the experimental purpose and methods were fully explained to them. The experiments were also conducted in accordance with the relevant guidelines and regulations. We took appropriate precautions to prevent any risk or burden on the infants.
Video-based movement analysis. Movement measurement. We recorded the infants in a supine position without any marker using a single video camera (HDR-PJ760V, Sony, Japan) situated just above them. The video camera was fixed directly above and parallel to the crib surface, which was covered by a unicolor fabric spread. The video images were stored on a personal computer with a frame rate of 30 Hz. The infants were recorded approximately twice a week until 15 weeks of corrected age (see Table 1). In our video image analysis, we used the periods during which the infants could be seen to be moving (mean: 249.7 ± 145.5 s; median: 180 s; range: 30-690 s), avoiding any period of sleeping and crying. We also excluded the periods that seemed to have external stimuli, such as parental interference, from the analysis. Table 1. Measurement period and number of measurements.
Scientific RepoRtS | (2020) 10:16827 | https://doi.org/10.1038/s41598-020-74006-y www.nature.com/scientificreports/ Feature extraction. First, a background difference image was created from the measurement image. Using this image, we calculated the changes in the body posture (A k ) P l , expressing the size of the infants' body as follows: where A k (k = 1, 2, . . . , 9) represents the area of the infants' body. The left upper limb, right upper limb, left lower limb, and right lower limb are denoted as A 1 , A 2 , A 3 , and A 4 , respectively. By combining these four areas, the upper and lower body, the left and right sides, and the whole body were determined as A 5 , A 6 , A 7 , A 8 , and A 9 , respectively. In addition, (x, y) (x = 1, 2, . . . , W; y = 1, 2, . . . , H) are the image coordinates, l (l = 1, 2, . . . , L) is the frame number of the image in the video, and O l (x, y) is the pixel value of the background difference image (where O l (x, y) = 1 , if the pixel is white, and O l (x, y) = 0 , if it is black). Moreover, using the image center of gravity (COG) calculated from the background difference image, the COG velocity ( G v l, x , G v l, y ), which is the distance moved by the COG between adjacent frames, and COG fluctuations, which is the change in COG from the average COG position ( G d l, x , G d l, y ) between 1 ≤ l ≤ L , were calculated using the following equations: where (A 9 ) P avg is an average value from the maximum value to the E-th value ( E ≤ L ) in (A 9 ) P l between 1 ≤ l ≤ L . It should be noted that G v 1,x = 0 and G v 1,y = 0 . G l,x and G l,x represents the COG coordinates calculated from the background difference image by the following equation: Next, an interframe difference image was created from the measurement image and motor alteration (A k ) M l , expressing the magnitude of the movement with respect to the body size; it can be calculated as follows: where O ′ 1 (x, y) = 0. The analysis parameters were set to L = 900 and E = 10 . We used the system developed by Tsuji et al. for feature extraction and movement analysis 14 . Because W and H depend on the measurement environment, they differ for each image.  Table 2). In the table, A 5 , A 6 , and A 9 represent the upper-, lower-, and whole-body areas of an infant, respectively. The evaluation indices were calculated every 30 s; subsequently, the average daily values were calculated. In addition, each index was standardized by calculating its average value and standard deviation within each participant.

Analysis of longitudinal changes in movements.
The changes in each evaluation index with the corrected age were statistically analyzed. The developmental changes have been reported as linear and u-shaped 11 ; therefore, linearity and u-shaped tests were performed on the evaluation indices with the corrected age. In the linearity test, the significance of the correlation coefficient between an evaluation index and corrected age was tested with a significance level of 5%. In the u-shaped test, a two-lines test 15 was used, wherein an extreme value of the u-shaped change was first calculated based on the approximation by a quadratic function. The significance of the correlation coefficient was then tested for each side (right and left) of the data, separated by the extreme value. A significant u-shaped change was observed if the two correlation coefficients had opposite signs and were individually significant. When the two-lines test demonstrated a change from a decrease to an increase, with the extreme value as the boundary, the trend was defined as u-shaped, and when it showed a change from an increase to a decrease, it was defined as inverted u-shaped. If significant results were obtained for both the linearity and Scientific RepoRtS | (2020) 10:16827 | https://doi.org/10.1038/s41598-020-74006-y www.nature.com/scientificreports/ u-shaped tests, the trend that yielded smaller residual sum of squares among the approximate straight lines was adopted. If the results were not significant, the index was classified to be uncorrelated with the corrected age. PCA was performed to reduce and interpret the information obtained from the 25 indices. Let X ∈ R n×25 ( n = 272 : 272 is the number of samples) be the matrix in which the 25 indices for all participants are arranged chronologically. Note that each index in X was standardized over all participants to have a mean of zero and unit variance. PCA was then conducted via singular value decomposition using the following equation: where U ∈ R n×n and W ∈ R 25×25 are square matrices and ∈ R n×25 is a rectangular diagonal matrix. The columns of W denote the eigenvectors of the covariance matrix and represent the loading vector of the principal component space. U is the principal component score and represents the projection of X on the principal component space. PCA was used to examine the relationship between the principal component scores with the highest contribution rate and corrected age. We also evaluated the principal component space by confirming the indices with large loadings. The principal components extracted through PCA were selected based on the Kaiser criteria 16 , with an eigenvalue greater than one. Figure 2 illustrates examples of the motor alteration (upper body: (A 5 ) M , lower body: (A 6 ) M ) and power spectral density of the COG fluctuations (x-axis: G d x , y-axis: G d y ) of Participants A and B at 2 and 11 weeks, respectively. Examples for other participants (C-I) are presented in Figs. S1 and S2. Figure 3 shows the changes in each index value with the corrected age in all participants. For indices that produced significant results in the linearity and u-shaped tests, one and two straight lines have been shown in the figure, respectively. Among the 25 indices, 9 positive, 1 u-shaped, and 7 inverted u-shaped correlations were obtained. Eight of the indices were uncorrelated with the corrected age. The mean and standard deviation of the extreme values of the u-and inverted u-shaped changes were 9.92 ± 0.98 weeks, which converged to the same approximate value. Table 3 presents the correlation coefficients and p values of the indices for which significant relationships were obtained. Figure 4 shows the interrelationship between the u-shaped and inverted u-shaped indices. The horizontal axis shows the standard deviations of the rhythm of the COG fluctuations (A 9 ) I 12 x , which is the only u-shaped index, and the vertical axis represents the evaluation indices with an inverted u-shaped. Only the three highest correlation coefficient values ( |r| ≥ 0.4 ) are shown. Table 4 presents the principal component loading of each index in the principal component, whose eigenvalue is greater than one according to the Kaiser criteria. Figure 5a illustrates a biplot visualizing the influence of each index on the principal component with respect to the first and second principal components. Figure 5b shows the first principal component score with the corrected age. The correlation coefficient and p value obtained after the test are also shown in Fig. 5b.

Discussion
In Fig. 2a, the motor alteration (A 5 ) M , (A 6 ) M of the upper and lower body at a corrected age of 2 weeks is smaller than that at 11 weeks; it should be noted that the motor alteration of the lower body is very small. Moreover, in Fig. 2b, the power spectral density of the COG fluctuations (G d x , G d y ) at 2 weeks of corrected age was concentrated in the narrow band of the low-frequency range, in comparison to 11 weeks of corrected age. These tendencies were also observed in other participants (see Figs. S1 and S2), and they suggest that as the  www.nature.com/scientificreports/ development progresses, whole-body movements may increase and become more complex by incorporating a variety of velocities. In Fig. 3, we can observe the changes in the movements of participants with the corrected age; accordingly, the different changes in each index can be confirmed.
An index that demonstrates a positive correlation with the corrected age denotes the symmetry of movements (A 5 , A 6 ) I 6 , representing the correlation between the upper-and lower-body movements. This positive correlation indicates that as development progresses, the upper-and lower-body movements become synchronized. We believe that increased movement symmetry can be attributed to lifting legs 17 , which is a spontaneous movement. The lifting of legs increases movements in the lower body as well as the contact between knees and hands. In addition, the rhythm of motor alteration (A 5 ) I 7 , (A 6 ) I 7 and the rhythm of the COG fluctuation (A 9 ) I 11 x , (A 9 ) I 11 y increased significantly, showing that the movement velocity increases with development. These indices increase owing to spontaneous swiping movements 17 , which have a large amplitude and fast velocity. Moreover, the standard deviations of movement rhythm ( (A 5 ) I 8 , (A 6 ) I 8 , (A 9 ) I 10 x ) may increase owing to the increase in fast motions.
It should be noted that only the standard deviations of the rhythm of the COG fluctuations (A 9 ) I 12 x exhibited a u-shaped change, a finding that corroborates previous reports 11 . The changes in this index demonstrate that complicated movements are first simplified, and then become complex again. Therefore, it is possible that the (A 9 ) I 12 x index reflects the conventionally reported u-shaped change.
Seven indices were confirmed to show inverted u-shaped changes. However, the phenomenon of movement change characteristics taking an inverted u-shape with development has not been reported in the past. Therefore, we decided to find the relationship between such indices and conventionally reported the u-shaped changes (see Fig. 4). In Fig. 4, we confirmed that the indices showing u-and inverted u-shaped changes were significantly negatively correlated. The vertical axes of Fig. 4a-c show the movement count (A 6 ) I 3 and rhythm of the COG velocity (A 9 ) I 9 x , (A 9 ) I 9 y , which reflect the movement degree and velocity. Therefore, when the movement is complex, it is possible to confirm that a relatively smaller number of slow movements are performed, and when the movement is simple, a larger number of fast movements are performed. Infants must practice and organize many movements to gain mature dexterity; this can be implemented by sending sensations from the body muscles and joints to the brain and integrating them 18 . Hence, infants may increase the sensory input Table 3. Correlation coefficient and p value in Fig. 3. The u-and inverted u-shaped indices have two correlation coefficients and p values, which were calculated for the left-and right-sided data, separated by the extreme value.  www.nature.com/scientificreports/ potentially reflect the large individual differences accompanying the development. In future work, we plan to increase the number of participants and study them in detail. Figure 5b shows that the first principal component (PC1) with the corrected age acquired an inverted u-shape after being plotted. Figure 5a and Table 4 illustrate that the indices with the largest main component loadings of PC1 included movement frequency (A 5 ) I 1 , movement count (A 5 ) I 3 , (A 6 ) I 3 , and rhythm of COG velocity (A 9 ) I 9 x , (A 9 ) I 9 y , which all demonstrated an inverted u-shape. Therefore, increasing or decreasing the frequency and velocity of movement to an inverted u-shape at approximately the same time may be important in infant development.
Interestingly, in Fig. 5b, the extreme value of the inverted u-shape occurred at approximately 10 weeks. In healthy infants, 9-10 weeks of corrected age is the time at which writhing GMs transition to fidgety GMs 19 . In general, writhing GMs are movements of small to moderate amplitude and slow to moderate speed, whereas fidgety GMs are movements of smaller amplitude and moderate speed with variable acceleration of the neck, trunk, and limbs 20,21 . The transition from writhing GMs to fidgety GMs occurs at 6-9 weeks of the corrected age, with sporadic fidgety patterns in this period [22][23][24] . Accordingly, infants gradually acquire faster and more complex movements with the transition from writhing GMs to fidgety GMs. It has also been reported that fidgety GMs become slower and smoother movements at 11-16 weeks of the corrected age after the transition to fidgety GMs 24 . These previous findings correspond with our results stating that the indices related to the frequency and speed of motion that strongly contributed to PC1 increased toward the extreme value and then decreased, i.e., the inverted u-shaped change was observed around the extreme value of 10 weeks. Therefore, the extreme value of the inverted u-shaped PC1 and GM development state may be related. In the future, if we could confirm the loss of inverted u-shape or deviations in their extreme values by analyzing participants who may have an abnormality, we may be able to determine whether the findings obtained in this study can be applied for the early detection of a disorder.
In this study, we analyzed nine infants to confirm the changes in their spontaneous movements from markerless video images. In these experiments, we confirmed that the 25 evaluation indices calculated by the system change with the corrected age. The trends in these changes suggest the possibility of classification as previously reported u-shaped changes, as well as newly reported u-shaped and inverted u-shaped changes. Furthermore, PCA suggested that the major changes over time demonstrate an inverted u-shape.
The length of the video images used in the analysis varied (from 30 to 690 s) depending on the participants and recording date. Relatively long periods were extracted from some of the videos, but in most cases, the data length was 180 s. This is because we excluded the periods when the infants were sleeping and crying, or when external interference was observed, from the analysis. Therefore, although the mood of infants was not completely controlled, we believe that the intrinsic spontaneous movement characteristics of infants could be analyzed with a high degree of sensitivity. To conduct more extensive and detailed analyses in the future, it is necessary to establish an experimental protocol to control the behavioral states efficiently, including the moods of infants,  www.nature.com/scientificreports/ as well as the criteria for selecting videos for the analysis. We also plan to perform measurements at neonatal intensive care units and clarify these developmental patterns in greater detail.

Data availability
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.