Computer vision enables short- and long-term analysis of Lophelia pertusa polyp behaviour and colour from an underwater observatory

An array of sensors, including an HD camera mounted on a Fixed Underwater Observatory (FUO) were used to monitor a cold-water coral (Lophelia pertusa) reef in the Lofoten-Vesterålen area from April to November 2015. Image processing and deep learning enabled extraction of time series describing changes in coral colour and polyp activity (feeding). The image data was analysed together with data from the other sensors from the same period, to provide new insights into the short- and long-term dynamics in polyp features. The results indicate that diurnal variations and tidal current influenced polyp activity, by controlling the food supply. On a longer time-scale, the coral’s tissue colour changed from white in the spring to slightly red during the summer months, which can be explained by a seasonal change in food supply. Our work shows, that using an effective integrative computational approach, the image time series is a new and rich source of information to understand and monitor the dynamics in underwater environments due to the high temporal resolution and coverage enabled with FUOs.

From present knowledge it seems that geographic location and local conditions, such as prey availability and depth, influence the diet 13 . In some locations CWC appear to mainly be feeding on zooplankton [13][14][15] . In coastal and oceanic waters in northern Norway copepods are by far the most dominant group within the mesozooplankton, and the most important herbivorous planktotrophic species is the copepod Calanus finmarchicus 16 . Wax ester serve as the storage lipid in C. finmarchicus and can comprise up to 50% of this copepod's dry weight 17 . Calanus finmarchicus also contains the carotenoid astaxanthin 18 , which is a red pigment often used as a food dye to attain coloration in cultivated salmonids and crustaceans 19 .
In addition to the visual documentation with mobile sensor platforms, an increasing number of fixed underwater observatories (FUO) are equipped with digital cameras. This way visual information with high temporal resolation can be collected over a large time period in order to monitor small areas in CWC ecosystems. The in-situ image collections have the potential to provide new insights in ecosystem processes and natural environmental variations on both short (hours and days) and long term (seasons and years). The polyp behaviour of L. pertusa has previously been studied to further understand the biology of the coral and to investigate the possibility of using the polyps as an indicator of stress [20][21][22][23] . In general, the polyp behaviour reflects feeding patterns, and probably other physiological processes 3,24 . During a petroleum drilling campaign in the Norwegian sea in 2009-2010, the first attempt to measure polyp activity of CWC from a fixed platform in situ was made 23 .
Computational support is necessary to efficiently process and interpret the huge amounts of image data gathered both from mobile platforms and fixed ocean observatories. This computational support ranges from data management and development of databases to complex analyses such as image enhancement, segmentation and classification. An additional advantage of computerized image analyses is their objectivity and repeatability. Analyses conducted by humans frequently have low intra-/inter-observer agreement 25 . This problem may be caused by human selective vision and limitation in dividing attention to a variety of simultaneous information 26,27 , such as detecting different organisms and/or their movements over time. Furthermore, detecting small scale changes in object features, such as colour change, is hard for the human brain without a proper scale or colour reference. Several studies have documented that observers have considerable difficulties in recognizing changes in colour or orientation, in particular, when distracting patterns can influence the subjects' visual attention (see for instance 28,29 ).
At present, only a limited number of approaches for computational marine image analysis has been presented. Previous studies have solved different image analysis tasks, such as CWC segmentation in ROV images 30,31 coral classification 32 , marine resource assessment 33,34 or megafauna classification [35][36][37] . Computational methods for image data from stationary observatories have been developed to solve tasks such as object detection/ screening 38,39 , monitoring of shrimp distribution in and underneath a L. pertusa reef 40 , quantification of colour change in L. pertusa over time 41 and detection and classification of polyp activity in selected parts of a L. pertusa reef 42 . Recently, also a short time series study has been published on the gorgonian coral Paragorgia arborea, including manual assessment of visual features of the corals 43 . A comprehensive overview for underwater image pre-processing can be found in 44 .
This study represents the next step, where images computationally extracted from a stationary observatory are combined with other sensor data. Image time series and sensor data are obtained from the Lofoten-Vesterålen (LoVe) ocean observatory (love.equinor.com). The aim of the data analyses is to identify and describe, if any, short and long-term dynamics in the behaviour and visual features of L. pertusa. The main factors that might influence the behavioural changes are discussed. Short-term dynamics refer here to short periods of hours to days and long-term dynamics are defined as period of months or the total time period analysed, about seven months. www.nature.com/scientificreports www.nature.com/scientificreports/ Material and Field Applications the study area. The Lofoten-Vesterålen (LoVe) ocean observatory is located at approximately 250 m depth about 12 km (i.e. ∼20 nautical miles) off the coast of Northern Norway (N 68°54.474′, E 15°23.145′, see Fig. 1). The location is a biological hot spot area in the Norwegian sea, hosting the main spawning area for the North Atlantic cod and has more than 300 L. pertusa reefs 45 . equipment and sensors. The ocean observatory was deployed in October 2013. In addition to a waterproof housing (METAS DSF5210) for a digital camera (Canon EOS 550D) and a flash (METAS DSF 4365), the observatory is equipped with a range of sensors (see Table 1 for an overview). In our study, environmental data such as salinity, conductivity, temperature, depth, current speed and direction, and chlorophyll, were correlated with polyp activity. The camera was oriented at an angle of 45° towards a L. pertusa reef. The camera covered a visual field of approximately 10 m 2 , including both live parts of the reef and coral rubble as shown in Fig. 2. One image with a resolution of 5184 × 3456 pixels was recorded per hour.
Current velocity data were checked for consistency of measurement between the two available ADCPs (Nortek Aquadopp and Nortek Continental) in the overlapping part of their range. ADCP alignment was compared with readings from an independent compass (OASITILT sensor cf. online documentation on the data portal). Environmental sensor data (temperature, salinity/conductivity) were checked for quality using internal consistency checks of the equipment (e.g. comparison of temperature series from the AADI device with that accessory sensors from the ADCPs). Data recorded at the LoVe FUO is publicly available and can be downloaded from http://love.equinor.com.

Methods
The image sequence from the LoVe observatory which is considered in this study consists of 4862 images {I 1 , …, I N } recorded in the time from April 3 rd to November 10 th , 2015. In order to extract coral features time series of interest, we used two machine learning based algorithmic approaches that were applied to each image I t . The first approach was used to compute a coral colour feature ξ t for each image and time point that describes the colour of the coral. The second approach was used to describe the relative polyp activity γ t for each image. The approaches are described briefly in the following. A detailed description of all processing steps is also given for both methods in the Supplementary Materials S1 and S2. Additional information can be found in 41,42 . An overview for the computational extraction of both the image-derived time series is shown in Fig. 3. The image analyses have been implemented using the software packages OpenCV (http://openCV.org), NVIDIA DIGITS (http://developer. nvidia.com/digits) and the CAFFE framework (http://caffe.berkeleyvision.org) as well as C++ code developed by the Biodata Mining Group, Bielefeld University. Afterwards we describe the application of linear regression and wavelet coherence analysis that was used to investigate linear relationships between the different time series and their periodicities.
Coral colour time series computation. Monitoring the corals' colour change requires automated extraction of colour features from the digital photos that represent the CWC colour at time point t (a more detailed description for this method is given in the Supplementary Method S1). First, all images are spatially aligned 46 and colour transformed (i.e. → ′ I I t t ) with a fixed white balancing. After this pre-processing step, pixel-labelling based segmentation is applied in order to compute a binary mask image M coral for the live corals (see Fig. 3 upper middle). The pixel labelling function is constructed using a hierarchical hyperbolic self-organizing map 30,47 , which is trained with Gabor features from a small subset of the images. Then, we computed for each image I t ′ the average CieLab colour values 48 for the full image (i.e. all pixels)  www.nature.com/scientificreports www.nature.com/scientificreports/ polyp activity time series computation. A time series γ t describing the relative polyp activity (feeding vs. non-feeding) was computed (and a more detailed description can be found in the Supplementary Method S2). To this end, a region of interest (ROI) was chosen that showed CWC with sufficient contrast and in focus (see green frame and lower row in Fig. 3). After pre-processing the images, 13 image ROIs were displayed to human experts (i.e. the authors JJ, IN, PB) to mark positions of active polyps and inactive ones using the annotation software BIIGLE 2.0 49,50 . Based on the three users' point annotations, a binary mask M ∼ inside the ROI was determined that describes the locations of all polyps (i.e. ∼ M = 1, for all polyp pixels and = 0, otherwise). In order to train, validate and test a deep neural network, image patches of equal size (46 × 46) were extracted at the positions marked by the human experts, that summed up to 2410 image patches showing polyps. A background class was generated by extracting 100 image patches at random locations in the same ROI but outside the mask M ∼ . The number of training sample patches was further increased by flipping (augmentation factor: 2x) and rotating (augmentation factor: 12x) the patches and by adding Gaussian noise to them to avoid overfitting (augmentation factor: 2x). Employing these boosting procedures, we achieved a data augmentation by the factor of 48, i.e. an increase of the training data volume by 48-times. Image patches for all categories (active/inactive polyp) are divided into training (70%), validation (20%) and test (10%) patches. Training and validation set were used for training and parameter optimization. The test was left out in in this step so it could be used for a final assessment of the network's performance for new and "unseen" data. A deep convolutional neural network with a LeNet-5 layout 51 was trained for classifying image patches to feature an active polyp or an inactive one using the NVIDIA DIGITS software. To finally compute the polyp activity γ t for one time point, all pixels (x, y) inside the mask M ∼ are classified by the network to show an element of an active polyp or an inactive one. The network output C(p (x,y) ) represents the likelihood of the image patch p (x,y) centered at pixel (x, y) showing an active polyp, i.e. C(p (x,y) ) = 1) or not (i.e. C(p (x,y) ) = 0) for retracted polyps or other structures. The aggregated number of pixels classified to be "active" in relation to the size of the positive area in ∼ M gives the estimated percentage of active polyps for each ROI  I t , referred to as polyp activity γ t in the rest of this manuscript: Multivariate data analysis. In order to investigate functional relations between the introduced image-derived coral colour ξ t (see eq. 3) and polyp activity γ t (see eq. 4), and the other physical conditions measured by the other sensors ), several analyses were conducted. The different sensors of the LoVe observatory have different sampling rates and show varying short episodes of sensor dysfunctions, the latter additionally causing different kinds of temporal delays between measurements of different sensors. Depending on the type of analysis, a mapping and/or interpolation for the different time series was required. In the following we briefly describe the different methods of statistical data analysis we have applied.
To test linear correlations between the image derived time series (γ t and ξ t ) and sensor time series we computed a time series x t for each sensor f t that had the same temporal resolution as the image time series. Given an original sensor time series f t and the original time points t f { } t | ≠ ∅ of its measurements, we computed the new sensor time series x t by interpolation Pairs of time-series were plotted in scatterplots to identify potential relationships between data sets. Linear correlation was tested between polyp activity, γ t and time series x t . Daily averages ( x , t t γ ) of γ t and x t were computed and tested for pair-wise correlation. For the latter, no linear interpolation was applied, ignoring days where no data was recorded either for γ t and/or x t .
Wavelet-based analysis methods were performed 1) to identify dominant frequencies/scales that contribute to the overall variance in individual time series, and 2) to assess covariation between the image-derived time series and the environmental variables measured by the other sensors. In contrast to a Fourier transformation-based analysis, which discards the time dimension when data are transformed to frequency space, the wavelet transformation retains time and frequency/scale information, thus facilitating identification of patterns in data that have limited temporal extent 52 . Dominant frequencies/scales in individual time series were identified in the bias-corrected wavelet power spectrum 53,54 . The default Chi 2 -test from the R function was used to test for significance of variance contributions at the 95% significance level. Scale-and time dependent covariation between two time series was investigated with a wavelet coherence analysis 55 . Wavelet coherence is the wavelet transformation analogue of a cross-correlation analysis or a Fourier coherence analysis. It is calculated from the wavelet cross-spectrum of two series by dividing it by the power spectra of the individual time series 56 . It is reported as a squared value between 0 and 1, similar in interpretation to a squared correlation. Significance of the wavelet variance and coherence was tested by Monte Carlo randomization tests 54 To allow a pairwise analysis of image-derived time series with other sensor data f t was mapped ( f x t t → ′ ) to interpolate measurements for those time points of the set γ The wavelet transformation was applied to each individual time series with a Morlet mother wavelet function, followed by pairwise coherence wavelet analysis between γ t and all x t . The analyses were conducted in the R Statistical software (R Core Team, 2016) using the addon package "biwavelet" 57 .

Results
To assess the accuracy in polyp activity, several processing approaches were evaluated and compared. A comparison of the human expert annotations showed an inter-observer agreement of 0.86 and an intra-observer agreement of 0.89. The classification accuracy of the trained network was 0.98 for the training data and 0.96 for the validation and test data. The accuracy of the relative polyp activity γ t was assessed for the 13 images, that were annotated by the human experts (more details are given in the Supplementary S2). From the manual annotations, another value for relative polyp activity was computed and compared to γ t . A spearman's rank correlation of the two measurements was calculated to be 0.98.
An overview of all data attributes and temporal resolutions is given in Table 1. Raw data are displayed in Fig. 4a-i. As can be seen in the plots, the time series show some gaps due to sensor problems and/or break downs of the communication with the ocean observatory.
Nevertheless, the time series of polyp activity, coral colour as well as temperature and conductivity (data not shown) show long-term trends and patterns with marked changes starting in May. www.nature.com/scientificreports www.nature.com/scientificreports/ The colour value (ξ t ), describing the redness of the polyp tissue, changes gradually towards more reddish hue from mid of May to August (Fig. 4a). The onset of this change seems to follow a marked peak in the chlorophyll concentration at the end of April (Fig. 4d). The current velocity did not display any clear seasonality patterns (Fig. 4e-h). The variation in depth reflects the tidal variation (Fig. 4i).
Polyp activity γ t increased rapidly between a bottom water temperature 7.4 and 7.5 °C and reached a maximum between 7.5 and 8.0 °C (Fig. 5). At higher bottom water temperatures polyp activity decreased slightly to remain more or less constant beyond temperatures of 8.3 °C (Fig. 5). Pearson correlation coefficients (Supplementary Table S3)  , the values indicated no linear dependency. The periodicity of the different time series was analysed using wavelet transformations. Linear interpolation was used to compute time series with equidistant time points from γ t and f t (see equation (8)). First, single time series were analysed with respect to periodic patterns, starting with the polyp activity γ t . The result of this analysis is visualized in Fig. 6a.
The wavelet spectrum of the polyp activity γ t exhibited only statistically significant variance contributions at scales below 16 hours (black contours in Fig. 6a (left panel), plots without the black contours are shown in the Supplementary S4). These statistically significant regions occurred throughout summer, but particularly from September to November (region indicated with "A" in Fig. 6a), left panel). Although no significant variance contribution in poly activity γ t was detected at the 24-hour scale band in the wavelet variance plot (Fig. 6a, left  panel), the corresponding global wavelet variance spectrum (right panel in Fig. 6a), averaged by scale, exhibited a distinct peak at the 24-hour scale, indicating that polyps showed some increased variation in feeding activity with a 24-hour scale. Additional variance peaks were also found at longer scales, including that of the lunar cycle (approx. 700 h = ~29 days).
The wavelet spectral plot (Fig. 6b (left panel), a plot without black contours is shown in the Supplementary S4) and global averaged wavelet spectrum (Fig. 6b, right panel) of the northward component of the bottom current velocity (Vn1) showed a significant and dominant contribution of the 24-hour scale to the overall variance, indicative of a tidal dominance in the current velocity. A bit surprisingly, no spectral peaks were found at the 12.5 h scale, which may be due to interactions of the tidal wave with the basin morphology. Some significant contributions of smaller time scales were found as well but were mainly restricted to the beginning and end of the time series.
Similar analyses of the wavelet variance demonstrated a strong contribution of the lunar phase to the overall variance in the eastward current velocity component and a dominant tidal (12.5 h) contribution to the water depth (data not shown). ) are plotted against daily averages for the polyp activity (γ t ). The estimated pearson correlation coefficient between both is 0.54. ) (see b)) and the corresponding scale-averaged global power spectra (right panels in the subplots). Significant contributions to the time series' variance (Chi 2 -test, 95% significance level) are indicated by the black contours. The horizontal lines in the global wavelet spectra mark the tidal scale (12.4 h, daily (24 h) and lunar (~707 h) scales.
www.nature.com/scientificreports www.nature.com/scientificreports/ Significant coherence (covariation) between the polyp activity and the northward current velocity was found at various scales up to 256 hours (~10 days), throughout the entire measurement period (Fig. 7 (left panel) and Supplementary Fig. S5). At the 24-hour scale the two signals tended to be in phase (arrows predominantly pointing to the right in Fig. 7 left panel), whereas they tended to be in counter-phase at larger scales (arrows predominantly pointing to the left).
In the Supplementary Fig. S6 we show a plot of polyp activity and current velocity Vn1 for a selected interval of 4 weeks to illustrate coherences (see Supplementary Fig. S3).

Discussion
Our results show that the polyp behavior of the monitored L. pertusa display different, alternating temporal patterns, or periodicities. The analyses indicate that the main factors influencing the polyp behavior and color varies both with respect to the internal relationship between the various factors measured and with time. Behavioral periodicity has been observed in many warm water corals (see references in 58 ) but such patterns have not been previously documented in CWC in situ.

Long term dynamics.
In general, the long-term (7 months) trend of polyp activity and coral color ( Fig. 4a,b) follow a similar pattern with a low level from April to May, followed by a gradual increase from May to July. The onset of the increase follows an early sharp peak in chlorophyll content (Fig. 4d). This peak marks the spring bloom of phytoplankton, which is the basis for the growth of the zooplankton communities, dominated by calanoid copepods. The long-term change in the colour in the L. pertusa reef (Fig. 4a) was first published in 2016 38,41 and is further supported by our study. We suggest that the colour changes documented in L. pertusa between April and November, is a result of increased feeding on calanoid copepods containing the red pigment astaxanthin. According to 13 and 15 , copepods and other planktonic animals within reach of the reefs during summer constitute the main food sources for L. pertusa together with phytodetritus 59,60 . The increased polyp activity is likely to be related to feeding activity, and as a high food uptake continues throughout the summer and early fall, the pigments from the food probably accumulate in the coral tissue. Changes in polyp activity related to feeding activity has previously been documented in laboratory experiments 61,62 . The calanoids show both seasonal and diurnal vertical migration patterns. During winter, the populations enters diapause in deep and cold water. Patches of high abundance of overwintering C. finmarchicus in the Lofoten Basin, just west of our study site, has been documented at 550-800 meters 63 . In mid-late winter, the population ascends to surface waters where it matures and spawns (see for example 64 ). During its life cycle C. finmarcicus goes through six Nauplius and five Copepodite stages (CI-CV). The maximum abundance of the first generation CI often occur during or right after the peak of the phytoplankton bloom (reviewed in 65 ). In June and July, the first generation of CIV and CV starts to migrate into the deep for diapause and a second generation starts their migration in August and September 16,63,66,67 .
In addition to the seasonal migration, the calanoid perform diurnal vertical migration (DVM). This is a strategy to avoid visual predators (e.g. fish) [68][69][70] . Copepods reside deeper in the water column during the day than at evening and night time, when they migrate towards the sea surface to graze on phytoplankton (e.g. 66 ).
With the maturation of the calanoids, the depth of the migration increases towards the end of the summer 16,66,69 before it is followed by the migration to diapause. The CWC habitats in Norway are relatively shallow 71 and it is therefore likely that the seasonal migration, in combination with the DVM, accommodate the CWC with high quality food during summer and autumn. www.nature.com/scientificreports www.nature.com/scientificreports/ Our suggestion that copepods are causing the colour change in L. pertusa is further supported by 16 , showing that the storage lipid signature of L. pertusa contained a higher level of calanoid copepod lipid biomarkers at the shallow Mingulay reef (130 m) than the deeper reefs at Rockall (900 m) and New England (1200-1300 m). Additionally, the Mingulay corals contained the highest number of wax esters of the sites, which is the storage lipid of the copepod C. finmarchicus 17 .
Temperature (Fig. 4c) also shows long-term seasonal changes, with a steady increase from April/May until end of the measuring period in mid-November. Between May and September, there is a positive correlation (r 2 = 0.54) between polyp activity and temperature. Such a correlation has also been observed in laboratory experiment 72 . However, from September until mid-November, the temperature continues to increase while the polyp activity decreases. L. pertusa occur within a rather wide temperature range worldwide, from 4 to 14 °C, and has a high tolerance to an unpredictable temperature regime 73 . This may suggest that food availability influence polyp activity to a higher degree than temperature. But given duration of the measurements and the lack of verification of food available in the data, there are currently not enough data available to conclude on how the temperature or a combination of temperature and food influences the polyp activity of L. pertusa. Analyses on longer time series in the future will determine whether the correlation with temperature and other drivers at the seasonal scale is of ecological significance or just an artefact of the cyclicity in two auto-correlating time series with a limited temporal extent.
short term patterns. The short term (hours and days) dynamics (i.e. short periodicities) of the different sensors seem to be more complex than the long-term (months) changes controlled by factors varying both diurnally (Figs 6 and 7) and with the lunar phase (tidally).
The wavelet coherent study for current speed and polyp activity showed a match for 24 hours and an approximate match for 33 days. The results from the coherence analysis suggest opposite responses of polyp feeding activity to tidal currents relative to residual currents (larger than 24-hour). However, the coherence at the 24-hour scale does not necessarily have to originate from tidal variability. Daily vertical migration of prey species could also induce variability of polyp feeding activity at a 24-hour scale. From August onwards significant coherence was also detected at the lunar scale. Signals were roughly in counter-phase at this scale.
Even though no long-term pattern was documented for current speed and direction in the present study, it is documented that food transport to corals is directly linked to current speed and direction 74 . Laboratory studies have shown that slow current speeds, with velocities less than 7 cm/sec, are optimal for feeding in CWC 61,62 . At higher flow velocity fewer polyps are active, likely due to greater drag on the tentacles and lower probability of food capture 61,62 . The authors of 62 also describe the variety of food capture efficiency with food source in addition to water flow. Furthermore, also a field study 13 has indicated changes in the proportion of polyps in different expansion states caused by natural stress from water motion around the polyps and/or by different water quality or food content related to changes in current speed and direction.
The tidal wavelength of 12.25 hours is directly linked to variation in current speed and direction. Because of a tidal periodicity of 12.25 hour, high and low tide does not occur at the same time every day. This implies that there are days within this cycle when the currents are disadvantageous while the zooplankton is present around the corals, and vice versa. During a tidal cycle, highest current velocities occur twice (6.12 hours apart). A combination of wave patterns can feature interferences and this may explain why the corals' polyp behaviour seems to swap between different periodicities (Fig. 6a).
Our findings demonstrate the importance of using a sampling frequency relevant for capturing dynamic patterns at the scale they occur. Which time scale is relevant for sampling of biological and environmental data depends on the aim of the study.
Conclusions and future perspectives. The high temporal resolution and coverage of the data measured at the LoVe ocean observatory opens up for a new insight, and jet unknown opportunities for understanding natural variations and the dynamic in CWC reefs. These first results reveal that various parameters influence L. pertusa at different time scales, and that the various periodicities also influence each other.
Understanding the natural variations and dynamics influencing key species or ecosystems is essential to interpret environmental monitoring data.
To gain more insight and knowledge about these interactions there is first and foremost a need to gather more experience and better statistics for both long-and short-term changes. Furthermore, to generalize the present findings, the methodology must be further developed for applications in broader areas. The results also show that to reduce the uncertainty in our analysis there is a need of technology development of sensors that can measure and specify food availability with high temporal frequency and coverage. The documentation of L. pertusa changing color through the year is one example of new knowledge that is a result of interdisciplinary collaboration on the LoVe data.
In a broader perspective, the anticipation of L. pertusa feeding on C. finmarcicus causing this color change indicates a strong predator-prey interaction between the two species. We can therefore expect that fluctuations in zooplankton also could influence L. pertusa's general health and reproductive cycles. With more than 10.000 L. pertusa reefs documented in Norwegian waters this new knowledge should also be considered included in CWC ecosystem modelling. As indicated in this study L. pertusa might be an important predator that at present is missing in the modelling. Ultimately, this new insight could also have implications for understanding details in food web dynamics involving commercial important fish stocks grazing on copepods, such as herring.

Data Availability
All Images and sensor data used in this work were made available via the Lofoten-Vesterålen Ocean Observatory web portal (love.statoil.com). The single time series are also provided with this submission (data_matrix.zip).