Object based classification of a riparian environment using ultra-high resolution imagery, hierarchical landcover structures, and image texture

Land cover mapping is an important part of resource management, planning, and economic predictions. Improvements in remote sensing, machine learning, image processing, and object based image analysis (OBIA) has made the process of identifying land cover types increasingly faster and reliable but these advances have not been able to utilize all of the information encompassed within ultra-high (sub-meter) resolution imagery. There have been few known attempts to try and maximize this detailed information in high resolution imagery using advanced textural components. Hierarchical land classes are also rarely used as an attribute within the machine learning step of object-based image analysis. In this study we try to circumnavigate the inherent problems associated with high resolution imagery by combining well researched data transformations that aid the OBIA process with a seldom used texture transformation in Geographic Object Based Image Analyses (GEOBIA/OBIA) known as the Gabor Transform and the hierarchal organization of landscapes. We will observe the difference made in segmentation and classification accuracy of a random forest classifier when we fuse a Gabor transformed image to a Normalized Difference Vegetation Index (NDVI), high resolution multi-spectral imagery (RGB and NIR) and Light Detection and Ranging (LiDAR) derived canopy height model (CHM) within a riparian area in Southeast Iowa, United States. Additionally, we will observe the effects on classification accuracy when adding multi-scale land cover data to objects. Both, the addition of hierarchical information and Gabor textural information, could aid the GEOBIA process in delineating and classifying the same objects that human experts would delineate within this riparian landscape.


Scientific Reports
| (2022) 12:11291 | https://doi.org/10.1038/s41598-022-14757-y www.nature.com/scientificreports/ errors by decreasing noise or attempting to increase classification accuracy using either a rule-based classifier or fuzzy classifier [15][16][17][18][19][20] . This approach does not leverage the information supported by landscape hierarchy theory, a framework for scaling and understanding the relationship between spatial pattern and ecological process. Within landscape ecology, O'Neill et al. 21 conducted a meta-analysis of hierarchal frameworks in biology. They concluded that the various scales within an ecological system define, or limit, one another in a way that could support that a super-object could be a useful property in defining sub-objects within a landscape. If multiple classifications are performed at several scales, the attributes of larger scaled objects (i.e. super-objects) can be tied to the smaller scaled objects (i.e. sub-objects), thus potentially increasing the classification accuracy of the sub-objects. This approach uses what we know about the organization of complex multipart ecosystems. However, remote sensing analytical techniques, such as GEOBIA, have not incorporated hierarchical landscape ecology theory in classification methodologies nearly as in depth. The primary use of hierarchical landscape organization in OBIA is the iterative process of classifying a landscape into sub-classes from super classes. An example of this approach can be found in Mao et al. 22 . In their paper they first classified their segmented image into wetland/non-wetland classes and then classified the wetland objects into smaller and smaller subclasses using thresholds or rule-based classifiers.
As opposed to dissecting classified objects into smaller subclasses, in this paper we use the information from a separate classification that uses a higher-level class schema to contribute to the classification of smaller sub-objects that uses a lower-level class schema. This allows us to examine the role of hierarchical information inherent in natural landscapes and image processing techniques to better develop automated replication of visual interpretation of natural landscapes. Specifically, we examine the impact of hierarchical segmentation on object classification, relative to a multi-scale visual classification, of high-resolution aerial imagery.
Segmenting images into hierarchical objects consistent with visual delineation could also be enhanced by image enhancement that is consistent with human interpretation. The Gabor textural transformation has been lauded for replicating the same directional textural information that humans use to identify and interpret objects [23][24][25] . However, few studies have been conducted that investigate the use of this transformation for objectbased image analysis 26,27 .
We aim to examine the accuracy improvement from hierarchical delineation and classification of complex floodplains by combining well-researched data transformations, that aid the OBIA process, with a seldom-used texture transformation in GEOBIA known as the Gabor Transform. We used a random forest classifier, three band (near-infrared, red, and green) 7.9-cm imagery, Normalized Difference Vegetation Index (NDVI) and a Light Detection and Ranging (LiDAR) derived canopy height model (CHM) within a riparian area in Southeast Iowa; allowing us to observe the difference in segmentation and classification accuracy that a Gabor transform and hierarchical land cover data can provide to object based analysis.

Data and study area
Data. The aerial imagery, used for our study, is a three band (near-infrared, red, and green) 7.9-cm resolution image taken with an Applanix 439 Digital Sensor System on May 18, 2014. The images were taken by the U.S. Fish & Wildlife Service, Region 3, and the U.S. Geological Survey's Upper Midwest Environmental Sciences Center. The CHM used in this paper is from the Iowa LiDAR Project 28 . LiDAR data was downloaded as several four-square kilometer, las tiles that encompassed the study area and was originally collected on May 5, 2010. The files were converted into a last-return digital terrain model (DTM) TIFF files using the ArcGIS Lidar Analyst Extension. The CHM was then created by subtracting the DTM from a digital surface model (DSM) derived from the first return values. All imagery and vector files were projected and processed within the Universal Transverse Mercator zone 15 spatial reference. All sets of data were collected during leaf-on conditions. Reference polygons were hand delineated and classified by experts from US Fish and Wildlife Service Region 3, Port Louisa National Wildlife Refuge, and the USGS Upper Midwest Environment Sciences Center. This data allowed us to perform a two-tier classification as the visual classification used two object classification schemas; a broad 7-class scheme and a narrower 13-class scheme. Using these schemas to train and base our classifications upon, we examined the improvement in classification accuracy of floodplain sub-objects. Study area. The Horseshoe Bend Division of the Port Louisa National Wildlife Refuge (NWR) is a mixture of grass and wetland habitat along the Iowa River four miles upstream from the confluence of the Iowa and Mississippi River. This 2606-acre NWR is composed of grassland, wet meadows, forest, and semi and permanently flood emergent wetland habitat. Prior to the 1993 flood, this land was primarily used for agricultural purposes and was protected from flooding by a levee along the Iowa River. Since then, the levee has broken along the upper reach where the Iowa River intersects the NWR making the land susceptible to frequent inundation. This study area is in Port Louisa County Southeast of Wapello, Iowa (see Fig. 1).

Methodology
Gabor transform. The Gabor transform has rarely been used as a feature in a landscape classification OBIA approach but has been used in other OBIA processes such as fingerprint enhancement and human iris detection and for data dimensionality reduction 24,[29][30][31][32][33][34][35] . Gabor filters are a bandpass filter applied to an image to identify texture. The different Gabor bandpass filters mathematically model the visual cortical cells of mammalian brains and thus is expected to improve segmentation and classification accuracy when compared to a human delineated and classified image 26,27 .
Samiappan et al. 36 compared Gabor filters to other texture features (grey-level co-occurrence matrix, segmentation-based fractal texture analysis, and wavelet texture analysis) within the GEOBIA process, of a wetland, using sub-meter resolution multispectral imagery. These Gabor filters performed comparably, in overall  36 where they found that Gabor features were one of the least useful/influential that contributed to the classification of a mostly agricultural landscape. Gabor filters are a Fourier influenced wavelet transformation, or bandpass filter, that identifies texture as intervals in a 2-D Gaussian modulated sinusoidal wave. This modulation differentiates the Gabor transform from the Fourier transform 23,26 . These Gabor transformed wavelets are parameterized by the angle at which they alter the image and the frequency of the wavelet. Rather than smoothing an image at the cost of losing detail through Fourier transforms or median filters, Gabor transformed images identify the repeated pattern of localized pixels and gives them similar values if they are a part of the same repeated sequence. Gabor features can closely emulate the visual cortex of mammalian brains that utilize texture to identify objects 26,27 . This is based on the evaluation  where f is the user-determined frequency (or wavelength); θ is the user-determined orientation at which the wavelet is applied to the image; γ and n are the standard deviations of the Gaussian function in either direction 23,38 . These parameters define the shape of the band pass filter and determines its effect on one-dimensional signals. Daugman 26 , created a 2-D application of this filter in Eq. (2); where u' = ucos − vsin θ θ and v' = usin − vcos θ. In order to implement Gabor filters on multi-band spectral images, we used Matlab's Gabor feature on the University of Iowa's Neon high performance computer (HPC) 40 which has up to 512 GB of RAM, which was necessary for processing these images. The first implementation of Gabor filters was performed on a 1610 × 687 single band pixel array (a small subset of the study area), a filter bank of 4 orientations and 8 wavelengths, on a 32 GB RAM computer, and took approximately 8 h to complete. Filter banks are a set of Gabor filters with different parameters that is applied to the spectral image and are required to identify different textures with different orientations and frequencies. By lowering the number of wavelengths from 8 to 4 on an 8128 × 8128 single band pixel array on the same machine 32 GB RAM, the processing was reduced to an hour. Using the HPC, this was further reduced to approximately 90 s using the same filter bank. Before implementing on the HPC, the original spectral image was divided into manageable subsets with overlap in order to prevent 'edge-effect. ' These images were converted to greyscale by averaging values across all three bands 33 . When wavelengths become too long, they no longer attribute the textural information desired from the image and therefore add unnecessary computing time. The wavelengths that were used for the filter bank were selected as increasing powers of two starting from 2.82842712475 ( 24/ √ 2 ) up to the pixel length of the hypotenuse of the input image. From this, we used only 2.82842712475, 7.0710678, 17.6776695, and 44.19417382. The directional orientation was selected as 45° intervals, from 0 to 180: 0, 45, 90, 135. These parameters were based on the reasoning outlined within Jain and Farrokhina 25 . More directional orientations could have been included but four were used for computational efficiency. The radial frequencies were selected so that they could capture the different texture in the landscape represented by consistent changes in pixels values within each landcover class. When frequencies are too wide or fine of a width they no longer represent the textures of the different landcover classes and thus are not included. This selection of filter bank parameters are similar or the same as other studies that look into the use of Gabor features for OBIA 25,30,31 .
From the different combinations of parameters (four directions and four frequencies) in the Gabor Transform filter bank, sixteen magnitude response images were created from the converted greyscale three band average image. To limit high local variance within the output Gabor texture images, a Gaussian filter was applied. The magnitude response values were normalized across the 16 different bands so that a Principal Component Analysis (PCA) could be applied. The first principal component of the PCA, from these Gabor transformed images, was used for this study since it limits the computation time to process 16 separate Gabor features, in addition to the other data sources, while still retaining the most amount of information from the different Gabor response features. The Gabor band that was used for this study can be viewed in Fig. 2.
Segmentation. For this study, we used the watershed algorithm for the segmentation of GEOBIA, implemented by ENVI version 5.0 Feature Extraction tool, due to its ubiquitous use within GEOBIA, its ability to create a hierarchy of segmented objects, and support within the literature as a reliable algorithm 37,[41][42][43] . The watershed algorithm can either use a gradient image or intensity image for segmentation. Based on the observed results, this study used the intensity method. The intensity method averages the value of pixels across bands. Scale, a user-defined parameter, is selected to identify the threshold that decides if a given intensity value within the gradient image can be a boundary. This allows the user to decide the size of the objects created. A secondary, user-defined, parameter defines how similar, adjacent, objects need to be before they are combined or merged. The user arbitrarily selects the parameter value based on how it reduces both under and over segmentation. The parameters selected for this study were visually chosen based on a compromise between over and under segmentation relative to the hand demarcated objects.
The merging of two separate objects was based on the full lambda schedule where the user selects a merging threshold t i,j which is defined by Eq. (3): where O i is the object of the image, |O i | is the area of i , u i is the average of object i , u j is the average of object j , �u i − u j � is the Euclidean distance between the average values of the pixel values in regions i and j , and To compare the segmentation of a riparian landscape, with and without Gabor features, we conducted segmentation on two separate sets of data. One dataset was a normalized stacked layer of NDVI and CHM (see Fig. 3) with the original multispectral image used as ancillary data; the other dataset differed only by the inclusion www.nature.com/scientificreports/ of the Gabor feature. For both instances, the bands were converted to an intensity image by averaging across bands rather than being converted into a gradient image for segmentation. The dataset that included the Gabor features had a scale parameter set at 30 with merge settings at 95 and 95.7 for the sub and super-objects, respectively. The dataset that did not include the Gabor features had a scale parameter of 10 with merge settings at 95.6 and 98.5 for the sub and super-objects, respectively. This resulted in the creation of 87,198 and 62,905 segments for the sub and super objects, respectively, that were created when the Gabor feature was included. 191,050 and 51,664 segments were created for the sub and super objects when the Gabor features, respectively, were not included within the segmentation process. As you will see in the next section, these segments also represent the number of training data that will be included within the supervised classification.
To create a hierarchy of land cover classes, two sets of segmentation parameters needed to be selected for each dataset. One set of parameters would be used for the sub-objects within the hierarchy and the other set would be used to create super-objects. All parameters used the intensity and full lambda schedule algorithms for the watershed method. The only setting that changed between the sub and super-objects, for either dataset, was the merge parameter which helped maintain similar boundaries as much as possible. Despite this, boundaries could moderately change due to the Euclidean distance, between the pixel values of i and j , changing from the merging of objects; causing t i,j to cross the threshold which results in a new boundary being drawn. A representation of these results can be viewed and visually compared to the hand demarcated objects in Fig. 4.  shallow marsh, submerged aquatic vegetation, upland forest, wet forest, wet meadow, and wet shrub. Not every class from the 7-class scheme will have a sub-class (i.e. developed, open water) but some do for example wet and upland forest are sub-objects of the forest class and wet meadow and shallow marsh are sub-objects of marsh. Figure 5 visually illustrates both classification schemes across the study area. ENVI's feature extraction tool calculates several landscape, spectral, and textural metrics. These attributes were used for each random forest classifier. The Gabor and Hierarchical features will be included selectively to be able to compare their contributions to the (out-of-bag) OOB classification errors. When Gabor features are included within the classification, they are computed the same way as the other image bands. The number of trees, that were randomly generated, was large enough (n = 250) to where the Strong law of large numbers would take effect as indicated by the decrease in the change of accuracy. The default number of variables randomly sampled as candidates at each split variable (mtry parameter) was the total number of variables divided by 3 for each dataset. R also generates two separate variable indices: mean decrease in accuracy and mean decrease Gini. Mean decrease in accuracy refers to the accuracy change in the random forest when a single variable is left out. This is a practical metric to determine the usefulness of a variable. The Gini index measures the purity change within a dataset when it is split based upon a given variable within a decision tree. The random forest classification accuracy will be based on the OOB error. The random forest algorithm trains numerous decision trees on random subsets of the training set leaving out a number of training samples when training each decision tree. The samples that are left out of each decision tree are then classified by the decision tree that they were not included within during the training step. The OOB error is the average error of each predicted bootstrapped sample across the ensemble of decision trees within the random forest algorithm. Figure 6 illustrates how the Gabor and hierarchal features were included within the classification of the super and sub-objects. www.nature.com/scientificreports/ Hierarchical scheme. To attribute the hierarchical structure to the sub-objects, we first classified the larger segments that were created with and without the Gabor features using the broader 7-class scheme. These classified super objects were then converted to raster to calculate the majority overlap with the smaller sub-objects. This gave the sub-objects an attribute, the broader 7-class scheme, that could be used to contribute to the classification of the sub-objects with the finer 13-class scheme. This builds the hierarchical relationship between the two class schemes into the supervised classification of the sub-objects. Figure 6 illustrates how the hierarchal structure was included within two of the four sub-object's list of features used within classification. This methodological approach aligns with O'Neill et al. 21 landscape ecology principle that a super-object's class could be a useful property in defining or predicting a sub-object. This is also different than the more common rule-based approach of iteratively classifying the landscape into smaller and smaller sub-classes 22 .

Segmentation assessment.
Most studies rely upon the accuracy assessment of their classifiers to provide support for their analysis results. However, this does not provide evidence whether a new data fusion technique improves the ability to delineate objects of interest within an image. To assess the performance of our segmented   37 .
Our segmentation results were evaluated using an empirical discrepancy measure, used frequently in image segmentation evaluation 37,46,47 . Discrepancy measures utilize ground truth images that represent the "correct" delineated/classified image to compare the semi-automated image results. In our study, the objects that were delineated and classified by experts from the U.S. Fish and Wildlife Service, were used as training data for our random forest classifier and as ground truth for the discrepancy measure. The discrepancy measure used the percentage of right segmented pixels (PR) in the whole image. To calculate PR, we converted the classified segmented and ground truth polygons to raster and measured the ratio of incorrect pixels to total amount of pixels which was converted to a percentage.
Additionally, landscape metrics were calculated using FRAGSTATS 48 , an open source program commonly used for calculating landscape metrics. FRAGSTATS computed these metrics from thematic raster maps that represent the land cover types of interest. These thematic classes, used for analysis, were the classified objects at both the super and sub-object level. Since we are not attempting to compare the segmentation results for any specific class or area, we calculated metrics on a landscape level. Landscape metrics will represent the segmentation patterns for the entire study area.
FRAGSTATS can calculate various metrics representing different aspects of the landscape. The metrics for analysis attempts to understand object geometry. The metrics calculated, for these analyses, were the average and standard deviation for the area (AREA), the fractal dimension index (FRAC), and the perimeter area ratio (PARA). The number of patches (NP) was also included in each result. To take a more landscape centric approach, the area weighted mean was chosen over a simple average.

Results
The following will present the empirical results with comparisons between the OOB error, the random forest classification, and segmentation discrepancy.
Segmentation results. Like the classification results, Table 1 shows the PR segmentation results of the super objects, with and without the Gabor feature, with all features included (spectral, CHM, NDVI). Table 2 shows the PR segmentation results for the sub-objects. The inclusion of the Gabor feature for the super objects made very little difference (0.03 percentage points) in the segmentation results according to the PR metric. In the sub-object case, the inclusion of the Gabor feature greatly decreased segmentation performance.
Classification results. Table 1 exhibits the OOB classification results from the random forest classifier of the super objects, with and without the Gabor feature, with all features included (spectral, CHM, NDVI). These results were used as the hierarchical features for the sub-objects. The sub-object's OOB classification results can  Table 1. Out-of-bag error (super). Out-of-bag error results from the random forest classifier and percentage of right segmented pixels (PR) in the whole image for the super-objects.

Feature
Out-of-bag error (%) Pr % www.nature.com/scientificreports/ be viewed in Table 2. As shown, the only instance when the inclusion of the Gabor feature improved classification results, for both the super and sub-objects, was when the hierarchical feature was included. The hierarchal feature improved accuracy in both instances but was further improved when combined with the Gabor feature, resulting with the best performance of the four datasets.
Landscape metrics. Results for the landscape metric analyses can be seen in Table 3. The differences between sub and super objects were as expected, with super objects having fewer patches than the sub-objects and area weighted mean being larger for the super objects except for sub-objects created with the Gabor features. The sub-objects created with the Gabor features had area averages that were greater than any other results, including the human segmented results, which is due to large continuous patches of wet forests. Additionally, these instances had the largest standard deviations in patch size. This indicates that there was a broad mixture of large and small patches. When observing the automated results for the super-objects, it appears that not including the Gabor features provides similar results to the human segmented objects. The only instance where the inclusion of Gabor features makes the segmentation similar to the human segmented objects is the number of patches, which can also be a measure of landscape fragmentation. Average area and both measures of edge or shape complexity (FRAC and PARA) both show that the exclusion of Gabor features cause segments to be more similar to the human segmented objects.
Similar observations can be made for the sub-objects. In most cases, instances where the Gabor feature was excluded resulted in similar landscape metrics to the human segmented objects. Furthermore, the inclusion of Gabor features had a higher number of similar patches as the human segmented instances for the sub-objects. The exclusion of Gabor features severely over fragment the landscape whereas the inclusion of Gabor features slightly under-fragmented it. It was observed that when the Gabor features were included within the segmentation, the areas classified as wet forest (a significant proportion of land cover in the study area) were delineated into large patches. When the Gabor features was not included, the wet forest was over fragmented which contributed to the large number of patches. A characteristic of the wet forest class in the study area is that they existed as large continuous chains and perhaps the first principal component did a good job at capturing the textural attribute of this class. In future studies, more principle components should be included on the chance that they can capture the textural attributes of the other classes better than the first principal component alone.

Discussion
Our study yields valuable information to the inclusion of hierarchically organized vectors; it provides accuracy estimates for classified objects with and without the inclusion of hierarchal attributes. Of the identified papers that used hierarchical segmentation, few included hierarchical attributes in their object classification 18,49,50 and only one included the accuracy estimates with and without the inclusion of hierarchal attributes 15 . Other studies used one segmentation scale to guide the segmentation results of the next finer or broader scale 16,17,19,51 . Antunes et al. 15 report agreed with our results in that the inclusion of hierarchical attributes increased classification Table 2. Out-of-bag (Sub). Our-of-bag error results from the random forest classifier and percentage of right segmented pixels (PR) in the whole image for the sub-objects.

Feature
Out-of-bag error Pr %   50 , Laiberte et al. 18 , and Laiberte et al. 49 supported our findings by stating that including hierarchical attributes visibly improved their results 50 .
Observing both variable indices, the Gini and mean decrease in accuracy index, each instance the hierarchical features were included as an attribute for the sub-objects, the hierarchical features were indicated as providing more predictive power relative to the other included features. This coincides with the increase in accuracy when these features were included and, therefore, does not provide sufficient evidence that hierarchical features introduce noise into the dataset but rather provide valuable predictive information. This is contrary to the results observed when Gabor features were included, in the random forest, for predicting super-objects. The mean decrease in accuracy index indicated a high predictive power for the Gabor features, and an increased OOB error.
Gabor features did not provide additional information for increasing classification accuracies or improve segmentation results according to the sub-objects' PR metric. The super-objects' PR metric decreased insignificantly when Gabor features were included in the segmentation step. The PR metrics for the sub-objects display a significant decrease in segmentation accuracy when Gabor features are included. Based on these results, Gabor features should not be included in the segmentation step of the GEOBIA process. According to these results, Gabor features should not be included as part of the training and classification unless hierarchical features are included.
It is unclear to the authors why the inclusion of Gabor features improved the classification results only in the instance when hierarchical features are included. According to the Gini index and mean decrease in accuracy, the random forest algorithm utilized the Gabor features slightly more when hierarchical features were included than when they were not suggesting that the Gabor features improved the classification. When Gabor features were included for the other sub-object datasets that did not include hierarchical features, these same indices showed that Gabor features were utilized very little by the random forest algorithm (in addition to decreasing their accuracy). Similar effects were found when Gabor features were included within a patch based land cover analysis 33 . It is suggested that further research is conducted to observe why Gabor features have the opposite effect on classification when hierarchical features are included.
Limitations of this analysis of the segmentation results are as follows. To begin, the metrics used to evaluate segmentation results are still being developed. Most segmentation evaluations within geography use discrepancy measures, based off a classified ground truth image 37,46,47,52,53 . These measures depend on the correct classification of the objects and heavily relies on the accuracy of the classifier rather than measuring the quality of boundaries created by an algorithm. One proposed method is to measure the distance between the boundaries of the ground truth images and those generated by the proposed algorithm.
Another limitation is that most empirical methods for segmentation evaluation are based on ground truth images that are generated by human subjects, who subjectively delineate image object boundaries. Human interpretation can be inconsistent, biased, and differ from person to person despite any expert status. The PR metric is also influenced by the correct classification of the objects. One reason object-based analysis is widely used is that it produces consistent, predictable, and reproducible results. Rather than relying on correctly classified pixels for segmentation evaluation, object-based image analysis should begin using distance to reference boundaries 54 . Additionally, most users conducting an object-based image analysis, to aid in decision-making process, do a considerable amount of post-processing (i.e. dissolving small segments and holes, smoothing, merging) which could cause the PR metric, and other metrics to observe segmentation results, to change.
Natural multipart landscapes are complicated systems that have spatially interconnected parts that influence one another across space and scales. Not utilizing this information (i.e. pixel-based classification) or ignoring to identify spatial or hierarchical relationships does not fully exploit the information that can be obtained from delineation and classification of objects. Our results provided further support that including hierarchical structure to objects offers contextual information that can increase classification accuracy beyond what is provided by texture and spectral alone.