Introduction

It is well known that the development of tree canopy affecting both quality and yield of peaches1. Especially in Japan, to achieve high economic production, manipulation and management of the tree canopy are essential. As the increasing price of agrochemicals and labour, there is a need for efficient precision management. Precision farming applies the appropriate timing, amount, and location of fertilizer and pesticides to crop management2. The preliminary step of precision farming is acquiring as much growth data from the crop as possible3, which depends on accurately describing the morphological and structural characteristics of crops. In the pomological aspect, relevant morphological characteristics include crown width, height, area, and volume. Among these, crown width is important for precision spraying4,5,6 and machine harvesting7, while crown projection area (CPA) is important for determining tree growth during the growing season8.

Several methods are used to measure the crown characteristics. In the field, crown width can be measured by tape or laser rangefinder9, and CPA can be approximated as the area of a polygon with points on the crown’s drip-line or estimated from a circle10 or ellipse based on the crown spread. Such subjective measurements on a crude scale may introduce large errors11, and the precise field measurement of CPA is time consuming12. Alternatively, a terrestrial laser scanner can map three-dimensional (3D) plant structure in detail13, but the device is costly and accurate crown reconstruction requires high-density laser scan data14 which means it consumes more time in data acquisition and processing. The development of computer vision technology has made it possible to build a 3D model of a tree based on the captured photos by camera. This method is not only objective and repeatable compared with field manual measurement, but also more cost effective than using a laser scanner.

According to the platform, cameras mainly include shelf-fixed cameras8, handheld cameras15, and cameras mounted on unmanned aerial vehicles (UAVs)7,9,14,16. Among them, UAVs are rising in popularity for applications in plantations and orchards for its flexibility and relatively larger cover area in data collection17. High-resolution images collected from UAVs have been used to determine the height and crown size of olive trees14,16, blueberry bushes7, Mediterranean riparian forest18, and Norway spruce and Scots pine9.

In the characterization of trees from UAV images, one of the most important procedures is the extraction of individual trees. A commonly used method for crown segmentation is watershed transformation, which treats the height of trees as terrain and segments individual ‘drainage basins’ by identifying local maxima and nearest minima. Treetops are usually treated as local maxima, but this will cause over-segmentation in structurally complex trees: non-conical shape of crown causes multiple local maxima identified within a single broadleaf crown9. Several methods can solve the over-segmentation requiring explicit prior knowledge of the image structure19, e.g. multi-scale filtering20, marker-controlled watershed21, region merging watershed22, and watershed method using prior shape23. Therefore, finding the simple and effective way to get the location and rough extent of the crown before using watershed segmentation is critical. The commonly used morphological erosion method is sensitive to neighbourhood size and shape, which may eliminate the small trees. Popescu et al.24 proposed using small-footprint lidar data to first locate individual trees but assumes that crown diameter is in relation with crown height, which is not suitable for height-controlled fruit trees. In general, correctly isolating individual crowns remains difficult, especially for broadleaf trees with the non-conical shape.

For peach trees trained in an open-centre shape, it is difficult to isolate individual crowns. To the best of our knowledge, the use of UAV imagery of peach trees for crown characterization has not been explored. The primary purpose of this research was to develop an image processing procedure for crown characterization which takes advantage of the crown change of deciduous trees with seasons to extract individual trees. The specific objectives of this study were to (1) propose a new UAV image analysis method for the accurate and efficient characterization of crown width and crown project area of peach trees, (2) evaluate its performance, and (3) use it to analyse the growth of peach trees in 2017 in Okayama, Japan.

Materials and methods

Study site description

We tested our method on two plots in a peach orchard (~1.65 ha) in Okayama, Japan. Plot 1 includes twelve 11-year-old peach trees (Shimizu Hakutou) and plot 2 includes 32 peach trees with six different cultivars and nearly 11-years-old (Shimizu Hakutou, Hakuhou, Okayama Yume Hakutou, Hakurei, Ougontou, Sakigake) (Fig. 1). All the peach trees were trained to a modified open-centre shape with two predominant branches oriented approximately perpendicular to the row.

Fig. 1
figure 1

The cultivars and ages by 2018 (in the bracket) of peach trees in the orchard in Okayama, Japan

Full size image

Data acquisition

We collected three independent datasets: UAV images, manual delineated images, and related field measurement that include crown width and projection area.

Collection and pre-processing of UAV images

Images for peach trees in plot 1 were collected on five occasions in the orchard from September 2016 to September 2017. Images for peach trees in plot 2 were collected in December 2017 and July 2018 to increase sample size. The trees were photographed with a digital camera mounted on a DJI inspire 1 and inspire 2 UAV (Shenzhen Dajiang Baiwang Technology Co., Ltd, China) with wind speed ≤level 2 according to the Beaufort wind scale. Flight was controlled by Litchi software (VC Technology Ltd, UK), which directed the UAV flying along a double-grid image acquisition plan (Fig. 2) at a height of 30 m and a speed of 2.5 m/s with the camera looking downwards. Overlap of photos to the front and side was >80%. It took about 6 min to cover plot 1 (~0.2 ha) in one UAV flight by DJI inspire 1 and 17 min to cover the whole orchard (~1.65 ha) in one UAV flight by DJI inspire 2. See detailed information about the digital camera and the flight in Table S1. In order to improve the geolocation accuracy, the georeferencing of the point cloud was done using a combination of direct georeferencing and ground control points (GCPs, red points in Fig. 2). The coordinates of the GCPs were measured by Hemisphere RTK differential GNSS devices (Hemisphere GNSS, USA) in plot 1 and Propeller AeroPoints with Propeller correction network (Propeller Aerobotics Pty Ltd, Australia) in the whole orchard. The mean RMSE of the seven flights was 0.24 m.

Fig. 2: The UAV flight pattern (yellow lines) in plot 1 as an example.
figure 2

It followed a grid at a height of 30 m and a speed of 2.5 m/s, with >80% image overlap. The numbers label the flight order of way point and the red points mark the ground control points

Full size image

The raw images collected by UAV were used to generate a 3D point cloud, from which a 2D red-green-blue (RGB) orthomosaic and the digital surface model (DSM) were generated in Pix4Dmapper Pro v. 4.0.25 software (Pix4D Inc., Switzerland) by the Structure from Motion algorithm25. Full image scale was set for precisely extracting the key points and the relative camera positions were also taken into account to discard geometrically unrealistic matches. During the generation of DSM, noise filtering was used to correct the altitude of these points with the median altitude of the neighbouring points. The RGB orthomosaic and DSM image from the same point cloud model were exported at the same ground sampling distance. The ground resolution of all RGB and the DSM images was <1 cm/pixel.

Manual delineation of trees

The manual delineation dataset was obtained by manually drawing outlines of the 12 crowns from the RGB orthomosaic of September 2016 in Adobe Photoshop software (Adobe Systems Incorporated, USA) and then calculating the two orthogonal crown widths and the CPA based on the drawn crowns and ground resolution in Matlab v. R2017b software (MathWorks Inc., USA).

Field measurement

We did the field measurement of peach trees two times. One was in late August 2016 including 12 trees in plot 1 and the other was in early July 2018 including 32 trees in plot 2 as a supplementary experiment. In both the investigations, we measured the crown widths parallel (W1) and perpendicular (W2) to the row of the trees with a millimetre measuring tapeline. Based on the preliminary experiment, the CPA of each tree was estimated by a local empirical equation:

$$CPA_{\mathrm{f}} = 0.65 \times W_{1{\mathrm{f}}} \times W_{2{\mathrm{f}}}$$
(1)

where W1f and W2f represent the crown widths parallel and perpendicular to the row of the trees, respectively.

Characterization of peach trees

The geometric characteristics of crowns were calculated from the DSM in Matlab v. R2017b software (MathWorks Inc., USA). The crown geometry is derived using two kinds of DSM (bare-branch DSM and foliated DSM) by image analysis techniques in the following five steps (source codes and sample data are available at our surpport page: https://github.com/UTokyo-FieldPhenomics-Lab/Characterization-of-peach-tree-crown):

  1. (1)

    For both kinds, the crowns were first identified from the DSM at an adaptive threshold26, and the region of the target field was extracted with a fixed polygon mask (red lines, Fig. 3a, g). Then the DSM of the target peach trees was extracted (Fig. 3h).

    Fig. 3: Workflow of the proposed method.
    figure 3

    a Original DSM of trees in winter (red line is the boundary of the polygon mask). b Crown scaffolding extracted by adaptive threshold and morphology operation ‘Closing’. c Convex polygons (red line is the boundary). d Convex masks. e Centroids of the convex polygon (red points). f Marker used in watershed segmentation, which is the intersection of (d) and (h). g Original DSM of trees in growing season. h Extracted DSM of crowns by adaptive threshold algorithm. i Segmented trees by marker-controlled watershed algorithm. j Individual trees after merging some small parts (white boxes are bounding boxes)

    Full size image
  2. (2)

    For the bare-branch DSM (generated from photos taken on February 2017; Fig. 3a), the mathematical morphology operation ‘Closing’ (structural element: diamond, radius: 50 pixels (February 2017) and 80 pixels (December 2017)) was used to merge the branches of each tree as the scaffolding (Fig. 3b). Then the convex boundary of each tree was delimited (red lines, Fig. 3c).

  3. (3)

    For the foliated DSM (generated from photos taken in the growing season; Fig. 3g), using the intersections of the convex mask (Fig. 3d) and the extracted DSM (Fig. 3h) as a marker (Fig. 3f), watershed transformation27 was conducted on the extracted DSM, and then individual crowns were extracted (Fig. 3i).

  4. (4)

    Small isolated parts of images (enlarged view in Fig. 3i) were merged into the closest crown (determined from the distances to the centroids of each convex) (Fig. 3j) and then labelled with a unique ID.

  5. (5)

    The CPA of each tree was calculated as the product of the number of pixels in each crown and the square of ground resolution (m2/pixel). The two orthogonal crown widths (W1 and W2) corresponded to the width and length of the bounding box parallel to the row (Fig. 3j) and were converted to metres (m) by multiplying with the ground resolution (m/pixel).

Evaluation and comparison of the accuracy of the method

To comprehensively analyse the error of the proposed method, we evaluated the accuracy of the estimated crown widths and projection area against two referential datasets. The manual delineation dataset was used from the perspective of image analysis and field measurement dataset was used from the perspective of horticulture research. The accuracy of the proposed method was evaluated by calculating the square of the correlation coefficient (R2) and the root-mean-square errors (RMSE) and relative (R)-RMSE against the referential data in Matlab R2017b as:

$$RMSE = \sqrt {\frac{{\mathop {\sum}\nolimits_{i = 1}^n {Err_i^2} }}{n}} = \sqrt {\frac{{\mathop {\sum}\nolimits_{i = 1}^n {(v_i - v_{ri})^2} }}{n}}$$
(2)
$$R{\mathrm{ - }}RMSE = \frac{{RMSE}}{{\frac{{\mathop {\sum}\nolimits_{i = 1}^n {v_{ri}} }}{n}}}$$
(3)

where Erri is a measurement error of the ith tree, vi is a measurement value of the ith tree, vri is a referential value of the ith tree, and n is the number of trees.

Calculation of the growth rate

CPA can be used to determine tree growth during the growing season8, and growth rate was proved to be helpful to understand the vigour of a tree28. The growth rate (GR) of CPA was calculated as Eq. (4), and the relative growth rate (RGR) was calculated as Eq. (5):

$$GR = \frac{{CPA_t - CPA_{t - {\mathrm{\Delta }}t}}}{{{\mathrm{\Delta }}t}}$$
(4)
$$RGR = \frac{{CPA_{\mathrm{t}} - CPA_{t - {\mathrm{\Delta }}t}}}{{CPA_{t - {\mathrm{\Delta }}t} \times {\mathrm{\Delta }}t}}$$
(5)

where Δt is the interval of days.

Results

Peach trees extracted by the proposed method

By using the proposed method, peach trees in plots 1 and 2 were extracted and characterized. In plot 1, almost all the trees were segmented well (Fig. 4a) but the crown of tree Nos. 2, 7, and 8 lost a little bit. The loss of crown was caused by the lower sensitivity when using adaptive threshold algorithm as some parts of the crown was not high enough (Fig. 4b). In plot 2, most crowns were separated well except that of tree Nos. 44, 37, 38, 25, 15, and 16 (Fig. 5a). It was due to the crowns of tree Nos. 25, 15, and 16 were too close that the overlap of branches pushed up the height on the boundary (Fig. 5b), as well as tree Nos. 44, 37, and 38.

Fig. 4: Individual crowns (September 2016) extracted by the proposed method.
figure 4

a and b shows the outlines of individual crowns in RGB orthomosaic and DSM

Full size image
Fig. 5: Individual crowns (July 2018) extracted by the proposed method.
figure 5

a and b shows the outlines of individual crowns in RGB orthomosaic and DSM

Full size image

Evaluation of the accuracy of the proposed method with manual delineation

To evaluate the proposed method from the perspective of image analysis, taking crowns delineated (‘d’) manually from the RGB orthomosaic as referential data, the accuracy of the crown widths and CPA determined by the proposed method (‘p’) from a DSM was evaluated according to the calculated R2, RMSE, and R-RMSE.

The accuracy of W1p-d and W2p-d was high as shown in Table 1. The R2 of CPAp-d was also high, but the R-RMSE of CPAp-d was relatively large compared with W1p-d and W2p-d, and all CPAp were larger than the reference area (Fig. 6c). It may be caused by the coarser resolution of DSM compared with the RGB orthomosaic, by comparing the manual delineation outline in Fig. 7a, b. From a viewpoint of image analysis, the accuracy of the proposed method was acceptable.

Table 1 The square of the correlation coefficient (R2), root-mean-square errors (RMSE), and relative (R)-RMSE of the proposed method with manual delineation measurement as reference
Full size table
Fig. 6: Evaluation of the accuracy of crown width and crown projection area (CPA) against manual delineation.
figure 6

a, b, and c shows the accuracy of W1, W2 and CPA, respectively

Full size image
Fig. 7: Comparison of individual crowns delineated by the proposed method (blue) and manually (red).
figure 7

a and b shows the outlines of individual trees photoed on September 2016 in RGB orthomosaic and digital surface model

Full size image

Evaluation of the accuracy of the proposed method with field measurement

Considering crowns may overlap, from the viewpoint of horticulture research, the accuracy of the proposed method was also evaluated by the field (‘f’) measurement. From Fig. 8a, b, it is shown most of the crown widths determined by the proposed method were close to those measured in the field, except some crowns not well segmented as analysed in the ‘Peach trees extracted by the proposed method’ section. The R2 of W2p-f was lower than W1p-f as shown in Table 1, which was caused by the large error of those crowns, e.g. tree Nos. 25, 15, and 16 (Fig. 5).

Fig. 8: Evaluation of the accuracy of crown width and crown projection area (CPA) against field measurement.
figure 8

a, b, and c shows the accuracy of W1, W2 and CPA, respectively. The red points represent trees measured on September 2016 and the blue points represent trees measured on July 2018

Full size image

Compared with field estimation, the CPAp tends to be underestimation with the CPAf increasing. The R2 of CPAp-f was also high, but the R-RMSE of CPAp-f was relatively large as shown in Table 2. The error may come from three aspects, (1) the propagation of error in crown width determination, (2) the underestimation of CPA due to the overlap of the crown on the boundary, (3) the error in estimating the area by using the empirical equation, Eq. (1). Taking the crowns in plot 1 as an example, Fig. 9 demonstrated the discrepancy in area calculated by Eq. (1) and the proposed method. Tree No. 5 has the closest value with field estimation with a difference of 1.31 m2, while that for tree No. 10 was 5.15 m2. It seems if the shape of the crown was non-ellipse-like, the error would be larger than ellipse-like.

Table 2 The square of the correlation coefficient (R2), root-mean-square errors (RMSE), and relative (R)-RMSE of the proposed method with field measurement as reference
Full size table
Fig. 9: Comparison of the field estimated area and the determined area by the proposed method.
figure 9

The ellipse (in red) was drawn with the same area of field estimation, by taking the W1 and W2 as the major and minor axis, which shrank with the same scale

Full size image

Application in growth analysis of peach tree crowns

CPA can be used to determine tree growth during the growing season8, and vegetative growth rate is closely related with vigour28. We calculated the CPA of the 12 trees in plot 1 from May to September 2017.

During the growing season, CPA increased consistently (Fig. 10a) by an average of 6.02 m2. From May to July, it increased at a mean GR of 8.14 × 10−2 m2 day−1. From July to September, it increased at a mean GR of 3.40 × 10−2 m2 day−1. The GRs of CPA from May to July were larger than that from July to September for all the trees (Fig. 10a). Thus, the trees grew faster from May to July than from July to September.

Fig. 10: Changes in crown projection area (CPA) and its relative growth rate of  trees in plot 1 during the growing season in 2017.
figure 10

a shows the changes in CPA and b shows the changes in relative growth rate of CPA

Full size image

The RGRs of CPA differed among trees (Fig. 10b). From May to July, the CPA of tree No. 1 increased most (4.43 × 10−3 m−2 day−1) and that of tree No. 5 increased least (0.93 × 10−3 m−2 day−1). From July to September, the CPA of tree No. 3 increased most (1.35 × 10−3 m−2 day−1) and tree No. 6 almost not increased. Overall, tree No. 5 had the lowest mean RGR of CPA (0.69 × 10−3 m−2 day−1).

Discussion

Accuracy comparison with other researches

We demonstrated a UAV image analysis method for accurate and efficient determination of crown width and CPA of peach trees and evaluated its accuracy against field measurements. In comparison with other researches as shown in Table 3, the proposed method had relatively high accuracy for both crown width and CPA (Table 3). However, the high ground resolution (<1 cm/pixel) may more or less contribute to the high accuracy.

Table 3 The accuracy comparison with other researches taking field measurement as reference data
Full size table

Comparison with field measurement in work efficiency

The UAV imaging method and the data processing workflow that we developed are applicable to practical orchard management. It took about 6 min to photograph 12 trees, while it took about 30 min to measure the crown widths by using tapeline in the field. Using UAV for data collection largely saved time consumed in the field. In data processing, it took about 3 h for DSM generation using ~150 photos followed by about 50 s to determine the canopy characters on a PC with 64 GB of RAM, Intel i7-5930K CPU and 64-bit Windows 10 operating system. All the processing steps could be done automatically, and the measurements are repeatable, which is considered more reliable and efficient than traditional field measurement.

Error analysis

From the viewpoint of image analysis, according to accuracy evaluation with manual delineation, the error of CPA > W2 > W1. The error of W2 was larger than W1 mainly due to crown segmentation error. There were more touching branches at the dominant branch direction, which not only increased the segmentation task by using the watershed algorithm, but also may push up the altitude by overlapping (Fig. 5b). The error of CPA mainly due to crown segmentation error by using adaptive threshold and watershed algorithm, and the coarser resolution of DSM than RGB orthomosaic. The less sensitivity of some lower branch made them failed to be extracted. The representation of the convex hull in winter to the actual shape of the crown may also influence the segmentation while using watershed algorithm. In addition, the systematic overestimation of CPA (Fig. 6c) was likely caused by the noise filter during the DSM generation process: the same crown looks bloated in the DSM image than in the RGB orthomosaic. During the generation of the DSM, the points on the border of ground and crown may be interpreted as noisy and erroneous points by Pix4Dmapper because of huge change in altitude, and their altitude were corrected with the median altitude of neighbouring points. Thus made the height of points inside the crown become smooth. This kind of smooth effect made the crown looks bloated.

From the viewpoint of horticulture research, the overlapping of crowns may lead underestimation of CPA (Fig. 9). In addition, field measurement of crown widths, and more so estimation of crown project area, has inherent error14. In the measurement of crown width, it may bring error in confirming the farthermost point of the crown, especially for at the parallel to row direction, as people not easy to stand near the farthermost points at the same time, e.g. tree No. 11 (Fig. 9). Because the crown project area of peach tree is estimated by a local empirical equation, which calculated the ellipse area with coefficient adjustment, the shape of the crown being non-ellipse-like may get larger estimation error than ellipse-like.

Limitations of the proposed method

Though the proposed method was shown to be accurate and efficient, there are some limitations. One limitation is it only measured the crown on the top view, which means error will increase if there are too much overlaps between each tree. Another limitation of using the proposed method is that the scaffold branches themselves must be separated. Although this is applicable to most fruit trees as the requirement of orchard management, it is difficult to separate trees in the forest of high density. The method was tested in only one flat orchard of the same training strategy. More orchards with different training strategies need to be tested in the future.

Prospective

Growers pay particular attention to vegetative growth, which competes with fruit growth and influences potential yield28,29. And by setting varied fertilizer management zones using spatio-temporal analysis of field characteristic and previous yields maps, it can achieve high environmental and economic benefits30. The proposed method allowed us to calculate the GR and RGR of each tree (‘Evaluation of the accuracy of the proposed method with field measurement’ section), which could help not only make a reasonable pruning or growth regulation plan, but also a varied fertilizer management.

Conclusions

This study confirms morphological traits, such as crown size and area, can be extracted from height information of peach trees generated by using consumer-grade UAV imagery and computer vision techniques. The results revealed that the crown size and area derived by our method were close to the manually delineation and field measured, and it could replace field measurement to achieve significant labour saving. The change of CPA showed the growth rate of peach tree was larger from May to July than from July to September and have a large variety on different trees. Furthermore, the findings of this study lead us to believe that this technology would allow for the convenient surveillance of orchard trees to observe growing trends and could therefore provide guidance in fruit tree management.