Goosegrass Detection in Strawberry and Tomato Using a Convolutional Neural Network

Goosegrass is a problematic weed species in Florida vegetable plasticulture production. To reduce costs associated with goosegrass control, a post-emergence precision applicator is under development for use atop the planting beds. To facilitate in situ goosegrass detection and spraying, tiny- You Only Look Once 3 (YOLOv3-tiny) was evaluated as a potential detector. Two annotation techniques were evaluated: (1) annotation of the entire plant (EP) and (2) annotation of partial sections of the leaf blade (LB). For goosegrass detection in strawberry, the F-score was 0.75 and 0.85 for the EP and LB derived networks, respectively. For goosegrass detection in tomato, the F-score was 0.56 and 0.65 for the EP and LB derived networks, respectively. The LB derived networks increased recall at the cost of precision, compared to the EP derived networks. The LB annotation method demonstrated superior results within the context of production and precision spraying, ensuring more targets were sprayed with some over-spraying on false targets. The developed network provides online, real-time, and in situ detection capability for weed management field applications such as precision spraying and autonomous scouts.

In Florida, many broadleaf horticultural crops are produced using a plasticulture system. This system included raised beds covered in plastic mulch with drip irrigation installed to provide nutrients and moisture. Weeds within this system primarily occur within the planting holes or between the rows, except for purple nutsedge (Cyperus rotundus L.) and yellow nutsedge (Cyperus esculentus L.) which penetrate and emerge through the plastic mulch.
Within vegetable horticulture, the prevalent post-emergence weed management options for goosegrass control include hand weeding and herbicides. For pre-plant burn down and within row middles, broad-spectrum herbicides such as paraquat and glyphosate are widely employed. Consequently, both goosegrass and American black nightshade (Solanum americanum Mill.) have developed paraquat resistance 8,9 and ragweed parthenium (Parthenium hysterophorous L.) developed glyphosate resistance 10 . For weed control atop the bed during the cropping cycle, WSSA Group 1 herbicides are the most common post-emergence chemical control option. Group 1 herbicides are becoming increasingly utilized within herbicide mixtures for grass control in row middles depending on weed pressures and resistance issues faced.
Implementing precision technology into spraying equipment is a viable option to reduce production costs associated with weed management. Goosegrass and other grass species are excellent targets for precision technology to apply Group 1 herbicides to a variety of broadleaf crops. A prototype precision sprayer was developed to simultaneously detect and spray weeds in plasticulture production within Florida. Briefly, the system was a modified plot sprayer with a digital camera sensor, a controller linked with artificial intelligence for detection, and nozzles controlled by solenoids. The desirable detector for this system is a convolutional neural network.
Machine vision-based weed detection is typically conducted using either multispectral/hyperspectral or RGB imagery, the latter being more desirable for economic costs and practical adoption for producers 11 . Recent technological advances in graphical processing units permit training and employing deep learning convolutional neural networks as detectors 12 . Deep convolutional neural network frameworks have been reviewed elsewhere 12,13 . Briefly, neural networks take inspiration from the visual cortex, containing layers for feature extraction, convolution, pooling, activation functions, and class labeling 14 . The system relies on pattern recognition via filters within the convolutional layers for detection and classification 15 . Convolutional neural networks for weed detection have been employed in several crops including turfgrass 16,17 , wheat 18 , and strawberry 19 . For horticultural plasticulture row middles, a convolutional neural network has been developed to detect grasses among broadleaves and sedges 20 . Within broader agriculture, deep neural network applications include strawberry yield prediction 21 , sweet pepper (Capsicum annum L.), and cantaloupe (Cucumis melo var. cantalupo Ser.) fruit detection 22 , and detection of tomato pests and diseases 23 .
With the widespread registration of Group 1 herbicides in broadleaf crops and the widespread distribution of goosegrass, the successful development of a detection network would have far-reaching implications for conventional horticulture. Development of a multi-crop, within-crop grass detection network has challenges including training image availability, ease of image collection due to the patchy nature of weeds, and the diverse background of several crops as the negative space. Additionally, the goosegrass within-crop growth habit, as well as the general habit of grassy weeds causes issues for bounding box-based network training. Goosegrass has a tufted plant habit with stems that are erect to spreading and up to 8.5 m tall, and leaves which are 5 cm to 35 cm long and 3 mm to 5 mm wide 24 . For strawberry production, goosegrass leaves have been observed to either penetrate through the crop canopy, growing prostrate along with the plastic, or grow in planting holes where strawberry plants have died. For tomato production, goosegrass plants typically grow at the base of the tomato plants, which are vertically staked for fresh-market production. The study objectives were to (1) develop a network with utilities in multiple broadleaf crops, starting with strawberry and tomato plasticulture, (2) evaluate the use of small label annotation boxes along the leaf-blade length for goosegrass detection compared to boxes encompassing the entire plant habit, and (3) evaluate a piecemeal oversampling technique.

Results
For strawberry production, the entire plant annotation method (EP) (precision = 0.93; recall = 0.88; F-score = 0.90; accuracy = 0.82) resulted in an overall increased YOLOv3-tiny training fit compared to the leaf-blade annotation method (LB) (precision = 0.39; recall = 0.55; F-score = 0.46; accuracy = 0.30) ( Table 1). Convergence time, in iterations, declined rapidly for EP compared to LB (data not shown). This was expected since EP resulted in fewer bounding boxes and provided larger bounding boxes with a static location. Labeling of goosegrass leaf blades with narrow squares resulted in "ground truth fluidity" with resultant increased training time and reduced fit.
While the EP network appeared more successful in training, the network provided inadequate testing results. For goosegrass detection within strawberries, the LB (precision = 0.87; recall = 0.84; F-score = 0.85; accuracy = 0.74) outperformed the EP (precision = 0.93; recall = 0.62; F-score = 0.75; accuracy = 0.60) in terms of overall F-score and accuracy ( Table 2). The EP method demonstrated high precision but tended to miss targets ( Fig. 1). There was no impact of the annotation method on iteration time (Table 3). Compared to the EP, the LB network increased recall substantially at the expense of precision but resulted in the highest F-score.
For goosegrass detection in tomato, the EP (precision = 0.77; recall = 0.43; F-score = 0.56; accuracy = 0.38) had higher precision but struggled at detecting plants ( Table 2, Fig. 2). Comparatively, the LB (precision = 0.59; recall = 0.74; F-score = 0.65; accuracy = 0.49) had reduced precision but had an increased recall. The LB derived network resulted in the highest overall F-score and accuracy for goosegrass detection in tomato.

Discussion
Detection in strawberry production demonstrated suitable identification of goosegrass. For images taken within tomato production, success was limited ( Table 2). This was most likely a consequence of available goosegrass training images within strawberry production but not for tomatoes. While attempts were made to match image acquisition angles and growth stages for both goosegrass and tomato growing in isolation, not having additional training images of the desired target and background together was likely detrimental. This could be due to the degree of actual overlap between the crop and weed in competition, altered growth habit by the weed in competition, or natural variability in the tomato growth habit inducing a stoichiometric effect that requires additional training images to overcome. www.nature.com/scientificreports www.nature.com/scientificreports/ For detection in both tomato and strawberry, the LB outperformed the EP in terms of recall, F-score, and accuracy. The EP networks had consistently higher precision but lower recall. This was likely a consequence of selecting the entire plant habit, increasing the variability between targets, and reducing the number of potential targets for training. Such precision and recall neural network trade-offs have been noted elsewhere, including polyp detection 25 . For precision spraying, the EP network would miss many plants but would typically spray goosegrass only. Comparatively, the LB network would spray goosegrass more regularly with some degree of over-spraying upon undesirable targets. For weed detection in occluded winter wheat using a convolutional neural network based on DetectNet achieved 87% precision and 46% recall 18 . Comparatively, using an object detection convolutional neural network based on You Only Look Once to detect weeds in winter wheat images resulted in 76% precision and 60% recall 26 . Detection of Carolina geranium in strawberry using DetectNet and leaf-level annotation resulted  Table 2. Pooled relevant binary classification categories and neural network accuracy measures for goosegrass (Eleusine indica) detection in tomato (Solanum lycopersicum) and strawberry (Fragaria × ananassa) using two annotation methods on digital photography acquired in Central Florida, USA, in 2018 and 2019 a . a The neural network was the tiny version of the state-of-the-art object detection convolutional neural network You Only Look Once Version 3 (Redmon and Farhadi 2018). b EP = Entire plant annotation method. This refers to using a single, large square box to identify goosegrass within digital images. c LB = Leaf-blade annotation method. This refers to using multiple, small square boxes placed along leaf blades and inflorescence to identify goosegrass within digital images. www.nature.com/scientificreports www.nature.com/scientificreports/ in 99% precision and 78% recall 19 . Current results for goosegrass detection in strawberry obtained a relatively similar overall accuracy compared to similar studies using convolutional neural networks alone, but detection in tomatoes may require further sampling.

Entire plant annotation
Results indicate the potential for a unified network for use across multiple crops. Additional options for precision spraying multi-crop networks include Group 2 herbicides in vegetable plasticulture, group 4 within cereals, and groups 9 and 10 within associated genetically modified crops. While the piecewise image methodology results for tomatoes were limited, network desensitization for additional crops does provide some benefit. Existing networks for goosegrass can be expanded to additional crops and the number of necessary training images should be reduced.
Several kinds of grass infest vegetable fields. Since the network did not classify tropical signalgrass [Urochloa distachya (L.) T.Q. Nguyen] as goosegrass (data not shown), additional classes are likely necessary or grouping  Table 3. Impact of annotation style on testing iteration time for goosegrass (Eleusine indica) detection in strawberry (Fragaria × ananassa) and tomato (Solanum lycopersicum) production using a convolutional neural network developed at Balm, FL, USA in 2018 a . a The neural network was the tiny version of the state-of-the-art object detection convolutional neural network You Only Look Once Version 3 (Redmon and Farhadi 2018). b LB = Leaf-blade annotation method. This refers to using multiple, small square boxes placed along leaf blades and inflorescence to identify goosegrass within digital images. c Entire plant refers to the annotation method where a single, large square box to was used to identify goosegrass within digital images. www.nature.com/scientificreports www.nature.com/scientificreports/ multiple grass species into a single category 20 . A similar network (YOLOv3) was trained to detect broadleaf species that were not previously part of the training dataset 20 , so this option may be feasible but requires further study. If such is desirable, care should be taken to avoid class imbalances, which negatively impact network performance 27,28 .
Network performance enhancement within limited datasets may be improved using convolutional neural networks with traditional machine learning systems (support vector machines), as demonstrated with black nightshade (Solanum lycopersicum L.) and velvetleaf (Abutilon theophrasti Medik) in tomato and cotton (Gossypium hirsutum L.) 29 . The integration of segmentation techniques with neural networks has previously been successful and may help improve precision and recall 30,31 . For example, a weed detection system using blob segmentation and a convolutional neural network achieved 89% weed detection accuracy 32 . For some weed management scenarios, a resource such as CropDeep and DeepWeeds could be used for pre-training or supplementing datasets 33,34 . Using k-means pre-training may improve detection, which improved detection of an image classification convolutional neural network resulted from 2%, up to 93% accuracy 35 .
The developed networks demonstrated detection across two broadleaf vegetable crops within vegetable plasticulture production. The LB annotation technique provided superior results for goosegrass detection in strawberry production (F-score = 0.85) compared to the EP annotation technique (F-score = 0.75). Supplementing the model with a majority of isolated tomato and goosegrass images produced moderate results. The LB annotation technique provided better detection (F-score = 0.65) compared to the EP technique (F-score = 0.56). Results demonstrate that the use of the piecemeal technique alone does not provide adequate detection for field-level evaluation but may represent a suitable oversampling strategy to supplement datasets. The developed network provides an online, real-time, and in situ detection capability for weed management field applications such as precision spraying and autonomous scouts.

Methods
Images were acquired with either a Sony (DSC-HX1, Sony Cyber-shot Digital Still Camera, Sony, Minato, Tonky, Japan) or Nikon digital camera (D3400 with AF-P DX NIKKOR 18-55 mm f3.5-5.6 G VR lenses, Nikon Inc., Melville NY). Training images were taken at the Gulf Coast Research and Education Center (GCREC) in Balm, FL (27.76°N, 82.22°W) and the Strawberry Growers Association (SGA) field site in Dover, FL (28.02°N, 82.23°W). Images were acquired from the perspective of the modified plot sprayer camera (T-30G-6, Bellspray, Inc., Opelousas, LA).
Training data (Training 1, Table 4) were acquired during the strawberry growing season at GCREC and SGA. Images were taken in tandem with a previous study 19 . Strawberry plants were transplanted on October 10, 2017, and October 16, 2017, at the GCREC and SGA, respectively. Several datasets were acquired due to limited goosegrass emergence at GCREC, so a piecemeal solution was undertaken. Training images of tomatoes and goosegrass were acquired separately within a plasticulture setting. A training dataset was developed for goosegrass competing with tomatoes (Training 2, Table 4). Goosegrass was grown in isolation (Training 3, Table 4), with seedlings transplanted on March 12, 2018, and May 15, 2018. Images of only tomato plants were collected for network desensitization (Training 4, Table 4). After preliminary testing, additional images were collected for network desensitization for purple nutsedge (Desensitization , Table 4), which was at the 3-leaf stage, before blooming.
Five datasets were acquired for network testing to meet each crop objective and provide sufficient samples. Images were collected at two commercial strawberry farms (27.93°N, 82.10°W, and 27.98°N, 82.10°W) (Testing 1, Table 4) and supplemented with images from GCREC (Testing 3, Table 4). Images were collected approximately 134 and 136 days after strawberry transplanting from commercial farms and 60 days after transplanting at GCREC. For testing images in tomato production (Testing 3,  www.nature.com/scientificreports www.nature.com/scientificreports/ 5-leaf stage) were transplanted into planting holes containing tomato plants transplanted on March 4, 2019. The tomato data was supplemented with additional tomato images (Testing 4, Table 4) to evaluate the network's ability to discriminate goosegrass from another grass species. A fifth dataset included goosegrass growing in isolation (Testing 5, Table 4).
The image resolution of the Nikon digital camera was 4000 × 3000 pixels. Nikon images were resized to 1280 × 853 pixels and cropped to 1280 × 720 pixels (720p) using IrfanView (Version 4.50, Irfan Skiijan, Jajce, Bosnia). The Sony digital camera image resolution was 1920 × 1080 pixels and images were resized to 720p. Training images were annotated using custom software compiled with Lazarus (https://www.lazarus-ide.org/) in two ways. The EP annotation method used a single bounding box to encompass the entire plant habit. The LB annotation method used smaller bounding boxes along the leaf blade to reduce the overall variability of the target. This approach had been utilized previously to improve detection by focusing annotation to individual Carolina geranium (Geranium carolinianum L.) leaves 19 . Due to the leaf shape and potential goosegrass leaf angles, square bounding boxes were not an ideal solution to minimize background noise by annotating entire leaves. Instead, multiple small square bounding boxes, approximately the width of the leaves, were used to label goosegrass along the length of the leaves. Examples of each method are matched by corresponding bounding box output found in Figs. 1 and 2. Bounding box annotation was the preferable technique compared to pixel-wise annotation due to increased accuracy and reduced time consideration 22 .
The convolutional neural network utilized was tiny-You Only Look Once Version 3 (YOLOv3-tiny) 36 . YOLOv3-tiny was selected for the implementation into a developed prototype precision sprayer for in situ spraying of grasses in horticultural crops including strawberry and tomato plasticulture. The sprayer has a 50 cm distance between the camera and the solenoid-controlled nozzles. As such, image processing speed was considered a priority. The state-of-the-art object detection neural network for iteration speed and capacity for implementation into the controller was selected.
YOLOv3-tiny feature extraction is achieved with the convolutional-based Darknet-19 36,37 . Darknet-19 was derived for YOLOv2, using 3 × 3 filters within its 19 convolutional layers and 1 × 1 filters within its 5 max-pooling layers 38 . Localization is achieved by dividing the image into a grid, predicting multiple bounding boxes within each, and using regression to resolve spatially separated predictions 39 . Bounding box classification permits multiple classification categories and multi-labeling of predictions 37,39 , which is particularly useful for mixed weed communities.
YOLOv3-tiny was trained and tested using the Darknet infrastructure 40 and pre-trained with the COCO dataset 41 . YOLOv3-tiny contained augmentation parameters to reduce the opportunity for overtraining on irrelevant features through altering input images. These parameters included color alteration (exposure, hue, and saturation), flipping, cropping, and resizing. Network training continued until either the average loss error stopped decreasing or the validation accuracy (recall or precision) stopped increasing. For training, 10% of the available images were randomly selected as the validation dataset used during training.
To assess network effectiveness, classification output was pooled and categorized by binary classification for networks derived from both annotation methods. These categories included true positives (tp), false positives (fp), and false negatives (fn). A tp was when the network correctly identified the target. An fp was when the network falsely predicted the target. An fn was when the network failed to predict the true target. Precision, recall, F-score, and accuracy were used to evaluate the network effectiveness to predict targets 12 . Precision measures the effectiveness of the network in properly identifying its target and was calculated as 39,40 : = + Precision tp tp fp (1) Recall evaluates the effectiveness of the network in target detection and was calculated as 42,43 : = + Recall tp tp fn (2) The F-score is the precision and recall harmonic mean and gives an overall performance measure with considerations to both fp and fn, and is calculated as 42 : For comparison purposes, the testing network accuracy was calculated as: = + + Accuracy tp tp fp fn (4) To validate the network training fit, the "map" command was specified. This method used an intersection of union (IoU) with a threshold of 0.25 to evaluate predicted estimates compared to ground-truth annotation. This measure was included to evaluate the effectiveness of the annotation method on overall training. For network detection accuracy assessment of testing datasets, a separate approach was taken for precision sprayer considerations. For both annotation methods, should any of the plant falls within the predicted bounding box, it was considered a hit (IoU > 0). Additional predicted bounding boxes on the same plant were ignored. This method prioritized the detection of some part of the goosegrass plant and is reliant on the ability of the controller software to compensate and increase the area sprayed if necessary.