MAPPING BARLEY LODGING WITH UAS MULTISPECTRAL IMAGERY AND MACHINE LEARNING

Unmanned Aircraft Systems (UAS) are demonstrated costand time-effective remote sensing platforms for precision agriculture applications and crop damage monitoring. In this study, lodging damage on barley crops has been mapped from UAS imagery that was acquired over multiple barley fields with extensive lodging damages in two aerial surveys. A Random Forests classification model was trained and tested for the discrimination of lodged barley with an overall accuracy of 99.7% on the validation dataset. The crop areas with lodging were automatically delineated by vector analysis and compared to manually delineated areas using two spatial accuracy metrics, the Area Goodness of Fit (AGoF) and the Boundary Mean Positional Error (BMPE). The average AGoF was 97.95% and the average BMPE was 0.235 m.


INTRODUCTION
Crop lodging is a well-known damage defined as any permanent deviation from the normal vertical position of plants and occurs mainly in barley, wheat, oat, rice, and other cereal crops. It is related to weather events such as strong wind or rain, excess fertilizers, irrigation, cultivar, soil characteristics, and disease (Pinthus, 1974;Rajkumara, 2008). Crop lodging is usually most severe at the heading and ripening growth stages. Lodging damage is responsible for significant losses in crop yield. As a result, there is a need for an accurate damage evaluation tool to determine yield losses for crop insurance paybacks. This evaluation can be done by field surveys with hand-held equipment that is assisted with Global Navigation Satellite Systems (GNSS). However, such field surveys can be laborious and time-consuming with a high probability of human errors. A suitable alternative is to use remote sensing platforms that provide high spatial resolution imagery allowing detailed crop lodging mapping. Chauhan et al. (2019) reviewed remote sensing studies on crop lodging. Space-borne platforms can be used (Chauhan et al., 2020;Shu et al., 2020), but they lack in spatial resolution and temporal flexibility due to fixed-time overpasses while being prone to weather limitations such as clouds for optical instruments. By contrast, UAS campaigns are versatile as they can be conducted at any time. They provide spatially-accurate imagery, and the data acquisition is easy and cost-efficient (Zhang and Kovacs, 2012). Chu et al. (2017) used UAS Red-Green-Blue (RGB) and Near-Infrared (NIR) imagery to map maize lodging based on plant height thresholds estimated with multivariate regression. The same method was applied on UAS RGB images for mapping barley lodging (Wilke et al., 2019). Rice lodging was mapped with UAS RGB imagery and decision tree classification (Yang et al., 2017) and with RGB and thermal infrared imagery using support vector machine (SVM) classification (Liu et al., 2018). Canola and wheat lodging was mapped with multispectral UAS imagery using SVM classification (Rajapaksa et al., 2018).
The study aims to develop a method that used multispectral MicaSense RedEdge imagery acquired from a UAS to map lodging damages in seven barley fields during the ripening growth stage. Firstly, we compared the spectral responses of lodged and standing barley as a function of the bands. Secondly, a robust machine learning and information extraction pipeline is built, based on Random Forests and a Geographic Information Systems (GIS) vector analysis of the classification results, to automatically provide the delineated areas and boundaries of the lodging zones. The resulting areas and boundaries are then compared to manually delineated areas by using two spatial accuracy metrics, the Area Goodness of Fit (AGoF) and the Boundary Mean Positional Error (BMPE).  ESRI Satellite (ArcGIS/World Imagery).

MATERIALS AND METHODS
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2021 XXIV ISPRS Congress (2021 edition) The UAS consisted of a DJI Matrice 100 light Unmanned Aerial Vehicle (UAV) and the MicaSense RedEdge3 multispectral camera (MicaSense Inc., U.S.A.) capturing data from the nadir ( Table 1). The UAS MicaSense image datasets were processed using the methodology described in Figure 4. First, the images of each survey were individually inserted into a photogrammetric pipeline using the Pix4D Mapper software (Pix4D SA, Switzerland), performing radiometric correction, orthorectification, and stitching to produce a reflectance orthomosaic for each band.    Each orthomosaic was then inserted into a machine learning and vector analysis pipeline. The QGIS software (QGIS, 2019) was used for visualization, map generation, and other GIS and vector analysis procedures. The orthomosaics were then subjected to a supervised classification that has the three following classes: Standing Barley, Lodged Barley, and Other Vegetation. Representative training areas for each class were manually delineated and used to compute the Jeffries-Matusita (J-M) distance, which is a measure of the spectral separability of the classes. The J-M distance is the distance between random probability distributions (Richards and Jia, 2006;Wacker and Landgrebe, 1972). It has a range between 0 and 2 and it computes using the Bhattacharya (B) distance (Equations 1 and 2), which has the [0, +∞) range. For a pair of multivariate distributions, in our case a pair of classes C1 and C2, the J-M distance is computed by: (1) where B is the Bhattacharya distance that is computed by: ( 2) where M is the root Mahalanobis distance (Equation 3) (Varmuza and Filzmoser, 2016) between the class means with respect to σ (Equation 4) (3) Where: (4) With: μ1, μ2 = mean of classes C1 and C2, respectively σ1, σ2 = covariance matrices for classes C1 and C2, respectively The J-M lowest value of 0 indicates that the distributions are highly correlated, thus the classes are spectrally not separable, whereas the upper asymptotic limit of 2 indicates excellent class spectral separability. Since the J-M distance assesses the separability of the training datasets before classification, high J-M distances indicate that photointerpretation and in-situ human perception of lodging damage are good enough for the machine learning pipeline.
The training areas were then used in a supervised classifier, Random Forests (RF). RF was introduced by Breiman (Breiman, 2001(Breiman, , 2003 and implemented in R (Liaw and Wiener, 2002). RF has been widely used in remote sensing (Belgiu and Drăguţ, 2016) due to its demonstrated statistical robustness and deployment advantages in parametrization and computational needs. RF is a non-parametric decision tree ensemble supervised classification model that does not assume a normal distribution of the data. In our processing, we set the number of trees grown (ntree) at 500 and we kept the number of random features selected for the growth of each tree from the feature space (mtry) at its default value, which is the square root of the size of the feature space used, rounded down. The classification accuracy is determined through the RF Out-Of-Bag (OOB) error rate, which is the aggregation of individual OOB error rates from each individual tree grown, when the data omitted (~37%) from the training of the decision trees due to the random bootstrapping with replacement strategy in the data The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2021 XXIV ISPRS Congress (2021 edition) sampling are parsed and classified by the trees that were not trained with them. The OOB error rate is the complementary percentage of the overall classification accuracy. The RF OOB confusion matrix allows the computation of the class user's and producer's accuracies (Congalton, 1991). The RF implementation in R generates two feature importance matrices: 1) the MeanDecreaseGini, which represents the mean decrease in the Gini index of node impurity when a feature is split at the creation of a node and 2) the MeanDecreaseAccuracy, which represents the mean decrease of prediction accuracy when a feature is permutated. For the graphical representations and plots in this study, the ggplot2 (Wickham, 2016) library was used.
The resulting classified images were afterward cleaned from some salt and pepper noise using a 1-pixel radius Majority filter of the System for Automated Geoscientific Analyses (SAGA) (Conrad et al., 2015). The resulting classified images were then inserted into the vector analysis pipeline. The polygonize function of the GDAL library (GDAL/OGR, 2019) was employed to vectorize the classified fields, and the results were refined by disposing minor patches of mixed lodged and standing barley within the damaged areas and finally smoothening the borders with equal vector buffering and debuffering. The resulting lodged areas were compared to lodged areas that were manually delineated over the UAS orthomosaics through photo-interpretation. Such delineation was made easier thanks to the very high spatial resolution of the data (~8cm) and the geolocation accuracy of the UAS data and the photogrammetric procedure in the making of the final orthomosaics. Following Vlachopoulos et al. (2020aVlachopoulos et al. ( , 2020b, the accuracy of the methodology was estimated by two spatial accuracy metrics: the Area Goodness of Fit (AGoF) and the Boundary Mean Positional Error (BMPE). AGoF computes the percentage of area overlap between the manual and the machine learning derived polygon areas as follows (Equation 5).
Where: A = manual delineated polygon area (m 2 ) B = machine delineated polygon area (m 2 ) C = manual and machine delineated polygons intersection area (m 2 ) AC = | A -C | BC = | B -C | BMPE calculates the aggregated minimum distances between the manually and the geographical points sampled at 0.5m on the machine delineated polygons (Equation 6). The smaller the BMPE output, the higher the overall positioning accuracy.
Where: N = number of point samples MinDisti = minimum distance between the i th point sample and the manually delineated polygons The RF model was finally assessed for its accuracy and transferability with a validation dataset, spatially independent from the training and testing dataset, using only data from Flight2 fields. Approximately 20,000 randomly chosen pixels from spatially dispersed training sites for each class were used for validation, with an overall accuracy of 99.7%. The related confusion matrix with UA, EC, PA and EO are shown in Table  4. Our overall validation accuracy of 99.7% is higher than the one of Yang et al. (2017), who used a decision tree classification on UAV RGB imagery to map lodged rice crops (96.17%), and Rajapaksa et al. (2018), who applied the SVM classification to multispectral images to map canola (96%) and wheat (92.6%) crops. Figure  5 shows the MeanDecreaseAccuracy and MeanDecreaseGini feature importance plots for the classification produced by RF. For both metrics, the Red reflectance is the dominant feature. The red reflectance is also the one that allows the highest graphical discrimination between the three classes as shown by the boxplots of the reflectance values computed with the training pixels of each class and each band ( Figure 6).  The classifier was trained and validated upon the fields from the first flight while the second flight barley fields were exclusively used to validate the generated RF classification. Both flight fields were used to assess the spatial accuracy of lodging damage assessment after the vectorization procedure with the AGoF and BMPE metrics. The RGB true color composite and the resulting classified image for the Flight1 fields are shown in Figure 8 and for Flight2 fields in Figure 9. The vector analysis for Flight1 resulted in an AGoF of 96.8% and a BMPE of 0.17m. For Flight2, the AGoF was 99.1% and the BMPE was 0.3m.

RESULTS AND DISCUSSION
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2021 XXIV ISPRS Congress (2021 edition)

CONCLUSIONS
Two flight campaigns were conducted for surveying seven barley fields with lodging damage during the ripening growth stage in late August 2018 in PEI, Canada, using a UAS equipped with a multispectral MicaSense RedEdge camera. Firstly, a photogrammetric pipeline was utilized to produce the orthomosaics from each flight. The orthomosaics were inserted into a machine learning pipeline that used the RF classification model followed by a vector analysis for the delineation of the damaged barley areas from the classified images of each flight. The overall accuracy of the RF classification was greater than 99%, showing that the RF classifier is a robust and consistent machine learning tool for the classification of UAS imagery in three classes (standing barley, lodged barley, other vegetation).
The vector analysis produced automatically delineated areas and borders were compared to a photo-interpreted delineation of the damaged areas. The comparison produced excellent results for both flights and all barley fields, with an average AGoF of 98% and an average BMPE of 0.23m. Such results can enable the timely and accurate estimation of the damages on barley crops for yield estimation and insurance purposes. The proposed methodology was limited to the classification of ripening lodged barley and similar studies should be undertaken to test the method over lodging damages in other growth stages for barley and on other crops. Additional information derived from spectral data such as textural and vegetation indices should be tested in future work, especially when surveying lodging damage on different crops and growth stages. Such applications will allow early detection of crop lodging and strengthen agricultural decision support systems for possibly preventing more extensive damages.