USING MULTITEMPORAL HYPER- AND MULTISPECTRAL UAV IMAGING FOR DETECTING BARK BEETLE INFESTATION ON NORWAY SPRUCE

Various biotic and abiotic stresses are threatening forests. Modern remote sensing technologies provide powerful means for monitoring forest health, and provide a sustainable basis for forest management and protection. The objective of this study was to develop unmanned aerial vehicle (UAV) based spectral remote sensing technologies for tree health assessment, particularly, for detecting the European spruce bark beetle (Ips typographus L.) attacks. Our focus was to study the early detection of bark beetle attack, i.e. the “green attack” phase. This is a difficult remote sensing task as there does not exist distinct symptoms that can be observed by the human eye. A test site in a Norway spruce (Picea abies (L.) Karst.) dominated forest was established in SouthernFinland in summer 2019. It had an emergent bark beetle outbreak and it was also suffering from other stress factors, especially the root and butt rot (Heterobasidion annosum (Fr.) Bref. s. lato). Altogether seven multitemporal hyperand multispectral UAV remote sensing datasets were captured from the area in August to October 2019. Firstly, we explored deterioration of tree health and development of spectral symptoms using a time series of UAV hyperspectral imagery. Secondly, we trained assessed a machine learning model for classification of spruce health into classes of “bark beetle green attack”, “root-rot”, and “healthy”. Finally, we demonstrated the use of the model in tree health mapping in a test area. Our preliminary results were promising a nd indicated that the green attack phase could be detected using the accurately calibrated spectral image data.


INTRODUCTION
Forests provide a multitude of critical ecosystem services, such as climate regulation, water supply and regulation, timber, energy, habitat for biodiversity, clean air, erosion control and many others. Various biotic and abiotic stresses are threatening our forests. For instance, climate change affects tree species composition, the extreme climate effects such as droughts impair the tree health and resilience, the wild forest fires burn down large forest areas all over the globe, and the attacks of insect pests are causing timber losses at increasing intensities; these stresses are increasing due to climate change (e.g. Trumbore et al., 2015).
Modern remote sensing technologies provide powerful means for monitoring forest health, which provides a sustainable basis for forest management and protection. The objective of this study was to develop unmanned aerial vehicle (UAV) based remote sensing technologies for tree health assessment, particularly, for detecting the European spruce bark beetle (Ips typographus L.) attacks. Bark beetles have caused serious spruce forest mortality in the recent years in central Europe, Russia, and Canada, and now they are building outbreaking populations up in the North, where the warming climate is providing suitable conditions for their spreading.
Of special interest is the detection of the so-called "green attack" phase, which is the outcome of the first interaction between the beetle and the host tree, and occurs when the host is being colonized by the pioneer individuals of the bark beetle (Abdullah et al., 2018). In the green attack phase, the beetle lays its eggs under the bark of the tree, and the infested trees do not have any distinct symptoms that can be observed by the human eye (Niemann and Visintini, 2005). The only visible symptoms of the initial colonization phase are entrance holes of beetles in the tree trunks, fresh resin flow and brown sawdust on the ground that are very difficult to observe by satellite or UAV remote sensing. During this stage, the infested tree is still physiologically green and very much alive, although exhibiting stress in the near infrared (invisible to the human eye) (Niemann and Visintini, 2005). Näsi et al. (2015) were the first to utilize UAV-based hyperspectral imagery for identifying different infestation stages of the European spruce bark beetle (Ips typographus L.). The UAV data were captured at 100 m height from the ground level with a GSD of 10 cm using a hyperspectral camera based on the Fabry-Pérot interferometer from the test site in Southern Finland. The same sensor was also operated from an airplane over larger areas with a GSD of 50 cm (Näsi et al., 2018a). Trees were classified to healthy, infested and dead classes based on their colour symptoms using the Support Vector Machine classifier with an overall accuracy of 81% and 73% with the UAV and aircraft, respectively (Näsi et al., 2018a). Klouček et al. (2019) used a low-cost UAV-based system with RGB and modified CIR cameras to capture multispectral time series of the bark beetle (Ips typographus) outbreak area in the northern part of the Czech Republic. The results indicated that the low-cost UAV-based sensor system provided information of various stages of bark beetle infestation across seasons; results also indicated that early phases of bark beetle attack could be detected. Safonova et al. (2019) studied the detection of the invasion of the bark beetle (Polygraphus proximus Blandford) in fir forests (Abies sibirica Ledeb) in Russia using RGB photogrammetric imagery. They proposed a two-stage solution; first searching for the potential tree crowns and in the second stage using a convolutional neural network (CNN) architecture to predict the fir tree damage stage in each detected candidate region.
Our objective was to develop methods for the detection of the bark beetle (Ips typographus L.) infestation at early phases utilizing hyperspectral and multispectral imaging from UAV. Firstly, we explored deterioration of tree health and development of spectral symptoms using a time series of UAV hyperspectral imagery. Secondly, we trained assessed a machine learning model for classification of spruce health into different health classes. Finally, we demonstrated the use of the model in tree health mapping in a test area.

Test area
The test site was a Norway spruce (Picea abies (L.) Karst.) dominated forest area of size of 10 ha in Espoo, in Southern Finland. The area has an active bark beetle infestation but also other deficiencies, especially due to the root and butt rot, later referred as root-rot of spruce (Heterobasidion annosum (Fr.) Bref. s. lato). Field surveys were carried out by a forester several times during the summer 2019. He identified a total of 28 spruces in the area with active bark beetle attack and 16 trees with root-rot through intersecting stump-level drillings. Other spruces were classified as healthy. We selected a total of 28 spruces to the healthy tree sample. There was one spruce that died during the period due to the bark beetle infestation. It went through phases from a visually healthy crown (green attack), yellow, red/brown (red attack), and finally grey (dead) colors. Other infested spruces did not develop visible crown color symptoms, thus they could be considered as green attack samples. The test area is presented in Figure 1.

UAV data captures
Altogether seven remote sensing campaigns were carried out in the area from August to October 2019 (Table 1). A quadcopter drone was equipped with a multisensory remote sensing instrumentation consisting of two oblique RGB cameras, a hyperspectral 2D format frame camera, and a multispectral camera. The RGB sensors were used for the object 3D modeling and the hyperspectral and multispectral imagery were used for the spectral studies of the crown symptoms. The RGB cameras were two Sony A7R II 42.4 megapixels cameras, with a Sony FE 35 mm f/2.8 ZA Carl Zeiss Sonnar T* lens. The hyperspectral camera was the 2D-format frame camera Rikola, which is based on an adjustable Fabry-Pérot interferometer (FPI) acquiring spectral bands using a time sequence based process, i.e., the acquisition is not simultaneous for all bands of the same hyperspectral cube. The sensor was set to collect 46 bands in the spectral range of 502-907 nm with 4-10 nm of full width at half maximum (FWHM). Furthermore, the Micasense Altum camera was used; it has five multispectral bands in blue, green, red, red-edge, and near-infrared spectral ranges, as well as a thermal band. The flying height was 140 m above the ground level, thus the ground sample distances (GSDs) were 2.1 cm for the RGB camera, 8.6 cm for the FPI camera, 6.0 cm for the Altum multispectral bands, and 90 cm for Altum thermal band. Each camera was configured to capture images in two-second intervals. The forward and side overlaps were 87% and 76% for the hyperspectral camera, 89% and 83% for the Altum multispectral camera, and 91% and 88% for the RGB camera. The system was equipped with the Applanix APX 15 GNSS/IMU system for accurate direct georeferencing. Two sensors measuring incident irradiance were installed onboard the drone: the FGI's Aerial Imaging Reference System (AIRS) (Suomalainen et al., 2018) and the Altums's incident irradiance sensor.
We installed to the study site altogether five GCPs, which were measured using Topcon Hiper HR multi-purpose GNSS receiver's static measuring mode with a coordinate accuracy of 9 mm horizontally and 15 mm vertically. The GCPs were signalled with painted circular targets with a diameter of 0.6 m. For the reflectance transformation, reflectance panels of size of 1 m x 1 m were placed on ground with reflectance ranging from 0.03 to 0.50. Micasense Altum manufacturer provides its own reference panel, which was used on each campaign, by acquiring images before and after of each flight.

Data processing
Rigorous photogrammetric and radiometric processing was used to derive calibrated remote sensing data products (Nevalainen et al., 2017;Honkavaara et al., 2017Honkavaara et al., , 2018. A photogrammetric workflow was applied for each of the three camera datasets using Agisoft Metashape 1.5 software and FGI in-house software. The workflow includes aligning images through a self-calibrating process, followed by dense point cloud and orthomosaic generation procedures. The RGB and Micasense datasets were processed using the Metashape. For the FPI-camera, four bands were first processed with the Metashape together with RGB data images in the same processing (Nevalainen et al., 2017). Once the four bands of the data cubes were aligned, the in-house RadBA software  was used to estimate the position and orientations of the rest of the bands.
In order to compare the spectral features of multispectral and hyperspectral images over the time, it was necessary to calibrate the image radiometric values to spectral reflectance with a high accuracy. Firstly, the sensors were radiometrically calibrated to provide at-sensor radiance values. The radiometric processing of the Micasense dataset was done in the Metashape. Micasense data was loaded into the Metashape as a multispectral camera, where the metadata of the images holding information about camera GPS data, calibration parameters, and radiometric parameters is recognized by the software. For the reflectance transformation, we evaluated two methods. In the first method, images of the camera's reflectance panel taken before and after the flight were automatically located and identified by Metashape, and the reflectance calibration was then performed based on panel data and, in some cases, also DLS information (Agisoft Metashape, 2020). In the second method, we carried out the reflectance transformation using images of our own reflectance panels taken at the beginning and end of the flight. We manually located and identified the panels in the Metashape, and performed the reflectance calibration using the panel data. The second method was selected for the study as it provided better results. Next, the reflectance orthomosaics were generated and exported as GeoTIFF files. In the radiometric processing of the hyperspectral data, the radiometric block adjustment method developed by Honkavaara et al. (2013; in RadBA software was applied to each dataset. Examples of the RGB and hyperspectral mosaics from August 15 and October 15 are shown in Figure 2.

Data analysis and classification
The data analysis had three major objectives. Firstly, we studied the spectral characteristics of different vegetation health classes. Secondly, we developed models for The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2020, 2020 XXIV ISPRS Congress (2020 edition) classification of different health classes, and third, we demonstrated the entire analysis chain in production of spruce health maps in the test area. All of these analyses were carried out using the hyperspectral images. The Micasense multispectral images were used in the classification of health classes in order to compare these sensors.
The analysis was carried out using the FGI's highly automated Python-based processing pipeline that is based on works of Nevalainen et al., (2017) and Näsi et al., (2015;2018a, 2018b. The steps include individual tree detection, feature extraction, tree species classification, and tree health classification. Individual trees were detected from the 3D point clouds using the FUSION software (Nevalainen et al., 2017). The species classification was carried out using the Random Forest (RF) classifier utilizing the spectral features (Nevalainen et al., 2017). Finally, the health classification of the spruces was carried out using the RF classifier and spectral features (Näsi et al. (2018a(Näsi et al. ( , 2018b.
In the high-resolution drone images there are hundreds of pixels in the crown area. In the analysis we used different spectral values including average and median values of all, the brighter than median, the six brightest, the darker than median, and the normalized pixel values. The normalization was carried out by averaging the spectra with the average value of the entire spectrum. We used individual tree crown size for each tree based on the average tree crown diameter determined by the FUSION software. These methods resulted in a total of 368 spectral observations from the 46 bands of the hyperspectral images and 40 observations from the multispectral bands. In addition, 42 and 30 vegetation indices were extracted from the hyperspectral and multispectral images, respectively.
In phase 1, the spectra of different classes were compared to each other using one sided variance analysis with p value of 0.01 to find the spectral bands with significant differences. This analysis was carried out using the average spectral values over the crown area.
In phase 2, we used machine learning to develop models for classifying different health classes including "bark beetle green attack", "root-rot", and "healthy". The RF classifier was used in this study and we used all spectral and index features to develop the classification models (Näsi et al., 2018b). The leave-one-out method was used to evaluate the accuracy of the classification.
In the final phase, tree health maps were created over the entire area using the pipeline described above. In the tree health classification all spectral and index features were used utilizing the models developed in the phase 2.

Spectral analysis
One of the spruces went through the full range of visual symptoms from the green attack phase to dead during the period of monitoring as visualized in Figure 3a; for comparison, a stable reference tree is shown in Figure 3b. The normalized spectra of the dying and stable spruces were calculated as averages over the crown area from the hyperspectral UAV images in each date (Figure 4). During the first dates (August 15 and 18), the spectra followed the healthy vegetation spectra. However, on August 30 the tree deterioration became clearly visible in the visible wavelengths and, particularly, in the chlorophyll absorption feature at around 680 nm disappeared. The spectra of the stable reference spruce remained stable during the whole monitoring period.
(a) (b) Figure. 3. Visual examination of (a) a dying and (b) a stable spruce from different views on four dates July 24, August 28, August 30 and October 15. As there was only one tree that died during the monitoring period, the other trees with bark beetle infestation represented the "green attack" phase. Average spectra were calculated for the sample of healthy and bark beetle attacked trees for each date using the hyperspectral UAV data ( Figure 5). It could be seen that the spectra on the visible spectral range (500-680 nm) was higher for the spruces suffering from the bark beetle attack than for the healthy spruces. The variance analysis indicated that there were significant differences in the spectra in this range on dates 30.8, 2.10 and 15.10. In the first dates (15.8 and 22.8) the differences were smaller, which was consistent with the expectation that the symptoms caused by the bark beetles were not that serious in the beginning of the infestation. Also on 17.9 the analysis did not indicate significant differences, which could be due to some issues with the radiometric calibration.

Models for tree health classification
The model for the tree health classification to the classes of "bark beetle green attack", "root-rot" and "healthy" was trained using the RF. The model was assessed using the leaveone-out method ( Figure 6). The overall accuracy was 40-55% for the hyperspectral data and 40-50% for the multispectral data; overall, the accuracy was slightly better for the hyperspectral data. At best, the kappa-coefficient was 0.3 on September 17. From the confusion matrix, it can be seen that out of 28 spruces suffering from the bark beetle infestation, 18 were classified correctly, whereas two were classified to the class "root-rot" and 8 to the class "healthy". Considering the healthy spruces, 19 were correctly classified to the class "healthy", five were classified to the class "bark beetle green attack" and four to the class "root-rot". The accuracies were not high, but better than only random classification. It is important to notice that identifying the green attack phase is an extremely challenging task and the detection of "root-rot" might be even more difficult, and that the both of these disturbance factors can be within the same tree. Furthermore, the issues that there was no information about the severity of the bark beetle infestation and also the exact attack dates were not available might also have caused some confusion to the results. Thus bearing in mind these factors, the result can be considered promising.

Spruce health mapping
In order to carry out the spruce health map, we first performed species classification. In addition to the spruces (Figure 1), 10 pines and 10 deciduous trees were selected to the species training data. The species classification was carried out using the August 15 hyperspectral image dataset and the RF classifier (Figure 7). Then the health classification was carried out for the trees classified as spruces using the August 30 dataset. The results showed that there were 973 healthy spruces, 811 spruces with root-rot and 573 spruces with bark beetle infestation. (Figure 8). This result has not yet been confirmed. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2020, 2020 XXIV ISPRS Congress (2020 edition)

CONCLUSION
Our study developed techniques for detecting early phases of bark beetle (Ips typographus L.) infestation based on UAV hyperspectral and multispectral imaging. Firstly, we explored impact of the deterioration of tree health on the remote sensing spectral signatures using a time series of UAV hyperspectral imagery. Secondly, we trained a machine learning model for classification of spruce health into classes of "bark beetle green attack", "root-rot", and "healthy" using hyperspectral and multispectral UAV images. Finally, we demonstrated the use of the model in a tree health mapping task in the test area. Our preliminary results were promising and indicated that the green attack phase could be detected using the accurately calibrated spectral image data. The hyperspectral imagery provided slightly better results than the multispectral imagery. In the further studies our aim is to further improve the classification method utilizing more extensive reference datasets and more comprehensive classification techniques.