EXTRACTION OF DEBILITATED TREES ALONG THE ROAD BY BLOCKED NDVI

A method for extracting degraded trees was developed by using a near-infrared camera modified from a digital camera to photograph roadside trees. Traditionally, remote sensing has utilized vegetation index using near-infrared and red bands as a method to extract degraded trees. However, it was not possible to assess the health of roadside trees sufficiently because the observation from above only observed the canopy of the roadside trees. Observations from the ground can cover the shortcomings because they observe the sides as well as the canopy of the tree. However, ground-based observations are strongly influenced by sunlight, which needs to be compensated for. Also, since the target is trees on the side of the road, it is desirable to take a video of the trees from above the vehicle. The basic idea of this study is simple: a tree where the vegetation index is lower than other trees is considered a cautionary tree, and a tree where the vegetation index changes over time or month is lower than other trees is extracted as a degraded tree. In order to compare videos shot at different times, frame matching of videos and geometric correction between frames were performed. To account for geometric accuracy, pixels were grouped together as blocks, and changes in vegetation indices from block to block were analyzed. In order to improve the accuracy of the analysis, non-vegetation areas were removed from the images. As a result, blocks of debilitated trees were extracted from the trees along the road.


INTRODUCTION
The trees along the road are being planted in a poor environment where they are affected by exhaust fumes and are unable to fully develop their roots. As a result, it has been pointed out that they weaken faster than normal trees, and those weakened trees are at a higher risk of falling. In Japan, nearly half of all roads in the country are 50 years old when they were built. As a result, the trees planted on the side of the road have deteriorated significantly and accidents due to fallen trees have occurred. Therefore, trees along the road are regularly inspected by the road superintendent. The current inspection is done by professionals observing the trees along the road from their vehicles. When he finds a tree that appears to be deteriorating, he gets out of the car and makes a detailed visual inspection of the tree. If it is determined that there is a risk of deterioration, a detailed inspection will be carried out by contacting them. Experts have to visually observe all the trees along the road, which takes a lot of time and effort. The purpose of this study is to apply remote sensing technology to the routine inspection of roadside trees to improve efficiency. If remote sensing technology can be applied to narrow the area to be visually inspected, the efficiency of routine inspection work will be significantly improved. Furthermore, by using the near-infrared region, which cannot be visually confirmed, the quality of periodic inspection work can be expected to be improved. It is well known that there is a high correlation between nearinfrared reflectance and activity of plants, and many studies have been published on the calculation of vegetation indices from satellite and aircraft data and their use. Earth observation satellites observe the same location at about the same time, so the direction and altitude of the Sun at the time of observation will change seasonally, but will remain constant. Therefore, many previous studies process the acquired image data without any correction. However, if the target is a roadside tree, the observation of the tree canopy alone is not enough to examine the health of the tree, and it is necessary to observe it from the side. Therefore, the aim of this study is to develop a system for observing the health of roadside trees by photographing them with a near-infrared camera from a vehicle. The proposed method is envisioned to be used for screening in roadside tree inspection operations. This is expected to increase the efficiency of the inspection process, as the experts will only need to inspect the trees that are judged as debilitated by the system.

OBSERVATION EQUIPMENT
A digital camera (Canon PowerShot S100) was used as a nearinfrared camera with the infrared cut filter removed to allow near-infrared photography ( Figure 1). In order to obtain a visible image, it was also taken with a regular digital camera at the same time. The movies were shot in Full HD with a camera set up from the car towards the side road. The reason why we did not use a series of still images was that depending on the speed of the car, the shutter speed could not be reached in time, which would cause a loss in photographing the trees. In order to grasp the shooting position, the shooting position was measured every second by GPS. Experiments have confirmed that vegetation index values vary greatly with weather. We chose a cloudy day to reduce the amount of change in light intensity.
NIR camera & Digital camera white board Figure 1. Observation equipment The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2020, 2020 XXIV ISPRS Congress (2020 edition)

Automatic extraction of supported frames
A method was developed to automatically extract frames taken of the same tree from different videos to examine changes in vegetation indices of trees observed at different times. First of all, the reference data is the video of the day when the shooting started. The frame is extracted from the reference video so that all the roadside views are captured without any omissions. The frame is a still image, but by extracting the frame from the video, future processing will be able to apply an algorithm that targets the still image. The extracted frame images are extracted so that there are no gaps between the frames and the frame images do not overlap each other as much as possible. The interval at which the frames are extracted can be determined from the speed of the vehicle with the near-infrared camera, the angle of view of the image, and the frames per second (fps). The schematic diagram is shown in Figure 2.

Figure 2. Extraction of the frame image as a reference
Next, from the reference video to be compared with the reference data, the frames that were captured at the same location as the reference frame image are extracted. This method does not use the location information at the time of shooting. This is because, unlike a still image, no positional information is added to the video frame. In the future, a method of acquiring location information in conjunction with GPS will be considered, but this is an issue to be discussed in the future. Basically, since the object is photographed at a similar speed, the interval at which the frames are extracted can be assumed to some extent. The image that is most similar to the reference frame is extracted from a few frames before and after that frame. This process is equivalent to searching the video for the same image as the reference image. Several methods for measuring image similarity have been proposed, but since the target of this study is vegetation, it is difficult to extract the feature points. Therefore, the following methods were tested to see which ones were applicable: haming distance with AKAZE features, SSIM (structural similarity), and dHash (difference hash) were considered as candidates. Each method was applied to a video of a street tree to test whether the combination of images judged to be the most probable was the correct one. The difference between the frame numbers of the frames automatically extracted by these methods and those judged by visual inspection is shown in Figure 3 as an error. As a result, in the present case, automatic extraction with dHash gave the best results. Figure 3. Frame error from the reference image However, dHash was not able to extract frames taken at the same location perfectly, and as shown in Figure 3, the average deviation was 1.65 frames. There was a maximum of six frames off in some places. Figure 4 shows the dHash value at the location, and Figure 5 shows the image at that time.
As can be seen from Figure 5, the mismatched images are very similar. In order to avoid this error, it was found that it is difficult to make a decision based on the dHash value alone, and it is necessary to integrate it with other methods.

Geometric correction process
In the previous section, frames shot at the same location were extracted from videos shot at different times. However, due to the different position and orientation of the camera at the time of shooting, geometric corrections must be made in order to compare images. For geometric correction, feature points were extracted using the ORB (Oriented FAST and Rotated BRIEF) method ( Figure 6), and the corresponding vector (Figure 7) was calculated from these feature points, and from this vector, the pairs of corresponding points that were considered to be free from false detection were set as matching points, and the geometric correction was performed by projective transformation.

Removing the background
In the calculation of the NDVI of leaves, unnecessary background was removed. Luminance values for leaves, structures, and skies were obtained from near-infrared images, and the mean, maximum, and minimum values were calculated, respectively. The results are shown in Figure 8 and Figure 9.

Blocked NDVI calculation
Automatically extracting the corresponding frames and then geometrically correcting them does not exactly match the image. This is because the foliage of a tree is shaped differently depending on the growth of the tree, the changes caused by the wind, and the orientation of the camera. The most common vegetation index, NDVI, was adopted in this study, and the purpose of this study was to determine the changes in NDVI. Therefore, we decided to compare the NDVI blocked together, instead of the pixel-based processing. Although the block size is variable, the results of the blocking process of 240 x 270pixel in length and width are shown in Figure 12.

PRELIMINARY EXPERIMENTS WITH A MODIFIED NEAR-INFRARED CAMERA
To test whether degraded vegetation could be detected using a modified digital near-infrared camera, basic experiments were conducted on conifers planted in pots.

Automatic extraction of frame images
A potted conifer tree was prepared and observed daily at regular intervals with a modified near-infrared camera. As shown in Figure 13, the NDVI values of the conifers varied greatly. This variation is not indicative of a change in vegetation activity, as the plants are in the same condition. The scatter plots of the NDVI and white plate observations are shown in Fig. 14, since the white plate was observed at the same time as the conifer observations. As we can see, there is a positive correlation between light intensity and NDVI. Therefore, it is necessary to correct the NDVI value by the light intensity. Using the data obtained in this study, the correction equations were calculated using the least-squares method.

Observation of NDVI in a tree that was forced to die
Two potted conifer trees were prepared and one was injured and forcibly killed by applying an herbicide. A modified nearinfrared camera was used to observe the change in NDVI. The NDVI observations are shown in Figure 15. Between August 20 and 21, the NDVI change in healthy trees increased to +0.03/day, while the NDVI change in degraded trees decreased to -0.02/day. As shown in Figure 16, the difference in NDVI between healthy and degraded trees became greater after this date. On the other hand, from the August 23 photo shown in Fig. 17, it was not possible to visually identify any deterioration such as leaf wilt. I didn't post a photo, but I was able to see that the degraded tree was dying of leaves on August 27. This was followed by a progression of deterioration, and in the September 4 image, the conifer was completely withered. This confirms that vegetation degradation, which cannot be seen visually, can be captured by a modified nearinfrared camera.

A preliminary experiment to observe NDVI from a dolly
In the previous section, we verified that vegetation activity can be captured by a modified camera that removes the infrared cut filter from the digital camera. In this experiment, trees were observed from a moving object in order to observe street trees, which is the purpose of this study. The observations were made in a video. The final goal was to photograph the roadside trees from above the vehicle, so the interval shooting of the still images would not be possible in time. If a still image is taken, the location information at the time of shooting can be recorded in the image, but we gave up because of the possibility of losing the data. Five similar conifers were lined up, as shown in Figure 18, and trees C and D were damaged and forcibly withered. A nearinfrared camera installed on the dolly was used to take daily movies of the process until the wilting was confirmed by visual inspection while moving the dolly. The NDVI averages for each block were calculated by dividing the frames into 13×4 (138×270 pixels per block) and extracting the frames with conifers from the captured videos. The target frames were extracted from videos with different observation dates using the method described in the previous section, and the blocked NDVI values were compared. First, we performed automatic extraction of the frames, geometric correction of the frames, and blocked NDVI was calculated. The blocked areas are shown in Figure 19. Figure 20 shows the mask image created to remove the background.  Figure 21 shows the evolution of NDVI for each conifer. The data are averaged NDVIs for each block. No light intensity correction has been applied. Healthy trees, A and B, showed high NDVI values. C and E of the degraded trees were lower than those of A and B. As you can see, it is possible to separate healthy trees from degraded ones. However, the healthy tree D showed a similar trend to the unhealthy tree. The reason for this is that D originally had a small amount of leaves and was not dense. Therefore, the NDVI was estimated to be low due to the background of leaf gaps. This suggests that the proposed method is difficult to apply to trees with few leaves.

ROADSIDE TREE OBSERVATION EXPERIMENT
The car was equipped with a near-infrared camera to observe vegetation on the side of the road in Morimoto, Kanazawa City. The vehicle was traveling at approximately 20 km/h at the time of the observation. The reference month was set at June and the same location was observed once a month to compare vegetation indices. As a method of comparison, NDVI trends were analyzed by Pattern 1, which calculates the difference in each observed vegetation index from the base month, and Pattern 2, which calculates the longitudinal change of each month. Figure 21. Comparison of extracting unhealthy trees

Extraction methods for degraded vegetation
In general, in the Northern Hemisphere, vegetation indices are higher in summer and lower towards autumn. It also tends to be higher when it is clear and lower when it is cloudy. In addition, the value of the vegetation index varies by vegetation type. Therefore, it is difficult to determine from the value of the vegetation index itself whether the vegetation is healthy or weak. Therefore, in this study, we mark any vegetation index that has a lower rate of increase than its surrounding vegetation as possibly being weakened. For example, if the vegetation index at a location is increasing, we determine that the bottom 5%, which is not increasing as much, is a potentially debilitating tree; the 5% represents a range outside the two sigma of the normal distribution. Also, even if the vegetation index at the location is declining, the location where the amount of decline is greater than the other is considered a debilitating tree. A location of debilitating vegetation is defined when the difference between the data for the target month and the base data falls within this 5% range, for example, three times. This number and rate of change will color the image with an alarm as the probability that the vegetation is debilitated. Figure 22. Histogram of change amount of NDVI/day

Extraction results of degraded vegetation
The extracted frames are shown in Figure 23. The colored squares are where the proposed method indicated that the vegetation was degraded; in Fm10, areas that were not vegetation were extracted. The reason for this is that the background deletion didn't work. On the other hand, he had definitely captured the areas where the vegetation was weakening. Figure 23. Extracted degrading blocked NDVI Figure 23 The coloured blocks shown in Fm6 and Fm7 show the same tree. This tree showed a large reduction in NDVI in pattern 1 of the extraction flow, especially compared to June to October. Figure 24 shows the NDVI change per day from June to October for Fm6. The NDVI around the extracted yellow blocks also showed a decreasing trend, indicating that the NDVI of the entire tree was decreasing. Therefore, this tree can be considered to be degraded. At this point, the foliage is so overgrown that it is difficult to easily determine if the tree is deteriorating by visual inspection. However, if we follow the subsequent evolution of this tree, we can see that it is deteriorating and decaying. The other locations that were also extracted are shown in Figure 25. This tree can also be seen to have lost its leaves afterwards.

CONCLUSIONS AND DISCUSSIONS
In this study, we used a modified camera with the infrared cut filter removed from a commercially available digital camera and developed an algorithm for extracting debilitated vegetation from a moving vehicle by photographing the vegetation. From videos taken at different times of the year, frames capturing the same tree were automatically extracted and the frames were geometrically corrected from the feature points. We did not manually extract the feature points or acquire the reference points, so we were able to process them automatically. A light correction equation was devised to compare vegetation indices. He proposed blocking so that vegetation could be compared and showed that it was useful. In addition, we were able to successfully extract vegetation that was in decline from the transitional changes in the vegetation index. We were able to observe that the vegetation extracted from the proposed algorithm was debilitated, with subsequent defoliation. This indicates that we can expect the trees to decline in the future.
In the future, we plan to apply this method in different locations and under different conditions to improve the algorithm. In addition, a roadside tree inspection system will be developed based on this proposed method.