IMAGE BLUR DETECTION METHOD BASED ON GRADIENT INFORMATION IN DIRECTIONAL STATISTICS

Images are visually inspected for defects that affect downstream operations and qualitatively evaluated immediately after they are acquired. Therefore, there is concern that an increase in the number of images taken affects the quality and process of inspections. Among the defects qualitatively detected in visual inspections, blurring is a serious one, despite its low rate of appearance. However, the blurry images detected in the visual inspections can only be solved by the take another photograph. For this reason, it is not acceptable for any blurry images to be missed in the visual inspections. Therefore, quantitative evaluations are an issue when inspecting photographed images. In the present study, its characteristics in aerial photography were investigated and it was established that motion blur occurs in aerial photography. The motion blur is a condition in which the subject appears to have drifted. We focused on the gradient direction of the image, which is considered to be concentrated in a certain direction. The concept of directional statistics was used to statistically process the gradient direction. The evaluation values calculated from the gradient direction statistics tended to increase with the amount of blurring in the aerial photographs. An experiment was conducted to investigate whether images with blurring could be detected in a large number of aerial photographs. As a result, we were able to successfully detect blurred images that had been overlooked during the visual inspection as well as the images that had been previously detected during the visual inspection.


INTRODUCTION
In aerial photogrammetry, inspection of the aerial photographs must be carried out quickly and with high accuracy. However, there are numerous inspection items that must be examined, such as halation, clouds, smoke, and blurring. In particular, when blurring is found in aerial photographs, re-takes are required, and thus its degree of influence on the work process is higher than that of the other inspection items.
On the other hand, during the inspection of the aerial photographs, the inspector visually inspects each aerial photograph, one by one. Therefore, detection leakage due to individual differences may occur. Furthermore, the performance of digital aerial cameras has improved in recent times, and the number of photographs taken during each flight has increased, such as in the case of the collection of photographs with high overlaps in urban areas and the expansion of joint photographed ranges in adjacent cities. Therefore, there is also the issue of reducing the workload of inspectors.
Previous studies have shown a method of estimating the amount of blur using the geometric pattern of a marker (Okutomi, 2007), and a method of estimating a frequency characteristic of blur and restoring a blurred image (Yoneji, 2005). However, these previous studies aimed at restoring a clear image from a blurry image, which is different from the purpose of this study.
In this study, in order to quantify the amount of blur in aerial photographs and reduce the workload of inspection, we investigate a method to quantitatively evaluate the amount of blur * Corresponding author in aerial photographs from the gradient information of photographs.

CHARACTERISTICS OF BLUR IN AERIAL PHOTOGRAPHS
We first confirmed the characteristics of blurring in aerial photographs taken with digital aerial cameras. Generally, blurring that occurs in photographs that are taken by digital camera are classified into optical blur and motion blur ( Figure 1). In Figure 1(a), it can be seen that optical blur occurs in photographs in which the subject is out of focus. Furthermore, in Figure 1(b), it is evident that motion blur is a condition in which The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B2-2020, 2020XXIV ISPRS Congress (2020 the subject appears to have drifted. However, in the case of aerial photogrammetry, after an imaging plan is established to satisfy the required imaging scale, the aircraft performs imaging while maintaining a ground altitude that exceeds 1,000 meters. Therefore, an out-of-focus image is rarely taken. On the other hand, in Figure 2, a part of an image obtained by photographing the same spot twice, from an aircraft, is shown. The clear aerial photograph shows the roof of every house, however the blurry aerial photograph shows the edges of the roofs drifting in a certain direction. Therefore, it was found that the blurring that occurs in aerial photogrammetry is mainly due to motion blur.

DIRECTIONAL STATISTICS
In directional statistics, periodic data, such as that obtained from a compass or clock, is called angular data. The angular data indicates data distributed on the circumference and has periodicity, so that the angles θ and θ + 180 ° are considered to be the same. For this reason, the handling of data in directional statistics may be different from that in normal statistics. For example, the average of the direction obtained from angular data is not an arithmetic average, but is expressed as a declination of a composite vector.

Gradient Local Features
The evaluation value of blur in the proposed method is calculated using the gradient feature value, calculated based on the method proposed by Pham (2016).
Firstly, a pixel in the input image with an extreme luminance value is obtained as a key point. Secondly, a mask area is set around each key point, and a Sobel filter is applied. Finally, gradient magnitude, gradient direction, and circular variance are calculated by equation 1 and equation 2 using the key points and the Sobel filter-applied images. In this study, only the maximum value is searched, and the ROI size of the Sobel filter is 5 x 5. Furthermore, the ROI size for calculating the statistics is 25 x 25.
where ∇ = image of gradient gravity ∇ = image of gradient orientation = image with Sobel filter applied horizontally = image with Sobel filter applied vertically where = ROI = gradient direction on key points ( ) = number of key points in ROI

Histogram of Gradient Direction
In Figure 4 and Figure 5, the gradient direction histogram created by the gradient features obtained in the previous section are shown ( Figure 4 was created from clear aerial photographs, while Figure 5 was created from blurry aerial photographs).
As can be seen in Figure 4 and Figure 5, the clear aerial photographs in the histogram are almost evenly distributed, while those of blurry aerial photographs are concentrated in a certain direction. On the other hand, in Figure 6, the histogram created based on the angle with the lowest number of occurrences in the gradient direction in blurry aerial photographs, is shown. When the histogram in Figure 6 is divided into two regions near the center, it can be seen that the histogram of each region is similar to the normal distribution. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 6. Histogram of gradient directions based on the smallest frequency from blurry images

Evaluation Value of Blur
A method of calculating the evaluation value of the proposed method from the acquired gradient features will be described in this section.
In Figure 7, the flowchart of the proposed method is shown. The proposed method uses the features shown in Figure 6 to calculate the value of the blurring. However, the distribution of the gradient directions is likely to be different for each aerial photograph. Accordingly, we normalize the histogram based on the direction with the least number of gradient directions for each image ( Figure 7①). Therefore, when the normalized histogram is divided into two regions by the threshold value determined by the Otsu method, a distribution similar to the normal distribution is obtained in each region ( Figure 7②). Then, the kurtosis and skewness of the statistics showing similarity to the normal distribution are calculated from equation 3. Finally, the calculated kurtosis, skewness, mean value of gradient magnitude, and circular variance are used to calculate the evaluation value of blurring in the order shown in Figure 6. Furthermore, as the evaluation value becomes larger than, the possibility of the aerial photograph being blurred becomes higher. Note that although the above description is for a blurry image, the evaluation value is similarly calculated for a clear image by applying the proposed method.   The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

Experiments of Each Feature
The effectiveness of the proposed method is verified in this section. The target feature patterns: "houses", "houses and vegetation", "intersection of roads", "vegetation" and "construction sites" are shown in Figure 7. In addition, the images used for verification are not pseudo-blurry images generated by image processing. However, 512×512 sized cropped images from actual aerial photographs were used.
In Figure 9, the evaluation values calculated by the proposed method are shown. The vertical axis represents the mean value of the gradient magnitude, and the horizontal axis represents the evaluation value. Furthermore, the plotted result of • represents an image without blurring, and the plotted result of ▲ represents an image with blurring.
As can be seen from Figure 9, the evaluation value is greater in the blurred image than in the clear image. Moreover, the evaluation value of a blurry image is greater than that of a clear image. Therefore, the proposed method is effective for detecting a blurry image.

Classification of Blurry Images
In this section, the effectiveness of the proposed method is tested using 99 aerial photographs. Furthermore, the aerial photographs used in the experiment were taken with a digital aerial camera, the specifications of which are listed in Table 1 In this section, the mean evaluation values of cropped images from nine locations, calculated using the proposed method, are shown in Figure 10. The evaluation value of a blurry image varies depending on the features. In addition, the aerial photographs used in the experiment were visually inspected by the inspector in advance and only one image was detected to be a blurry aerial photograph.
In Figure 11, the evaluation values for each aerial photograph are shown. Furthermore, a number of aerial photographs classified from the calculated evaluation values are shown in Table 2. Clear / blurry (Success) 3

Clear / clear (Success) 92
Clear / blurry (Failure) 3 Table 2. Number of classified photographs by classification From Figure 11 and Table 2, it is evident that the aerial photograph with the largest evaluation value was a blurry aerial photograph, detected by visual inspection in advance. In addition, three aerial photographs that could not be detected by visual inspection were also detected (△ marker in Figure 11), indicating the effectiveness of this method. The amount of blurring in three aerial photographs-newly detected by the proposed method was so small that it could not be confirmed without enlarging the images. On the other hand, aerial photographs without blurring were confirmed despite large evaluation values (× marker in Figure 11). The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B2-2020, 2020XXIV ISPRS Congress (2020 In Figure 13, each image and the evaluation values used in the calculation of the evaluation value of one of the three misclassified aerial photographs, are shown. Furthermore, the common feature of these images is the high proportion of bare and cultivated land, which is of low texture. Therefore, the authors consider the cause as follows: 1. When there are few characteristic features (for example, houses) in the input image, the evaluation value of this method tends to show a large value. 2. Since the evaluation value is the average value of nine images, it is affected by outliers.
Therefore, the method of calculating the evaluation value was changed to trimmean, and it was confirmed whether the influence of outliers could be reduced. Note that the highest and lowest evaluation values were removed.
In Figure 12, the result of recalculation by trimmean is shown. It was confirmed that the aerial photographs without blur were distributed in a certain range. Therefore, it is considered that blurred aerial photographs can be detected with a higher accuracy using a combination of the evaluation value and a counting method according to the present method. Figure 12 Classification results by trimmean

CONCLUSION
In this study, we proposed a method to detect blurry aerial photographs by focusing on gradient information. Three characteristics of shake are obtained in this study: 1. The gradient direction of the aerial photograph is concentrated in a specific direction.

2.
There are two peaks in the histogram of gradient direction 3.
The histogram divided into two regions resembles a normal distribution The proposed method was able to classify blurry aerial photographs and clear aerial photographs by calculating characteristics using the above characteristics. On the other hand, some aerial photographs were similar to blurry aerial photographs based on the characteristics of the blur. This may be due to factors such as a low number of features in the input image. However, it is evident that the amount of blur can be quantitatively calculated using the evaluation values obtained in this study. Therefore, the proposed method is expected to make a significant contribution in improving the process of image inspection.
(  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)