ASSESSMENT OF INFLUENCE OF IMAGE PROCESSING ON FULLY AUTOMATIC UAV PHOTOGRAMMETRY

: Nowadays UAVs have been widely used for large scale surveying and mapping. Compared with traditional surveying techniques, UAV photogrammetry is more convenient, cost-effective, and responsive. Aerial images, Position and Orientation System (POS) observations and coordinates of ground control points are usually acquired during a surveying campaign. Aerial images are the data source of feature point extraction, dense matching and ortho-rectification procedures. The quality of the images is one of the most important factors that influence the accuracy and efficiency of UAV photogrammetry. Image processing techniques including image enhancement, image downsampling and image compression are usually used to improve the image quality as well as the efficiency and effectiveness of the photogrammetric data processing. However, all of these image processing techniques bring in uncertainties to the UAV photogrammetry. In this work, the influences of the aforementioned image processing techniques on the accuracy of the automatic UAV photogrammetry are investigated. The automatic photogrammetric data processing mainly consists of image matching, relative orientation, absolute orientation, dense matching, DSM interpolation and orthomosaicing. The results of the experiments show that the influences of the image processing techniques on the accuracy of automatic UAV photogrammetry are insignificant. The image orientation and surface reconstruction accuracies of the original and the enhanced images are comparable. The feature points extraction and image matching procedures are greatly influenced by image downsampling. The accuracies of the image orientations are not influenced by image downsampling and image compression at all.


INTRODUCTION
Compared with traditional surveying techniques, UAV photogrammetry is more convenient, cost-effective, and responsive.Nowadays UAV photogrammetry has been widely used for large scale surveying and mapping (Haala et al., 2011;Remondino et al., 2011;Colomina and Molina, 2014).The photogrammetric data processing mainly consists of image matching, image orientation, dense matching, DSM interpolation, orthorectification and orthomosaicing.The data processing pipeline usually starts with image matching.The SIFT algorithm proposed by Lowe (2004) has been extensively used for matching UAV aerial images (Rupnik et al., 2015;Rupnik et al., 2017).Keypoint descriptors indexed by approximate nearest neighbors algorithm (Arya et al., 1998;Muja and Lowe, 2014) are used to determine candidate tie points.Outliers from initial matches were removed using the fundamental matrix (Hartley and Zisserman, 2004) with RANSAC (Fischler and Bolles, 1981).After the image block is oriented, the dense matching is carried out using algorithm such as semi-global matching (Hirschmüller, 2008).After the dense matching, the dense point cloud of the surveying area is generated, which is used to construct the Triangulated Irregular Network (TIN).On the basis of the TIN, a raster DSM is interpolated.The aerial images are then orthorectified using the DSM and the orthomosaic of the surveying area is finally generated.
Aerial images, Position and Orientation System (POS) observations and coordinates of ground control points are usually acquired during a surveying campaign.UAV aerial photography is usually more sensitive to weather and lighting conditions.It is not uncommon that aerial images are acquired during bad weather and lighting conditions, especially in time critical UAV photogrammetry applications.Wind greatly influences positions and orientations of UAVs, which may result in image blur.Poor air conditions (haze and fog) degrade the quality of the aerial image.Image definition is greatly influenced by the weather and lighting conditions.
Aerial images are the data source of the feature point extraction, dense matching and orthorectification procedures.The quality of the images is one of the most important factors that influence the accuracy and efficiency of UAV photogrammetry.The influence of the image quality on the quality of the photogrammetric data processing has been investigated by a few researchers.Sieberth et al. (2016) developed a filtering process to automatically detect blurred images in UAV image sets.Liang et al. (2017) developed a three-dimensional simulation and visualization system to investigate the influence of weather conditions and the relief of the terrain on the flight plan, image matching, POSsupported direct georeferencing, and orthomosaicing.Wierzbicki et al. (2015) assessed the influence of UAV image quality on the orthophoto production and found that images acquired in poor weather conditions degrade the final quality and accuracy of a photogrammetric product.In practice, image processing techniques including image enhancement, image downsampling and image compression are used to improve the image quality as well as the efficiency and effectiveness of photogrammetric data processing.The images that collected during bad weather and lighting conditions are usually preprocessed to enhance contrast and filter noise.Image resolution influences the space and time efficiency of the photogrammetric data processing.It takes more space to store higher resolution images and the intermediate processing results.It takes more time to transmit and process higher resolution images.To reduce the amount of data storage and shorten the processing time, images are often downsampled before feature extraction.Image compression can be used to further reduce the size of UAV images.It takes less space to store compressed images.When processing large scale data on cloud computing platforms, it can save nontrivial amount of bandwidth and time to transmit compressed images over network (Untzelmann et al., 2013;Hartmann et al., 2016;Nocerino et al., 2017).
However, all of the aforementioned image processing techniques bring in uncertainties to UAV photogrammetry.These image processing techniques often smooth out details of images, which lead to loss of salient features.And the loss of salient features may lead to unstable image orientation and less accurate surface reconstruction.In this work, influences of the image processing techniques on fully automatic UAV photogrammetry are investigated.Section 2 details the methodology of the study.Experimental results are discussed in Section 3 and conclusion is made in Section 4.

METHODOLOGY
The automatic photogrammetric data processing mainly consists of image matching, image orientation, dense matching, DSM interpolation, orthorectification and orthomosaicing.SIFT feature points are firstly extracted from the aerial images and used for image matching.In the image matching procedure, image pairs to match are generated by using the simulation method presented in (Liang et al., 2017).The POS data are used for determination of the image pairs.The positions of the images are transformed from the geodetic coordinate system to the East-North-Up (ENU) coordinate system.The orientations of the images are used to construct the rotation matrix R defined in the collinearity equations (equation (1)).
where (X, Y, Z) is the position of a three dimensional point P under the ENU coordinate system, (x, y) is the position of p under the image plane coordinate system o-xy, (X S , Y S , Z S ) is the position of the center of projection S under the ENU coordinate system, f is the focal length, and the nine parameters (a1-c3) are the elements of the rotation matrix R (Wang, 2007).
The SIFT feature points of each image pair are matched using the approximate nearest neighbours algorithm.Then a fundamental matrix for the pair is robustly estimated using RANSAC.Finally, the matched feature points that are consistent with the recovered fundamental matrix are kept as tie points of the image pair.
The tie points generated by the image matching procedure are then used for relative orientation.The relative orientation begins with an initial image pair that has a large number of tie points, and proceeds in an incremental manner.We add a neighbouring camera that observes the largest number of already reconstructed 3D points, and initialize the new camera's extrinsic parameters using the PnP algorithm inside a RANSAC procedure.During the relative orientation, the interior and exterior orientations of the images are optimized by a bundle block adjustment procedure.The image orientations determined by the relative orientation are then transformed to the ENU coordinate system.The positions and orientations of the images under the ENU coordinate system are further optimized by a bundle block adjustment.
The dense matching is then carried out and the dense point cloud of the surveying area is generated.The normalized cross correlation coefficient is used as the similarity measure for dense matching.A Triangulated Irregular Network (TIN) is generated from the resulting dense point cloud and a raster DSM is interpolated.Finally, the orthomosaic is generated by orthorectification and orthomosaicing.The resulting DSM and orthomosaic are finally transformed backed to the geodetic coordinate system and output as GeoTIFF files.
To assess the influence of image enhancement on fully automatic UAV photogrammetry, the contrast of the images is firstly auto-adjusted using the software Photoshop.Then image matching, image orientation and surface reconstruction resulted from the original and the enhanced images are compared.
To assess the influence of image downsampling, the aerial images are downsampled using the free and open-source software ImageMagick.Then image matching and image orientation resulted from the original and the downsampled images are compared.
To assess the influence of image compression, the original images are compressed with lossless and lossy compression procedures.The lossy compression is carried out using the software Photoshop, and the lossless compression is carried out using the software jpegtran.The image matching and image orientation resulted from the original and the compressed images are compared.
To compare image matching results, the average number of extracted SIFT feature points and the ratio of the inlier matches are used as the measure.To compare the image orientation results, the accuracy of the estimated exterior orientation parameters is used as the measure.To compare the surface reconstruction results, the pixel-wise elevation difference is derived and used as the measure.

RESULTS AND DISCUSSION
Surveying area A is a basin and the elevation of the area is in the range of 70-650 meters above the sea level.The aerial images were taken by a Nikon D810 camera with a 35mm fixed focal lens during a flight on March 31, 2018.The flight height was about 1000 meters above the sea level.The forward/side overlapping of the images is about 80%.A total of 804 valid images were acquired.The average spatial resolution of the images is about 0.1m.The temperature of the area on that day was in the range of 9-18 degree Celsius.The reported air quality was at the level of light pollution.The reported PM2.5 and PM10 indices were 99 and 172 respectively.Figure 1 shows one of the original aerial images and the enhanced image.It can be seen that the original image is hazy, while its enhancement is much better in terms of definition and contrast.
Figure 1.The original aerial image and the enhanced image The resolution of the images is 7360 by 4912.The number of the extracted SIFT feature points from the enhanced images is 0.2% less than that from the original ones on average.However, the ratio of the inlier matches of the enhanced images is 2% more than that of the original ones on average.
No ground control points have been measured in surveying area A, therefore relative accuracy is used to measure the influence of image enhancement.The exterior orientations of the original images estimated by fully automatic POSsupported aerial triangulation are used as the ground truth.
Figure 2 shows the relative error distributions of the exterior orientations estimated from the enhanced images.The errors of the orientations exhibit normal distribution.The pixel-wise elevation differences are derived from the DSMs produced using the original and enhanced images.The elevation differences exhibit normal distribution.The mean of the elevation differences is 0.02m, which indicates systematic error is negligible.The maximal absolute value of the elevation differences is 209.378m.The standard deviation of the elevation differences is 0.494m, which is significant.The percentage of the elevation differences whose absolute values are below 1m is 56.28%.Figure 3 shows the distribution of these elevation differences.To assess the influence of image downsampling on the UAV photogrammetry, results of image matching and image orientation are compared between the original and the downsampled images.Data collected at surveying area B is used for the experiment.The rectangle-like surveying area is 6 km from east to west and 7 km from south to north (Figure 4).The elevation of the area is in the range of 200-1050 meters above the sea level.Most of the surveying area is covered by shrubs and trees.Seven ground control points were measured under the WGS84 coordinate system by using the static networked observation method.The absolute accuracy of the ground control points was at the level of decimetres.Figure 7 shows the positions of the ground control points.The accuracy of aerial triangulation on the control points after bundle block adjustment is given by Table 2.The planar accuracy is at the level of centimetres and the vertical accuracy is at the level of decimetres.The exterior orientations of the original images estimated by GCP-supported aerial triangulation are used as the ground truth for accuracy evaluation of the exterior orientations derived from the processed images.The original images are downsampled with a 1000-pixel step along the width direction and the aspect ratio of the images is kept.A total of six downsampled image sets are generated.The average number of SIFT feature points extracted from different resolution image sets are shown in Figure 8.It can be seen that the average number of extracted SIFT feature points is proportional to the image resolution.The higher the resolution, the more feature points will be extracted.
Figure 9 shows the average number of feature point matches of different resolution image sets.It is easy to see that the average number of feature point matches is also proportional to the image resolution.The relationship between the average ratio of inlier matches and the image resolution is shown in Figure 10.The average ratio of inlier matches is inversely proportional to the image resolution.The ratio drops from 98% to 34% when the image resolution increasing from 1000 pixel to full resolution.A larger proportion of the extracted feature points are inconsistent with the final fundamental matrix when matching the higher resolution images.The fewer feature points extracted from lower resolution images seem to be more stable and repetitive for image matching.
Figure 10.Average ratio of inlier matches of different resolution image sets The accuracy (RMSE) of the exterior orientations of different resolution image sets estimated by fully automatic POSsupported aerial triangulation is given by Table 3.The most accurate estimation of each orientation parameter is indicated in bold.It can be seen that there is no dominant relationship between the accuracy of the exterior orientations and the image resolution.The orientations derived from the lower resolution images may be more accurate than that from the full resolution images.To assess the influence of image compression, the aerial images of surveying area B are used in the experiment.The lossy compression is carried out using Photoshop, and a total of ten image sets are generated.The highest and lowest compression ratios are 7.66 and 1.24 respectively.jpegtran is used for the lossless compression, and the compression ratio is 1.08.
The average number of extracted SIFT feature points from the compressed and original images are shown in Figure 11.It can be seen that the average number of SIFT feature points is basically proportional to the compression ratio.The average number of SIFT feature points extracted from the original image sets is the same as that from the lossless compressed image sets.More feature points are extracted from the highly compressed images than from the original and lowly compressed images.The average ratio of inlier matches of the original and compressed image sets is shown in Figure 13.It can be seen that the average ratio of inlier matches is not influenced by the compression ratio when the images are lossless and lowly compressed.The average ratio of inlier matches of the original image sets is almost the same as that of the lossless compressed image sets.The average ratio of inlier matches drops dramatically to 18.45% when the compression ratio reaches 7.66.A larger proportion of the extracted feature points are inconsistent with the final fundamental matrix when matching the highly compressed images.
Figure 13.Average ratio of inlier matches of the original and compressed image sets The accuracy of the exterior orientations of the original and compressed image sets estimated by fully automatic POSsupported aerial triangulation is given by Table 4.It can be seen that the orientation accuracies of the losslessly and lossily compressed images are comparable to that of the original images.Figure 14 shows the accuracy of each exterior orientation.It seems that the influence of compression ratio on the accuracy of each orientation is different.There is no dominant relationship between the accuracy of Phi, X and the compression ratio.The compression ratio does not influence the accuracy of Kappa at all.Although the accuracies of Omega, Y, and Z decrease when the images are highly compressed, the changes of the accuracies are insignificant.Experimental results show that the influence of image enhancement on automatic UAV photogrammetry is significant.The orientations estimated from the original images are comparable to that from the enhanced ones.However, the elevation difference between DSMs reconstructed from the original and enhanced images is significant.Fully automatic UAV photogrammetry is partially influenced by image downsampling.The feature points extraction and image matching procedures are greatly influenced by image downsampling.However, the accuracies of the orientations are not influenced by image downsampling.Experimental results show that the orientations derived from the lower resolution images may be more accurate than that from the full resolution images.Image compression has little influence on automatic UAV photogrammetry.The orientation accuracies of the losslessly and lossily compressed images are comparable to that of the original images.Although image compression influences some of the orientations, the changes of their accuracies are insignificant.

Figure 2 .
Figure 2. Relative error distributions of the exterior orientations estimated from the enhanced images After normal distribution fitting, the mean and standard deviation of the errors of each exterior orientation is obtained.Table 1 lists statistics of these errors.It can be seen that the means of the errors of the three linear orientations are zero, and that of the angular orientations are almost zero.The standard deviations of the errors of the angular orientations are at the level of 0.001 degrees, and that of the linear orientations are at the level of centimeters.The maximal absolute errors of the angular orientations are at the level of 0.01 degrees, and that of the linear orientations are at the level of decimeters.The statistics of the relative errors indicate that the image orientation accuracy of the original images and the enhanced images are comparable.

Figure 3 .
Figure 3. Distribution of the elevation differences whose absolute values are below 1m

Figure 4 .
Figure 4. 3D visualization of surveying area B The aerial images were taken by a Nikon D810 camera with a 35mm fixed focal lens during a flight in autumn 2016.The flight height was about 1500 meters above the sea level.During the flight, the positions and orientations of the exposures were also recorded.The forward/side overlapping of the images is about 80%.A total of 1198 valid images were acquired.The average spatial resolution of the images is about 0.2m.Figure5shows one of the images captured.

Figure 5 .
Figure 5.One of the aerial images of surveying area B

Figure 6 .
Figure 6.Positions of the exposures and number of image observations of surveying area B

Figure 7 .
Figure 7. Positions of the ground control points of surveying area B

Figure 8 .
Figure 8.Average number of SIFT feature points extracted from different resolution image sets

Figure 9 .
Figure 9. Average number of feature point matches of different resolution image sets

Figure 11 .
Figure 11.Average number of SIFT feature points extracted from the original and compressed image setsThe average number of feature point matches of the original and compressed images is shown in Figure12.It can be seen that the average number of feature point matches is not correlated with the compression ratio.The average number of feature point matches of the original image sets is almost the same as that of the lossless compressed image sets.

Figure 12 .
Figure 12.Average number of feature point matches of the original and compressed image sets

Figure 14 .
Figure 14.Accuracy of each exterior orientation 4. CONCLUSIONS AND FUTURE WORK Image processing techniques are often used to improve the aerial image quality as well as the efficiency and effectiveness of photogrammetric data processing.The image processing techniques bring in uncertainties to UAV photogrammetry.In this work, influences of the image processing techniques including image enhancement, image downsampling and image compression on fully automatic UAV photogrammetry are investigated.

Table 1 .
Statistics of the errors of the exterior orientations estimated from the enhanced images

Table 2 .
Accuracy of aerial triangulation on GCPs Label X error (m) Y error (m) Z error (m) Total (m) Image (pix)

Table 3 .
The accuracy of the exterior orientations of different resolution image sets

Table 4 .
Accuracy of the exterior orientations of the original and compressed image sets