Aerial Photogrammetry Procedure Optimized for Micro Uav

This paper proposes the automatic aerial photogrammetry procedure optimized for Micro UAV that has ability of autonomous flight. The most important goal of our proposed method is the reducing the processing cost for fully automatic reconstruction of DSM from a large amount of image obtained from Micro UAV. For this goal, we have developed automatic corresponding point generation procedure using feature point tracking algorithm considering position and attitude information, which obtained from on-board GPS-IMU integrated on Micro UAV. In addition, we have developed the automatic exterior orientation and registration procedure from the automatic generated corresponding points on each image and position and attitude information from Micro UAV. Moreover, in order to reconstruct precise DSM, we have developed the area base matching process which considering edge information. In this paper, we describe processing flow of our automatic aerial photogrammetry. Moreover, the accuracy assessment is also described. Furthermore, some application of automatic reconstruction of DSM will be desired.


INTRODUCTION
Nowadays, due to the diffusion of low-cost GPS-IMU and also the progress of control technique of Micro UAV in past several years, the Low-cost Micro UAV system has been widely used as the useful platform in the application field of aerial photogrammetry such as agriculture, archaeology, traffic monitoring, and disaster area surveying (Remondino, 2011). The important ability of Micro UAV is the autonomous flight along previously planned waypoint. From this ability, fully automatic image data acquisition using digital still camera mounted on Micro UAV has becomes possible. Accordingly, a large amount of image data can be obtained for creating of precise orthophoto and Digital Surface Model (DSM) of object area. Additionally, the processing of aerial triangulation using a large amount of image data has to be performed automatically. The Structure from Motion (SfM) has been widely used as an efficient technique of 3D reconstruction from numerous image data. The modern SfM consists of efficient feature point description for automatic detection of corresponding points and bundle adjustment for numerous image data. The breakthrough of efficient feature point description has been given by SIFT and its improvement (Lowe, 2004). The good implementations of bundle adjustment using numerous image data have been given as Sparse Bundle Adjustment (SBA) and its improvement (Lourakis, 2009). Therefore, many applications of photogrammetry using Micro UAV have achieved fully automatic processing by using SfM approach. Nevertheless, the processing cost of feature point detection and matching and also bundle adjustment is still important problem. Moreover, if the information of global coordinate is not given, the results of SfM are obtained in model space that has arbitrary position, scale and rotation. Another important problem is that estimated parameters generally include the accumulative error based on feature point matching results. In order to resolve these problems, utilization of information from on-board GPS-IMU is one solution. In the case of standard application of automatic aerial triangulation, direct geo-referencing approach using GPS and IMU on UAV is often considered. However, GPS-IMU integrated on low-cost Micro UAV has not enough accuracy because GPS unit on Micro UAV only has ability of single positioning from limitation of cost. Thus, exterior orientation of SfM using Micro UAV has to consider ground control points (GCP) and GPS-IMU information simultaneously. For these problems mentioned above, we have been concentrating to investigate the Structure from Motion (SfM) technique using GPS in recent years (Anai et al., 2010). In these investigations, we have proposed the robust feature point tracking method based on the "Orientation Code" (OC) image processing (Ullah, 2001, Takauji 2005. Also, bundle adjustment method for video image that uses both SfM technique and GPS data with considering error in GPS observation. Additionally, the application for Micro UAV using these techniques has been reported (Anai et al., 2012). In this paper, we describe about improvements of our method, which include robust image matching supported by low accuracy GPS-IMU information and automatic exterior orientation process. On the other hand, In order to create precise DSM robustly, we propose extended Edge TIN-LSM method which integrates edges and which is able to cope with differences in right and left image shape, brightness changes and occlusions (Kochi et al., 2012). This paper is structured as follows. Section 2 describes our processing flow of exterior orientation procedure. Section 3 describes processing flow of creating DSM using proposed TIN-LSM Method. Moreover, accuracy assessments for some application test are described in Section4. The conclusion about our proposed method is described in Section 5.

AERIAL PHOTOGRAMMETRY PROCEDURE FOR LOW-COST MICRO UAV
This section describes about processing flow of our proposed aerial photogrammetry procedure. The most important problem of aerial photogrammetry using low-cost Micro UAV is low accuracy of GPS-IMU. The GPS-IMU of low-cost Micro UAV is enough for the navigation of Micro UAV in local area. However, the observation of GPS only has single positioning mode and the time synchronization between still image and GPS-IMU is incomplete. Furthermore, the accuracy information of GPS-IMU is not provided correctly. From these backgrounds, the proposed method is performed as shown in Figure 1. At the first, in order to obtain the dense corresponding pass points between each still image, feature point tracking process using common feature points is performed by the OC Image Processing supported by GPS-IMU information. As the next step, generation of tie point between each stereo model and flight line is performed using the result of feature point tracking process. In this process, more precise matching of pass points and the generation of tie points between each stereo model is performed by Least Square Matching. As the next step, bundles adjustment using pass points and tie points is performed in model space that has arbitrary position and scale.
In the next step, we perform registration procedure for reconstruction of 3D model in global coordinate system. In final step, accurate DSM is generated by the Extended Edge TIN-LSM method.

OC Image Processing
The feature point detection and matching of our proposed method is based on the Orientation Code (OC Ullah and Kaneko, 2001) Image Processing. The most unique point of OC image processing is that the coded images are used instead of the original gray-scale images. The OC means a quantization of the maximal intensity change direction around interest pixel. The OC of each pixel is defined as following equation, In this equation, ΔIx and ΔIy show horizontal and vertical gradient of pixel (x, y). N is the quantization level of direction.
The "γ" is the threshold value for the suppressing of small gradient pixel. Therefore, OC is independent from influence of intensity changing. In order to extract common feature points from OC image, we use Orientation Code Richness (OCR, Takauji, 2005) that extracts the pixels that have high entropy of OC. The entropy of the local limited area of the pixel size M-by-M region at the interest pixel (x, y) is calculated as follows, Where hxy(i) (i =0,1,,,,,,N-1) means histogram of OC of M-by-M pixel size region. When each OC goes with uniform distribution Pxy(i)=1/N, the maximum value of entropy Emax is log2N. Consequently, the richness Rxy is defined as Where, the threshold value αe is defined to remove low entropy area. The matching process using OC image (OCM, Ullah, 2001) is similar to other simple image based template matching. The difference between a template image patch from OC image (Ot) and search OC image (Oi) is defined as following equation, where D = difference between Ot and Oi M = Size of template image patch d = difference between a pixel and b pixel Thus, the robust template matching can be achieved in low processing cost by using OCM.

Feature Tracking Process
In order to perform automatic detection of dense corresponding pass points from discontinuous high resolution still images without using coded target or manual operation, the key point descriptor such as SIFT (D. G. Lowe. 2004.) are widely used. The automatic process of the creating dense pass points using these feature point descriptor is performed as brute-force processing for each image. This process occupies a large part of SfM procedure. In the case of Micro UAV, the still image acquisition is performed along planned waypoints considering altitude, image resolution and image overlap generally .Therefore, utilization of position and attitude information from GPS-IMU is considered to reduce the cost of corresponding point detection process. However, the GPS-IMU integrated on low-cost Micro UAV often includes error observation. By the way, the sequence of image acquisition is independent of positioning error of GPS-IMU. Moreover, we already proposed the feature point tracking algorithm for video image based on OC image processing in previous investigation (Anai, 2010(Anai, , 2012. In this investigation, we have improved the feature point tracking for the still images based on OC image processing by using GPS-IMU information. Figure 2 shows the processing flow of proposed feature point tracking procedure. The synchronization between still images and GPS-IMU information is first work of this process. Generally, synchronization is incomplete due to the accuracy of time data and error of GPS-IMU observation. Nevertheless, the position and attitude data from GPS-IMU is useful to support the feature point tracking process. In our process, the GPS-IMU information is used for estimating homography between two images along time sequence. At the first image of time sequence, feature point detection using OC Image processing is performed. The feature points are obtained as the local peak point of OCR estimation. Figure  3 show the example of Feature point detection. As the next step, initial value of homography between two images is estimated from GPS-IMU information. If reference plane of object area is assumed, the initial value of homography is obtained from projection and back projection steps. The OCM of each feature points starts from estimated homography with small search area. The search area of OCM is enlarged until the enough number of feature points obtains good result. When the enough number of feature points has good result, the homography between two images is estimated using the good result of OCM. Thus, the tracking procedure supported by GPS-IMU information can start the matching process from good assumption. Therefore, the cost of matching is reduced. The homography is estimated using robust regression Mestimator. The homography between two images is estimated by following equation, xi,j, yi,j: j-th point image coordinate on i-th frame b1~b8: homography parameters The M-estimator is defined as the least-square method that the each weight coefficient is decided to suppress the outlier. The influence function ψ of this investigation is Tukey's Biweight that defined as follows. The constant C in equation (7) is selected as 5~9. In our process, we use 6. Therefore, the parameters b1 ~ b8 are obtained by the weighted least square method using w eff without influence of outlier. These processing is repeated until the residual error of feature point converge. Finally, we estimate the epipolar constraint between two images by using eight-point algorithm (Hartley 1997) with Mestimator. The eight-point algorithm estimates the fundamental matrix F lineally by using at least 8 points. Therefore, the parameters of F are obtained using the least square method using w eff . Concretely, Here, the |z| in equation 7 is distance from epipolar.
These processing is repeated until a residual error converges. From this procedure, we obtain the candidate of pass points between two images without outlier. Figure 4 shows the example of tracking result. These procedures are performed from first image to last image along time sequences.  Figure 5 shows the processing flow of tie point matching and Bundle adjustment. From the tracking process mentioned above, the stereo pairs are selected considering the enough baseline length and the overlap. Moreover, precise matching for candidate pass points is performed using Least Square Matching (LSM). In order to perform adjusting between each stereo pair and flight line, tie point matching using candidate pass points is performed. At the first, exterior orientation only using candidate pass points is performed by bundle adjustment. The candidate pass points that have large residual of image coordinate are removed in this step. As the result of this exterior orientation, 3D coordinate of each candidate pass points and exterior orientation parameter for each still image are obtained in arbitrarily position and scale. From this 3D geometry information, the 3D coordinate of each candidate pass point can perform back projection to 2D image coordinate on each still image. Therefore, the tie point matching is performed in the ROI that located around back projection 2D image coordinate of candidate pass points on still image by using LSM. Finally, bundle adjustment is computed again using pass points and tie points. In order to obtain the 3D coordinate on global coordinated, the registration process has to be done. When we can use the GPS unit that has enough accuracy such as RTK-GPS and good time synchronization system, we can use this positioning information for registration directly. However, as mentioned above, utilization of RTK-GPS is difficult from limit of cost. Therefore, in the case of low-cost Micro UAV, we usually use the bundle adjustment with the minimum number of GCP for accurate registration and adjustment of 3D model. The robust bundle adjustment using M-estimator is performed by minimizing the following error function.

Tie Point Matching and Bundle Adjustment
From this procedure, exterior orientation parameter of each still image and 3D coordinate of each common feature point are obtained in global coordinate system, and also the influence of error observation such as mismatching points or outlier positioning of GPS at waypoint is suppressed by this procedure.

RECONSTRUCTION OF DSM
In order to reconstruct the precise DSM, the dense point matching performs in each stereo image. We use the Extended Edge TIN-LSM method which considering edge information (Kochi et al., 2012). The Extended Edge TIN-LSM method is robust stereo-matching method that integrates edges and which able to cope with differences in right and left image shape, brightness changes and occlusions. The unique point of our Extended Edge TIN-LSM method is that the precise edge information can be obtained regardless of the resolution of DSM. Therefore, the cost of processing for DSM can be reduced. Figure 6 shows the processing flow of reconstruction of DSM. At the first, In order to obtain processing area, overlap area of all images is checked. From this procedure, the region of interest (ROI) of processing area is determined. As the next step, stereo pairs are sorted in order of distance from the center of ROI. The stereo matching process is performed from the center of ROI towards outside. In order to reduce the cost of stereo matching, the Extended Edge TIN-LSM method for each stereo pair is performed only for the new area of DSM. of pass points obtained in previous bundle adjustment process. 4. The production of initial TIN is performed using these feature points of edge and pass points. 5. Moreover, the grid point generation using LSM is performed as coarse to fine process. The LSM of grid points in each step of coarse to fine procedure is constrained by TIN. In addition, the TIN is updated using grid matching results in each step. 6. The new generated DSM area is merged to previously generated DSM.
These procedures are repeated for all stereo pairs. Finally, whole DSM of ROI is obtained.

EXPERIMENT
We describe about the experiment of our proposed method in this section. In order to evaluate the effectiveness of our proposed method, we have performed some experiments in test field. Details of experiment are as follows. Figure 7 shows the Micro UAV of our experiments and Table 1 shows the specification of our Micro UAV.  Figure 8 shows the reconstruction result of DSM of our proposed method in first test site. This test site includes terrain and buildings. Our purpose was reconstruction shape of building and terrain simultaneously. The width and height of this test site was about 82 m x 113m and 156 images were used for automatic procedure. The automatic processing time (not include GCP measurement) was 1 hour 25 minutes. In this case, Grid pitch of DSM was 50 cm. However, the edge brake line of buildings and steep edge of terrain have reconstructed as shown in Figure 8. Figure 9 shows the orthoimage obtained from our proposed method. In order to evaluate the accuracy of our method we construct the reference points by Total Station and RTK-GPS survey.

Reconstruction of Terrain and Buildings
The six reference points are used for GCP and other eight reference points are used for checkpoints. The standard deviation of residual was 13mm in plane and 30 mm in height. The image resolution of Digital still camera of this experiment was 22mm at 60 m flight height. Thus, it is concluded that the result of this experiment has enough accuracy.

CONCLUSION
We have proposed the automatic photogrammetry procedure optimized for low-cost Micro UAV. The feature point tracking procedure based on OC image processing have improved by using position and attitude information from low-cost GPS-IMU information. Therefore, the estimation of homography between each image has started from good assumption. This improvement reduces the processing cost, and improves the stability of tracking procedure. Moreover, the extended bundle adjustment using both GCP and low accuracy GPS has been proposed. In order to obtain the precise DSM, we have performed the application of the Extended Edge TIN-LSM method. The good ability for steep edge reconstruction of the Extended Edge TIN-LSM method has been described through the experiment. Additionally, the accuracy assessment of our proposed method has been described. From these investigation, it is concluded that the proposed method have ability of creating the precise DSM from images obtained from Micro UAV. However, some problems remain for more fully automatic procedure. For example, automatic measurement of GCP image coordinate is still important issue.