CO-REGISTRATION AIRBORNE LIDAR POINT CLOUD DATA AND SYNCHRONOUS DIGITAL IMAGE REGISTRATION BASED ON COMBINED ADJUSTMENT

Aim at the problem of co-registration airborne laser point cloud data with the synchronous digital image, this paper proposed a registration method based on combined adjustment. By integrating tie point, point cloud data with elevation constraint pseudo observations, using the principle of least-squares adjustment to solve the corrections of exterior orientation elements of each image, high-precision registration results can be obtained. In order to ensure the reliability of the tie point, and the effectiveness of pseudo observations, this paper proposed a point cloud data constrain SIFT matching and optimizing method, can ensure that the tie points are located on flat terrain area. Experiments with the airborne laser point cloud data and its synchronous digital image, there are about 43 pixels error in image space using the original POS data. If only considering the bore-sight of POS system, there are still 1.3 pixels error in image space. The proposed method regards the corrections of the exterior orientation elements of each image as unknowns and the errors are reduced to 0.15 pixels.


INTRODUCTION
Nowadays, Light Detection and Ranging (LiDAR ) system can get high accuracy height information rapidly, however, LiDAR data is lack of semantic information compared to images which have higher precision plane accuracy and rich texture information (Baltsavias, 1999).So in order to effectively utilize these two kinds of data, part of airborne LiDAR system has been recently equipped with camera, which makes it capable to obtain point cloud data and high resolution aerial images simultaneously.Through the camera calibration parameters and POS (Position and Orientation System) system, the internal and exterior orientation elements at the exposure time of the images can be calculated and served as parameters for subsequent photogrammetric processing.However, due to installation error of camera or some other factors during shooting, it is hardly to measure the precise relative orientation between POS system and the camera, which leads to significant registration error between point cloud data and synchronized image and affect subsequent processing of orthorectification, and etc (Zhang et al., 2014).In order to make good use of these two kinds of complementary data sources, it is necessary to co-register airborne laser point cloud data with synchronous digital image.Existing methods can be roughly divided into two categories: one kind is the direct registration method based on feature matching (González-Aguilera et al., 2009;Renaudin et al., 2011); the other is by generating 3D point cloud from images, and then register two types of point cloud by Iterative Closest Points method (Zhao et al., 2005;Habib et al., 2006) .The former one depends on automatic feature matching, which is still a difficult problem; the latter one is less efficiency due to the time-consuming photogrammetry processing for producing 3D point cloud.Different from these two kinds of methods, Zhong et al. (2011) and Chen et al. (2012) proposed bore-sight calibration method to determine the orientation angle of the synchronous camera to realize the co-registration of the two kinds of data without ground control points.However, the human-computer interaction is needed for corresponding point's selection and it has limited precision since it takes only angle error into account.Aim at the above problems, this paper proposed a method of co-registration airborne laser point cloud data with the synchronous digital image, which dose not only consider the placement angle error of POS system, but also consider the GPS offset.

Overview of the workflow
The workflow of airborne LiDAR point cloud data and synchronous digital image co-registration based on combined adjustment is shown in Figure .1, it is composed of 3 steps.1) Point cloud constrained SIFT matching (Lowe, 2014): firstly, the corresponding area on image is obtained via the initial value of exterior orientation parameters (EOPs), then the corresponding points in the areas are matched by modified SIFT matching method, and thereafter RANSAC algorithm is adopted to remove possible mismatch (Fischler and Bolles, 1981).
2) Correspondence optimization: optimizing the distribution of the corresponding points to satisfy the demand of combined adjustment.
3) Combined adjustment: the error equation is formulated and the parameters are solved iteratively by least square adjustment method, consequently, the LiDAR point cloud is co-registrated with the synchronous digital image.

Combined adjustment model
The corresponding relationship between LiDAR point cloud and images can be described by co-linearity equation as equation ( 1): = rotation matrix derived by angle element , ,    of the EOPs.
Equation ( 1) can be expanded using Taylor series and linearized to error equation as equation ( 2).  3) are included in the error equations of the combined adjustment.
Where the first equation is relevant with tie points Least square adjustment is used to calculate the correction iteratively after the formulating equation (3).

Elevation constraint
The elevation of tie point object coordinates calculated by forward spatial intersection via collinearity equation will close to the DSM elevation interpolated by LiDAR point cloud, when the EOPs provided by POS system do not contain errors.However, errors generally existed in the EOPs which cause a large difference between the forward intersected elevation and LiDAR point cloud interpolated elevation.Therefore, the elevation difference is selected as the pseudo observation to eliminate the difference between the forward intersected elevation and interpolated elevation by combined adjustment.Zinter by the surrounding points: 0 Z Zinter   (4 ) Due to the errors in the EOPs of digital images, the equation ( 4) can be modified to the error equation: (5 ) Equation ( 5) is the pseudo observation equation constrained by point cloud elevation and 0 Z is the elevation value of the tie point calculated by forward spatial intersection using initial EOPs.
The interpolated elevation accuracy will be declined when the tie point locates at the edge of buildings or vegetation area.Due to the spatial intersected object coordinate of tie points and LiDAR point interpolated elevations are selected as the pseudo observation, only the corresponding points located in flat area are employed as the tie points to ensure that the pseudo observation can correctly constrain the combined adjustment.

Point cloud supported modified SIFT matching and optimization
The extraction and optimization of tie points are the key problem of the proposed method.However, collecting corresponding points manually are time-consuming.Traditional SIFT matching method is time-consuming and the distribution is affected by image texture, therefore it is not appropriate for our demand and in this research, SIFT matching is constrained by the point cloud.The steps are as follows: 1) LiDAR points are projected onto the images and the overlapping area is obtained by the projected points to produce a coarse disparity map.
2) SIFT points are extracted and matched in the overlapping area constrained by the derived disparity map and double-way matching method is used to ensure the reliability of the corresponding points.
3) Possible mismatch are removal based on relative orientation incorporated with RANSAC approach.Although a large number of corresponding points can be obtained by SIFT extraction and matching and the double-way matching has removed most of the error matches, a small number of errors maybe still remained.Therefore RANSAC algorithm is employed to calculate the fundamental matrix and then the remained errors are removed by the geometric constraints of the stereo image pair ( Wu et al. 2011).In this research, to ensure elevation constraint is available during the iteration of adjustment.Only the corresponding points that has certain number of Lidar points in a circular region with radius R (set to 50 pixels in this paper) and the elevation standard deviation of the these point is less than a given threshold  (  is set 0.1 meters) are retained.Due to the uncertainty of the distribution of the systematic error, the distribution of the neighbourhood is checked, and only the corresponding points whose 4 quadrant have at least one LiDAR point is kept.Through above steps, appropriate corresponding points are screened out and thereafter, average grid is set and one corresponding point of each grid is selected as the tie point used for the combined adjustment.

Weight determination
In equation ( 3), the tie points can be considered as equal weight observation, therefore 1 P is set as a unit matrix.The second equation has mandatory constraint function, and it can make the Z value of the forward intersected 3 dimensional coordinates constrained around the true elevation value, and through a number of iterations, it can be further constrained, finally, forward intersected elevation value will be very close to the true value.In this research, the weight 2 P of equation 2 is 2 orders of magnitude greater than the weight 1 P of equation 1.

Experimental data description
For validation of the proposed method, airborne laser point cloud data shown in Figure 3

Results of tie points extraction
For the stereo images as shown in   In order to verify the registration results, the point cloud data shown in Figure 4. (b) (c) (d) is back-projected to corresponding image respectively using the exterior orientation elements before and after co-registration.As shown in Figure 7. the back-projection image points are marked in red and green respectively for before and after co-registration.It is shown that there is large misalignment before data registration.While after co-registration, the projection region marked in green are aligned strictly in both left and right images, which shows that the method is capable of registering the two kinds of data.(the projection difference of the point generated by forward spatial intersection is projected to the left and right images) are calculated.After that, the maximum and the minimum value, the root of mean square error is calculated.The result is shown in Table 1, the proposed method is compared with the traditional bore-sight calibration method only considering rotation angels, and there is about 43 pixels error in image space using the initial POS data.If only considering the boresight of POS system, there are still 1.3 pixels errors in image space.The proposed method regards the corrections of the exterior orientation elements of each image as unknowns and the error is reduced to 0.15 pixels.The experiment results reveal that the proposed method can register the airborne point cloud and the synchronous digital image with higher accuracy.

CONCLUSIONS
Aim at the problem of co-registration airborne laser point cloud data with the synchronous digital image, this paper proposed a registration method based on combined adjustment.For the purpose that the corresponding points provide better constraint and to ensure the reliability of SIFT matching, an algorithm point cloud supported modified matching is proposed.At the same time, through analysing of the height distribution of the point cloud around the corresponding SIFT feature points, keeping the corresponding points only if there are enough points with similar height distributing on their neighbours, so the points located on plants or the edge of building which has a big interference could be filtered out.With the corresponding points optimized automatically, basing on the collinearity equation and using LiDAR point cloud data as a constraint, the method of combined adjustment is capable of the registration of the airborne laser scanning data and the synchronized digital images.The experiment shows that the method could achieve higher accuracy than the method only considering the rotation angle.Due to the limitation of the experimental data, only the experiment that the registration of the airborne laser scanning data and the corresponding synchronized digital image pair are conducted in this paper.The model in this paper would be extended to multi-view images.

Figure. 1
Figure.1 Workflow of the proposed co-registration method T x d d d dXs dYs dZs dX dY dZ     , , d d d    = correction of angle element , , dXs dYs dZs = correction of line element , , dX dY dZ = correction of object coordinates of tie points Two kinds of error equations as equation ( of 3D coordinate of tie point The second equation is the error equation of the pseudo observation equations relevant with elevation.3 x = elevation correction of tie points.

Figure. 2
Figure.2 Illustration of elevation constraint (a) and the corresponding stereo pairs shown in the Figure 3(b) (c) are used for experiments.There are 16,475,402 points and about 10/m 2 in the point cloud data; the camera parameters of digital aerial images are calibrated rigorously, so the distortion error correction and the translation of the principal points of photograph have been removed.The size of the images is 9000 pixels×6732 pixels, focal length is 93.071 mm.For comparison, this paper also implements the registration method only taking the rotation angle error into consideration.

Figure
Figure.3 Experiment data Figure 4.(b) (c), 11413 and 11686 feature points are extracted from left and right image respectively by using SIFT feature extraction algorithm.1846 matches are obtained by using the proposed method.After RANSAC approach, only 1348 corresponding points are kept as shown in the Figure 5.

Figure 5 .
Figure 5. SIFT matching result After the distribution optimization of corresponding points, the correct rate of corresponding points is 100% as shown in Figure 6.It can be seen that all the corresponding points are distributed in flat area.

Figure
Figure 6.Optimization result 3.3 Registration results Figure.7 Back-projection results of before and after registration From above figures, it is obvious that the point cloud data and images with an obvious matching deviation turn into accurate registration after co-registration based on combined adjustment.The projection location of point cloud data from the inaccurate area to the accurate area, the accuracy is greatly improved.For quantitative validation, 17 identical point pairs between two images are selected as check points by the way of humancomputer interaction under the guidance of airborne laser point cloud.Then z  (the difference between the height of point generated by forward spatial intersection and the point generated by the interpolation from airborne point cloud), s 

Table 1 .
Comparison of back-projection of 3D points (pxs is abbreviation of pixels)