A MULTI-CORE PARALLEL MOSAIC ALORITHM FOR MULTI-VIEW UAV IMAGES

As the spread of the error and accumulation often lead to distortion or failure of image mosaic during the multi-view UAV(Unmanned Aerial Vehicle) images stitching. In this paper, to solve the problem we propose a mosaic strategy to construct a mosaic ring and multi-level grouping parallel acceleration as an auxiliary. First, the input images will be divided into several groups, each group in the ring way to stitch. Then, use SIFT for matching, RANSAC to remove the wrong matching points. And then, calculate the perspective transformation matrix. Finally weaken the error by using the adjustment equation. All these steps run between different groups at the same time. By using real UAV images, the experiment results show that this method can effectively reduce the influence of accumulative error, improve the precision of mosaic and reduce the mosaic time by 60%. The proposed method can be used as one of the effective ways to minimize the accumulative error. * Corresponding author Fund Project: National Nature Science Foundation of China (40771178)


INTRODUCTION
According to the low cost, high flexibility and less limited by weather and the other characteristics, UAV has been an important way of acquiring data in measuring, GIS and remote sensing.So multi-view UAV image mosaic is also a research hotspot.One of the most influential factors for mosaic is the existence of accumulated errors.By its type it can be divided into two types.One is caused by symmetry error and the other is asymmetric error.To solve the problem, the researchers have proposed the use of satellite images with geographic information as a reference image for image mosaic (Keall, 2004); transformation of the benchmark map of the mosaic strategy (Li, 2008).The above method can effectively improve the quality of mosaic results under certain circumstances, but the treatment of accumulative error is not very satisfied.
Because there are many uncontrolled factors caused error accumulation, this kind of situation makes analysis and elimination the accumulative error of great difficulty, so assuming that the position error between adjacent images is the most important factor (Li, 2008).
In the paper, we proposed a multi-core parallel ring mosaic algorithm for multi-view UAV to illustrate problem above.The mosaic process is divided into several levels, which consists of multiple sets of adjacent images, and then two adjacent images in each group will be cut into four sub-images with overlap areas, between the two images using the feature-based image mosaic algorithm, and then stitch the four images to ring successively, then use the adjustment model of the image transformation model to eliminate the error, after these the model will be re-brought into the mosaic process to complete the mosaic before the real stitch of the two images.Each level of mosaic will be finished by the above-mentioned mosaic algorithm completed in parallel.Through the experiment, the method can effectively reduce the impact of accumulative error, and greatly reduce the operation time.

THE BASIC THEORIES OF IMAGE MOSAIC
The feature-based image mosaic method described above includes the following three processes.The SIFT feature is extracted from the two input images and then perform the image matching; use random sampling consistency (RANSAC) for removal of mismatch.;The perspective transformation matrix is used as the relationship between the images.Finally, use one of image fusion methods to smooth stitch edge and improve visual effects.

Image Matching and Elimination of Mismatching
SIFT is a well-known operator in the field of computer vision and can be used for pattern recognition and image matching.
The operator can find the important local information of the feature points accurately to determine the matching relation of the feature points, and finally reach the effect of prevent from rotation, translation and scaling to find the matching feature points accurately (Lowe, 2004).The key idea of the algorithm is to detect in the scale space first and then locate the key point accurately and determine its main direction to make it invariant; finally establish the descriptor of the key points.However, this cannot still guarantee that all the matching points are correct, but also need the elimination process.
RANSAC is an abbreviation for random sampling consistency.
It is a robust model parameter estimation algorithm.Since the input matching point data contains the wrong pair of points, unlike the least squares method, the model parameters calculated by the least squares method are the smallest sum of the distance from all points, but RANSAC can remove the wrong points, calculate the parameters that are appropriate for proper matching points.Even if in the case of the number of wrong pairs is large, RANSAC is still valid (Fischler, 1987).The calculation process of RANSAC is as follows: Firstly, extract the points randomly to calculate the model.After calculating the model parameters, the point set is divided into inner and outer points.If the number of points is enough, the model parameters are correct, so that recalculate the model that satisfied all the inner points.The process will be repeated a certain number of times to select the optimal model parameters, finally use precision to evaluate the quality of the model.

Image Transformation
An important step in image mosaic is to determine the transformations between two images, such as affine transformation and perspective transformation.Affine transformation can describe the rotation, scaling, translation between images and is "parallelism", and in addition perspective transformation can also describe the vertical sweep and horizontal sweep and other movements, the transformation is not "parallelism ".Affine transformation can be considered as a special case of perspective transformation.(Zhao, 2006).As the degree of freedom of perspective matrix is eight, so at least four pairs of points needed to calculate the parameters.

Image Fusion
In image mosaic, the problem of which is closely related to image transformation is the choice of image fusion.Based on the two images to be stitched are of better registration, the information of two images are integrated into the same image for a comprehensive expression, can not only reflect the characteristics of the two images but also reduce the redundancy of the data.Image fusion algorithm directly affects the visual effects of synthetic images.At present, scholars around the word put forward a lot of image fusion methods such as direct average method, linear weighted transition method and multi-resolution method (Burt, 1983).In order to better eliminate the lighting problem during shooting, reduce the ghosting produced by geometric transformation and reduce the discontinuity and mutation of the color at the mosaic edge, the paper uses the linear weighted transition method.
The basic idea of the linear weighted transition is similar to the direct averaging method, but the pixel values of the overlapping regions are not simply averaged but weighted averaged.Form like： 12 ( , ) ( , ) (1 ) ( , ) g x y ag x y a g x y    Where (x, y) means coordinate in image, g(x, y) means pixel value after fusion, g 1 (x, y), g 2 (x, y) means two images to be fused, a is weight parameter, its value between (0,1) (Yang, 2007).In theory for each image the weight of the middle part is higher than the weight of the edge.

MULTI-LEVEL PARALLEL CLOSURE MOSAIC ALGORITHM FOR MULTI-VIEW IMAGE
In the multi-vision UAV image mosaic, the most influence factor is accumulative error, by type it can be divided into symmetry errors and asymmetry errors.The relevant scholars have proposed that the accumulative error can be halved by divided between the adjacent images to eliminate the influence of it, and is most useful to asymmetric error even the precision can be doubled (Tang, 2003).The paper proposes a multi-core parallel ring mosaic algorithm designed to reduce the accumulative error while using multi-core technology to process algorithm parallel to reduce the operation time.

Multi-level Grouping and Parallelism
Since accumulative error can be scattered to effectively weaken its influence on the final result, so that the multi-level grouping parallel technology is proposed.OpenMP interface is a multiprocessor programming model for memory sharing model and is suitable for this case.For n pieces of input images, it can be divided into [log 2 n]+1levels, where [x] means rounding.
Here it is assuming that the input image is eight, three levels of mosaic, which also means eight input images will be stitched into four images and then mosaic by the same way, in the end stitched into the output image.The idea of the algorithm is that each level is divided into several groups, each group is parallel.
Between levels are serial.Because the input image of the next level is the output image of the previous level.The principle of multi-level and parallelism mosaic is shown in Figure 1.Where function Mosaic2img(Img2, Img1) executes the mosaic of Img1 and Img2, then return the result of it, each omp Sections describes a level, different Sections are serial which means when the execution of former Sections is done, the next Sections shall begin to work.In each Sections contains a number of sections, each section corresponds to a thread call Mosaic2img (A, B) to perform image mosaic.Different sections in Sections are parallel relationships.This will combine the idea of multi-level grouping and parallel computing to speed up the operation of the algorithm.

Mosaic Ring Composition Method
For the traditional method that mismatching still exists and there is no consideration of influence of errors, at the time the parameters of H matrix are not correct.Based on the characteristics of UAV images and research of relevant scholars, a ring mosaic method is proposed to reduce the influence of accumulative errors.
Since the operations done by H on the entire image are the same, the closed error calculated by using the matching pairs is the same for each point on the image.Since the points from outside the overlapping area is an external point, the H is correct if the above relation H is satisfied; otherwise, the closure error can be obtained and eliminated.Assuming that the two images to be stitched are Img1 and Img2, the algorithm steps are as follows: (1) Clip Image1 and Imag2 in to four sub-images: Img1U,Img1D and Img2U,Img2D.The method of clipping is shown as Figure 2. (2) Mosaic the four sub-images together with the order of the clockwise.Determining the perspective transformation matrix H D of Img2D and Img1D, the output is ImgD.By the same way H UD is the matrix of ImgD and Img1U, the result is ImgLUD.The relation between ImgLUD and Img2U is H U , the output is A.
(3) In Img2D with the non-overlapping region from Img2U, to find a point P with coordinate (x, y), the coordinate of Img2 is (x', y').Because they are the same point in Img2 itself, and the process of cutting Img2 into sub-images is known, the following relationship exists: where (4) By using formula (1) to establish the adjustment equation and linearize it, the following formula can be obtained: Where V x , V y represents the correction number for x and y; △x 0 , △y 0 Indicates the initial value of the closed error, △h11, △h12, △h13, △h21,……△h32 indicate the required parameters.
For all of the matching points there are:

h x h y h b h x h y h c h x h y
The least squares solution is: Where P is the weight matrix, let forty pairs of points involved in the calculation is of unit weight, each time the number of points involved in the calculation of H is (5) using H new as perspective transformation matrix to finish the mosaic.
Because of the mosaic method, on the one hand, the accumulative error can be effectively distributed between adjacent images, and the adjustment method is added to ensure the precision of stitching.On the other hand, because of the introduction of multi-core parallel computation, operation time is greatly shortened, although the algorithm cannot guarantee the real-time processing, but compared to single-core computing the running speed has been greatly improved.Figure 3 shows the overall process of this algorithm.
Figure 3. Process of mosaic operation

EXPERIMENT AND RESULT ANALYSIS
The experimental images are several multi-view images taken by a UAV from China.The height of the shooting is slightly changed.However, due to the cross wind, the attitude of the aircraft is not stable enough.In order to prove the effectiveness and advantages of the algorithm, the same input images are used to compare the traditional mosaic method and the proposed algorithm.
For all eight input images are shown in Figure 4.By using the algorithm, Figure 4

CONCLUSION
It can be seen from the above experiments that the multi-image multi-level parallel mosaic algorithm proposed in this paper can effectively distribute the accumulative error between the adjacent images and avoid the accumulation of the accumulated errors in the multi-view image mosaic.The adjustment model weakens the error to ensure the accuracy of the final mosaic results, while the parallel acceleration of the introduction can increase the speed and reduce the time cost.However, due to the huge amount of calculation, in daily use we should be based on the need of accuracy of mosaic and time to make the choice.The proposed method also provides a reference for the associated multi-view image mosaic.

Figure 1 .
Figure 1.Principle of multi-level and parallelism

Figure 2 .
Figure 2. Method of clipping sub-images

Table 1 .
Image transformation parameters

Table 2 .
It can be seen that after the acceleration, the operation time has reduced by 40%.

Table 2 .
Operation time comparison