A NEWMETHOD OF FAST REGISTRATION OF UNMANNED AERIAL VEHICLE REMOTE SENSING IMAGES BASED-ON AN IMPROVED SURF ALGORITHM

Remote sensing system fitted on UAV (Unmanned Aerial Vehicle) can obtain clear images and high-resolution aerial photographs. It has advantages of flexibility, convenience and ability to work full-time. However, there are some problems of UAV image such as small coverage area, large number, irregular overlap, etc. How to obtain a large regional map quickly becomes a major obstacle to UAV remote sensing application. In this paper, a new method of fast registration of UAV remote sensing images was proposed to meet the needs of practical application. This paper used Progressive Sample Consensus (PROSAC) algorithm to improve the matching accuracy by removed a large number of mismatching point pairs of remote sensing image registration based-on SURF (Speed Up Robust Feature) algorithm, and GPU (Graphic Processing Unit) was also used to accelerate the speed of improved SURF algorithm. Finally, geometric verification was used to achieve mosaic accuracy in survey area. The number of feature points obtained by using improved SURF based-on PROSAC algorithm was only 9.5% than that of SURF algorithm. Moreover, the accuracy rate of improved method was about 99.7%, while the accuracy rate of improved SURF algorithm was increased by 8% than SURF algorithm. Moreover, the improved running time of SURFGPU algorithm for UAV remote sensing image registration was a speed of around 16 times than SURF algorithm, and the image matching time had reached millisecond level. Thus, improved SURF algorithm had better matching accuracy and executing speed to meet the requirements of real-time and robustness in UAV remote sensing image registration. * Corresponding author: J. H. Cheng ( chengjiehai@hpu.edu.cn) and S. Li (lishen@swjtu.edu.cn)


INTRODUCTION
As a new remote sensing technology, UAV remote sensing was widely used in natural disaster monitoring, land resource management, land use survey, agricultural production activities, environmental pollution monitoring, urban construction, coastal zone monitoring, because of its advantages in high resolution, flexible operation, low cost, low-flying under clouds and ability to work full-time (Lei et al.,2016). However,UAV remote sensing image has the problems of small coverage area, large number, irregular overlap, which cannot meet the actual needs, such as natural disaster relief (Lei et al.,2017). Also, it is difficult to quickly and accurately stitch a regional panorama for lots of UAV remote sensing images. How to quickly and efficiently splice a lot of UAV remote sensing images and obtain a regional full coverage, wide field of vision and high precision image is particularly important.
Image matching technology integrated photogrammetry, digital image processing, computer vision and pattern recognition. According to the different methods of using image information, the image matching algorithms was divided into gray-based and feature-based matching technology (Zhu et al.,2017). Image matching algorithm based on gray level had been studied earlier in theory but less in practice (Kuglin,Hines,1975). Graymatching can be achieved by comparing the gray distribution correlation between the search image window and target image window. The accuracy and reliability of gray-matching algorithm were affected by the amount of information in the matching window. It was difficult to solve the problems of discontinuity, shadow and distortion in remote sensing image. Therefore, it was difficult to obtain satisfactory results using gray-based registration methods.
The feature-based images matching algorithm extracted the features from two or more images and then used the described parameters to complete the final image mosaic. At present, the popular algorithms were Harries (Harris,Stephens,1988), SUSAN(Small Univalue Segment Assimilating Nucleus) (Smith,Brady,1997) , SIFT (Scale Invariant Feature Transform) (Lowe,1999), SURF (Bay et al.,2008), ORB (Oriented FAST and Rotated BRIEF) (Rublee et al.,2011), BRISK (Binary Robust Invariant Scalable Keypoints) (Leutenegger et al., 2011),KAZA (Okawa , 2016) , AKAZE (Accelerated-KAZE) (Alcantarilla et al., 2013) and so on (Tareen, Saleem, 2018). Harries and SUSAN algorithm appeared earlier, but the adaptability of this algorithm was poor. SIFT algorithm was a corner detection algorithm that was scale and rotation unchanging. SIFT feature points extracted from the constructed scale space for image splicing had been widely used (Lowe,2004). At the same time, PCA-SIFT (Ke, Sukthankar, 2004), M-SIFT (Zuo et al.,2011) and so on had been proposed based-on SIFT algorithm. SURF algorithm of a variation of SIFT algorithm introduced integrated image and box filter to optimize the feature point extraction, which greatly improved the operation speed on the premise of ensuring the feature extraction effect. Both ORB and BRISK algorithms used binary feature descriptors to match feature points, which was fast in calculation but poor in robustness. KAZE and AKAZE algorithms appear in a short time, but they had strong robustness and good adaptability. In order to further improved the accuracy of feature extraction algorithm, some scholars used RANSAC (Random Sample Consensus) to improve the image registration accuracy (Patel et al.,2015). Later, some scholars used RANSAC (Choi et al.,1997), PROSAC (Chum, Matas,2005), MLESAC (Maximum Likelihood Estimation Sample Consensus) (Li et al.,2012), NAPSAC (N-Adjacent Points Sample Consensus) (Myatt et al.,2002), GASAC (Genetic Algorithm Sample Consensus) (Rodehorst, Hellwich, 2006) algorithm to eliminate the mismatch point pairs, and improve the accuracy and efficiency of image registration. In recent years, with the continuous development of neural network technology, more and more researches had been carried out based on deep neural network, convolutional neural network, multilayer perceptron and other neural networks. For example, there were more and more image mosaics combined traditional SIFT and SURF algorithms with deep learning technology. Some researchers combined the feature points extracted by convolutional neural networks such as VGG and AlexNet with the feature points extracted by SIFT, which proved that this method had great research value (Rashid et al.,2019;Shao et al.,2019).
With the development of image sensor technology, image resolution was also improving, and the number of feature point extraction was increasing rapidly with the increasing of image resolution. It was difficult to meet the real-time requirement for UAV image matching. In recent years, GPU used to optimize and accelerate the algorithm has become popular in worldwide. Experiments show that GPU accelerates significantly the operation speed of the algorithm compare with CPU and improves the operation efficiency of the algorithm greatly without reducing the accuracy of the algorithm (Terriberry et al.,2008). CuML (GPU Machine Learning Algorithms) heterogeneous accelerates machine learning algorithm library contains common PCA (Principal Components Analysis), SVD (Singular Value Decomposition) and other methods, which can be used to improve efficiency. Some scholars had introduced GPU parallel operation into image matching of SIFT (Acharya et al., 2018), SURF ( Terriberry et al., 2008) and KAZE (Ramkumar et al., 2019) algorithms to obtain more highprecision homonymous points in a short time and completed real-time processing of massive data. Based on the above discussion, GPU parallel computing is introduced into the application of high-resolution and massive remote sensing image registration, which can better solve the problem of low efficiency of remote sensing image registration.
At present, some research work has been carried out on image registration technology, and many scholars have conducted research on feature extraction base on feature extraction algorithm and GPU. However, there are still relatively few researches on high-resolution UAV images, and many problems such as low accuracy and slow speed need to be solved, especially the rapid processing of UAV Remote Sensing for flood disaster. Guided by the actual demand, GPU and PROSAC are used to improve SURF algorithm to splice a lot of UAV remote sensing data, to meet the needs of real-time splicing of emergency disaster monitoring. In this paper, we proposed an innovative method to improve the accuracy of SURF algorithm based on PROSAC algorithm, and used GPU parallel to accelerate the efficiency of the improved SURF algorithm, in order to further promote the development of image mosaic field.

Experimental Environment and Data
In this paper, the original SURF algorithm based on the CPU (Central Processing Unit) serial operation and the improved SURF algorithm optimized by GPU parallel operation was studied. The experimental environment of the study was operating system Windows 10, RAM 8G, CPU Intel (R) core (TM) i5-8300h (8 cores), GPU graphics card NVIDIA Geforce GTX1050 Ti, programming tool and development platform Visual Studio 2017, Opencv 3.4.1 and CUDA10. The experimental data were UAV remote sensing data set of Poyang Lake on June 24, 2016, as shown in Figure 1. The image resolution was 7360x4912 pixels. For the remote sensing data set captured by UAV, the flight attitude of UAV was estimated based on POS point data, and the images with small overlap and poor imaging quality were removed. And for the remaining UAV images, because of the large lens distortion, the image edge distortion will be more serious. It was necessary to cut out the areas around the image with more serious distortion, and use the central area with less distortion for subsequent image mosaic.

SURF Algorithm Introduction:
The speed up robust feature (SURF) algorithm was a fast feature extraction and image registration algorithm. The operation result was the same as SIFT, but the operation efficiency was higher. In order to avoid wasted a lot of time to perform complex convolution steps on different scale images, the algorithm proposed a simplified calculation of integral graph and Hessian matrix to improve the efficiency of image feature point calculation and completed feature point detection. The SURF algorithm flow was as follows: 1) Integral Images The integral image was generated from each pixel value in the original images. The establishment of the integral image reduced the calculation complexity, taking the first pixel coordinates as the coordinate origin, the value of any point in the integrated image was the sum of the pixel values in the rectangular region from the point in the original image to the coordinate origin. Only according to the value of the vertex of the original image rectangle in the integrated image, the sum of the required region pixels can be obtained.
2) Hessian matrix In SURF algorithm, box filter was used to calculate feature points. In fast Hessian detection, Hessian matrix was constructed by Gaussian filter, and then the discriminant of Hessian matrix was obtained by convolution of box filter and image. If the discriminant was the local maximum of the Hessian within the constructed scale space, the point was confirmed as the key point of SURF feature. Given a point as (x, y), the formula of Hessian matrix was formula (1): In the formula, L was the input image, the scale was σ, Lxx (x, y, σ) was the result of convolution of Gaussian function and image at this point. It can be seen from the formula that each pixel can form an H matrix, and the image of different scales can be obtained by changing the box filter. The formula for the Gaussian function was equation 2: The formula for the Lxx (x, y, σ) was equation 3: It can be seen from analogy that Lxy (x, y, σ), lyy (x, y, σ), Dxx, Dxy and Dyy were used to replace Lxx, Lxy and Lyy. When Hessian matrix gets the local maximum value, this point was the key point, so the discriminant of Hessian matrix was formula (4): In the formula, 0.9 was the weight coefficient, which was introduced to balance the accurate value with the approximate value.
3) Constructing scale space The scale space of SURF was constructed by changing the filter with the fixed image unchanged. This way could improve the operation efficiency. The size of scale space groups was the same, but the difference was that the size of the filter was different. The size of the filter was the same between different layers of the same group, but the fuzzy coefficient was not the same, so as to generate pictures of different sizes and build the scale space purpose. 4) Keypoint identification In the scale space, three Hessian determinant images were found in each group, and each point in the middle layer was compared with the surrounding 26 points. The key points were located in the scale space domain. 5) Keypoint orientation In the neighborhood of the feature points, the sum of horizontal and vertical Harr wavelet features of all points in the 60-degree sector was counted, then the sector was rotated and calculated with a fixed value, and finally the sector direction with the largest value was taken as the main direction of the feature points. Main direction of feature points was shown in Figure 3. 6) Keypoint descriptor creation Based on the key point and its main direction, a square box with side length of 20σ was selected, which was further divided into small 4x4 square. The Haar wavelet features of the sum of the absolute values of the horizontal direction and the horizontal direction of all pixels in each small square and the Haar wavelet features of the sum of the absolute values of the vertical direction and the vertical direction were calculated respectively. Therefore, the descriptor was represented by a 64dimensional feature vector. 7) Feature point matching Feature point matching was to calculate the Euclidean distance between two feature descriptors. If value was lower than the requirements of set threshold, it was a right match. The next step was to remove mismatching according to additional constraints，in order to improve the accuracy.

PROSAC Algorithm Introduction: PROSAC algorithm
was an improved algorithm based on the RANSAC algorithm. It was used to perform a geometric verification. The algorithm arranged the feature matching points in descending order according to the similarity value, and then drawn forms from a growing set of top-ranked correspondences to calculate the optimal model. Compared with RANSAC algorithm, PROSAC was a semi-random and semi-artificial parameter estimation method, which had high robustness and efficiency, especially in dealing with a lot of mismatching points generated by highresolution images. There were three steps in the experimental process, as follows: 1) Set parameters Set the maximum number of iterations, number of inliers and other parameters.
2) Judge the number of iterations In the range of iteration times, according to the difference of match point pairs, the feature point pairs were arranged in descending order from high to low. Four groups of match points were randomly selected in the point set with high matching quality to calculate the model parameters. The model parameters were taken back to the data samples to calculate the model error and the number of inliers and outliers points in the point set. Finally, the set with the most inlier points in the iteration times was returned.
3) Determine the current number of inlier points If the current number of inlier points was greater than the set value, the set of inlier points of the point was returned, and the program ends. Otherwise, repeated the iteration until the maximum number of iterations was reached. PROSAC linear fitting experiment and flowchart was shown in Figure 2.

GPU Method Introduction:
With the rapid progress of science and technology, GPU as an efficient graphics accelerator, has a lot of executable units and larger bandwidth memory. With the advantages of high parallelism and low power consumption, GPU can process massive data in parallel and quickly, which has widely used in the field of scientific computing. GPU is suitable for large-scale data parallel tasks with high computing density and simple logical branches. The combination of CPU and GPU can improve the operation efficiency. The logical architecture comparison between CPU and GPU was shown in Figure 3.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 3. Comparison between CPU and GPU

AN IMPROVED SURFGPU ALGORITHM FRAMEWORK
According to the processed UAV remote sensing image, used the improved SURF algorithm in this paper to accomplish image mosaic. The experimental flow chart was shown in Figure 4. The innovation of this paper was that GPU was used to accelerate feature point extraction and PROSAC algorithm was used to remove a large number of matching point pairs;we used GPU parallel operation to accelerate the calculation of improved SURF algorithm and PROSAC algorithm was used to improve the accuracy of remote sensing image registration based-on SURF algorithm. GPU parallel operation and PROSAC algorithm were used twice in this experiment. Four groups of high-resolution UAV remote sensing images were used as test data in the experiment. Each group of experiments was repeated once, and relevant experimental data were recorded. Then, this paper used control point data to complete geometric correction process and finally obtained the completed result map of the research area.

Improved SURF Image Mosaic Algorithm Based on PROSAC
Traditional SURF algorithm was used to register UAV remote sensing images, which could obtain a lot of feature points with low registration accuracy. Therefore, in order to improve the matching accuracy of feature points, PROSAC algorithm was introduced to improve SURF algorithm. First of all, we used SURF algorithm to extract feature points and used PROSAC algorithm to remove some feature points, and then NNDR (nearest neighbor distance) was used for UAV remote sensing image coarse registration, and the feature point pairs with threshold value of 0.5 was retained. This method could eliminate most of the obvious wrong matching pairs. Finally, PROSAC algorithm was used to filter the matching points after coarse registration, and the set of retained excellent matching points was used for image precise registration. In the process of image matching, the combination of rough matching and fine matching not only reduced the computational complexity, but also reduced the mismatch rate, improved the efficiency and accuracy of the operation, which was conducive to the subsequent image processing.

Improved SURF Registration Algorithm Based on GPU Parallel Acceleration Optimization (SURFGPU)
In view of the high resolution of UAV remote sensing image, the traditional SURF algorithm only completed the extraction of feature points and the description of feature points in the CPU, which had a large amount of calculation and processing speed cannot meet the requirements of speed. Therefore, the improved SURF registration algorithm based on GPU parallel acceleration could complete feature point extraction and feature point descriptor calculation. GPU could start enough thread and each thread can complete simple calculation tasks to effectively improve the performance of data processing. Finally, GPU could improve efficiency of UAV remote sensing image registration. In this experiment, GPU was used to accelerate the maximum point location, feature main direction calculation and feature descriptor calculation in SURF algorithm. The process framework of feature description and feature extraction of improved SURF algorithm based-on GPU was as follows: 1) Transferring the input remote sensing image from the host memory to GPU displays memory. 2) In GPU, the image was first integrated to generate integral image: parallel computing in GPU could greatly improve the operation speed in the processing of remote sensing image row or column. 3) The integral image was used to detect the feature and obtain the location and scale parameters of the feature. 4) The it main direction of feature were calculated by integrating image and feature parameters, the GPU function call was used once for each. 5) Get the eigenvector: the GPU calculation process of SURF eigenvector was relatively intuitive. Because there were 16 areas in SURF descriptor, 16 threads can be used to calculate the eigenvector of one feature, and each thread could calculate one area. 6) GPU was used to accelerate the process of feature point matching. 7) Downloading the image from GPU device memory to CPU hosts memory. The transformation matrix of image could be obtained by correct feature matching points. The image registration was realized by the transformation matrix, and then the registered image and the reference image were fused into a complete image.

Experimental Results
In the experiment, comparison the original SURF algorithm and the improved SURF algorithm by high-resolution UAV remote sensing images. The one was the reference image and the other was to be registered image. Compared with SURF algorithm and improved SURF algorithm, the deformation of the improved SURF was smaller and more accurate. Figure 5 and 6 were the matching feature points based-on SURF algorithm and improved SURF algorithm based-on PROSAC algorithm. The homography matrix calculation of two images was shown in formula (5):  Figure 7 was the registration of two images. The accuracy of SURF algorithm and PROSAC-SURF algorithm was shown Table 1. When the resolution of the test image was 7360x4912 pixels, the number of feature points obtained by using improved SURF based-on PROSAC algorithm was only 9.5% than that of SURF algorithm. Moreover, the accuracy rate of improved method was about 99.7%, while the accuracy rate of only using the SURF algorithm was about 92%. The accuracy rate of improved SURF algorithm was increased by 8% than that of SURF algorithm. The performance of improved SURF algorithm based-on PROSAC algorithm was very satisfactory.  At the same time, GPU-accelerated optimization was performed on the improved SURF algorithm to achieve higher efficiency and meet actual application needs. The detailed statistics on the running time between SURF and SURFGPU algorithm, as shown in Table 2 and Figure 8. Furthermore, the running time of SURF and SURFGPU algorithms was counted.
As can be seen in Figure 8,

Accuracy Verification
In order to verify the effectiveness of this method, the relevant information of two image feature matching point pairs saved in the experimental process was counted. If the correlation coefficient was relatively high, which proved that the matching points were the same point and otherwise the lower the probability that the matching points were the same point. The correlation coefficients of a set of experimental data rough matching were shown in the In order to verify the experimental accuracy of this paper, 37 groups of control points were selected from eight images and displayed in this paper. The distribution of the control points was shown in the Figure 9. The coordinates of the control points on the reference image and the control points after the transformation of the image to be registered were counted. The statistical results were shown in the Table 4.  Table 4 and Figure 9, the accuracy of the four groups of control points in this paper was within 0.4 pixel, which improved the registration accuracy of the traditional SURF algorithm. The result showed that GPU parallel acceleration was better than CPU. The improved method not only improved the running time of SURFGPU algorithm of UAV remote sensing image registration by 16 times, but also made the registration accuracy of remote sensing image registration algorithm reached 0.4 pixel and the calculation time reached millisecond level, which proved the effectiveness of the improved method. In fact, the improved SURFGPU algorithm can meet the requirement of rapid, automatic and efficient registration of UAV remote sensing images. In some degree, the images can be matched fast and correctly in the mass characteristics database based on the experiment results. Indeed, other scholars also found that the improved SURF algorithm had better match effect. However, matching effect and timeliness can be significantly improved if the use of cluster computers can realize real-time transmission and mosaic based-on cloud computing resources, and meet the entire process of real-time operations, mosaic processing, information extraction, and mapping applications. It was particularly important of rapid automatic registration for disaster emergency relief. The efficient was less when only using Euclidean Distance to SURF feature vector match; if PROSAC algorithm was used to remove the mismatch further, SURF algorithm could achieve higher correct matching rate. After the real-time splicing of the image in the study area, the distortion error of the image was corrected according to the data of the ground control points to generate the final study area Orthophoto Image. We used the improved SURFGPU algorithm to stitch a completed full coverage area map in near real time, as shown in Figure 10.

Discussion
Based on the improvement of image mosaic, many researchers put forward a variety of improved methods, mainly from improved the accuracy or speed. We compare many of the literature, in terms of improving the accuracy, KNN (k-Nearest Neighbor), BBF (Best Bin First), RANSAC and other algorithms were used to obtain accurate feature matching point pairs, but those methods can improve the operation speed in a certain extent. Moreover, the operation efficiency of this method was limited by the host CPU, and the acceleration ratio was limited. A series of improved RANSAC algorithms, such as PROSAC, MLESAC, NAPSAC, GASAC, still had many obvious defects in the application fields with high real-time requirements (Choi et al.,1997). It had many problems, such as many times of iterative calculation and large amount of calculation. When the number of inliners was lack, it fall into a cycle of iteration, unable to get the correct model or get the wrong model. In the future, the calculation speed and accuracy of the model can be accelerated by setting a larger weight for the inliners. With the increasing popularity of deep learning, the combination of neural network, SIFT and SURF algorithm had become a research hotspot. However, for a specific research area, this method needs a lot of learning samples for training model, so although the accuracy of this method was high, it is still in the research stage.
Combined with the actual emergency disaster demand, an improved UAV remote sensing image processing method was proposed. This paper used the improved SURF algorithm to achieve image registration, which showed that experimental speed was increased by 16 times, the accuracy was improved by 8% and the accuracy was kept within 0.4 pixel than the traditional SURF algorithm. Many scholars had studied the improvement of SURF algorithm. Hu's method improved the accuracy by 10% and the pixel accuracy was kept within 0.6 pixel, but there was no significant change in speed (Hu,et al.,2019). Block and relative distance were introduced to improve SURF, which had increased the matching correctness by 12% and the pixel accuracy was up to 2.69 pixels (Pan et al., 2017). By combining GPU parallel computing with SURF algorithm, the speed had been increased by more than 10 times (Liu et al.,2014). The performance was 8 times higher by SURF algorithm based on the FPGA platform than the dualcore CPU's operating efficiency, and the number of feature points was less than 1% of the total (Bouris et al.,2010). Patel proposed to obtain the basis direction based on SVD to align points obtained by SURF algorithm, which made search performance improved by 10%. In view of the large number of parallel computing units, GPU can be used to accelerate the speed (Patel, Patel, 2014). It could be seen that running time of registration was obviously better than the single SURF algorithm and other improved algorithms based on SURF. It can obtain better effectiveness because that GPU has the advantage of high computing efficiency, and reasonable use of GPU to accelerate optimization. It improved the accuracy and real-time performance of the algorithm, and could meet the real-time requirements for high-resolution UAV remote sensing image registration and promoted the further development of image matching field. In view of the strong practicability of the improved method in this paper, it could be used to develop a set of UAV data processing software. The whole process of UAV image processing was systematically realized to meet the actual needs of scientific research, engineering application, disaster emergency relief, and promoted the rapid development of UAV field.

CONCLUSION
In this paper, we used PROSAC algorithm to improve the accuracy of remote sensing image registration based-on SURF algorithm, and GPU parallel operation was used to accelerate the calculation of improved SURF algorithm. We introduced PROSAC to perform good registration and completed image registration according to the saved high-quality matching point pairs. Additionally, we used GPU parallel calculation to accelerate the experiment based-on improved SURF algorithm. Finally, an improved SURFGPU algorithm framework was carried out, and the number of feature points obtained by using improved SURF based-on PROSAC algorithm was about 9.5% than that of SURF algorithm. Moreover, the accuracy rate of improved method was about 99.7%, while the accuracy rate of improved SURF algorithm was increased by 8% than SURF algorithm. Moreover, the improved running time of SURFGPU algorithm for UAV remote sensing image registration was about 16 times as fast as SURF algorithm, and the computing time had reached millisecond level. Thus, improved SURF algorithm had better matching accuracy and faster speed to meet the requirements of UAV remote sensing image registration speed, accuracy and robustness. The improved method can solve the problem of long running time and low accuracy of the traditional algorithm, which had achieved realtime and efficient registration of remote sensing image. The registration result was more reliable, providing an efficient and cheap platform for UAV remote sensing image processing and had high engineering application value.