OPTIMIZATION ALGORITHM OF TILTED IMAGE MATCHING BASED ON ADAPTIVE INITIAL OBJECT ASPECT

When adopting the matching method of the least squares image based on object-patch to match tilted images, problems like the low degree of connection points for images with the discontinuity of depth or the discrepancy in elevation or low availability of aerotriangulation points would frequently appear. To address such problems, a tilted-image-matching algorithm based on an adaptive initial object-patch is proposed by this paper. By means of the existing initial values of the interior and exterior orientation elements of the tilted image and the information of object points generated in the matching process, the algorithm takes advantage of the method of multi-patch forward intersection and object variance partition so as to adaptively calculate the elevation of the object-patch and the initial value of the normal vector direction angle. Furthermore, this algorithm aims to solve the problem of difficulties in matching the tilted image with its corresponding points brought about by the low accuracy of the initial value of the tilted image when adopting the matching method of the least squares image based on object-patch to match the tilted image with high discrepancy in elevation. We adopt the algorithm as proposed in this paper and the least squares image matching method in which the initial state of the object-patch is horizontal to the object-patch respectively to conduct the verification process of comparing and matching two groups of tilted images. Finally, the effectiveness of the algorithm as proposed in this paper is verified by the testing results. * Corresponding author


INTRODUCTION
The accuracy in matching the aerial images is a key step in acquiring the information of spatial position of terrain objects by photogrammetry and in conducting the three-dimensional reconstruction of terrain and terrain objects. In the past decades, scholars in the field of computer vision and photogrammetry have completed a lot of fruitful researches on image matching. Among them, the image-based matching has developed from gray-based image matching method to feature-based image matching with higher accuracy and reliability, such as aerial image matching based on feature relaxation method (Jiang Wanshou et al.2003), SIFT feature matching algorithm combined with global information (Jihua, 2009), and multi-view image matching method based on feature point guidance (Ji Song et al. 2018), and etc. Image matching based on object-surface elements has been developed on the basis of traditional Vertical Line Locus (VLL), such as single-stereo VLL method (Zheng Shunyi, 2004) and MVLL method (Zhang L, Gruen A. 2006) based on multi-view image matching. Besides solving different types of image matching, these methods of image matching have greatly promoted the development of digital photogrammetry towards its automation and intellectualization, and in the meanwhile, reduced difficulties of technology application as well as promoted developments of the industry of aerial photogrammetry industry.
In addition, on the basis of classical multi-Photo Geometrically Constrained Matching (MPGC) (Baltsavias E.P., 1991), domestic scholars have, by introducing the concept of object-patch, eliminated the independent affine deformation among images, established a new least-square image matching model, and proposed a least squares image matching method based on object-patch. This method reduces the degree of freedom of MPGC algorithm and generates Digital Surface Model (DSM) at the same time of image matching, for which reason it is also called image matching method for generating DSM. When using this method for image matching by the literature (Jiang Wanshou, 2004), the image points on the base image are projected to the level of the initial state, and the approximate coordinates of the object points are obtained on the object elements with the outline elevation. Then, the object elements with the object points as the center are projected back to the search images and matched to the same name points by the correlation coefficient method, and then multi-slices are used. The initial value of the object point is obtained by forward intersection. Finally, the initial value of the normal vector direction angle of the object point and the object element is brought into the error equation to iteratively solve the exact coordinates of the object point and the image point.
When used for traditional frame aerial image matching, the least squares image matching method based on object-patch has achieved good results as of the accuracy and efficiency for matching as well as multiple purposes of the algorithm. However, when applied to tilted image matching, due to conditions like the tilted angle of tilted image being large and the object conditions being complex (there are many areas with discontinuous depth and large height difference), the image points on the base image are projected to the initial level and the object coordinates on the object element with approximate elevation are erroneous, so that the object points are projected to the initial level and the object coordinates are obtained from the object element with approximate elevation. After back projection to each search image, the correlation coefficient method is used to match the image points with large residual error or matching failure, which further results in the empty three points being too small in some areas where the depth is discontinuous and the height difference is large. The three-dimensional reconstruction of tilted photography requires a higher degree of image connection points and the density of empty three points. In this vein, there will be holes in the model after dense matching, networking and texture mapping based on empty three points. Therefore, the algorithm needs to be further improved and optimized when used for tilted image matching.
In order to solve problems as addressed above, this paper proposes a tilted image matching algorithm based on adaptive initial objectpatch on top of the least squares image matching method based on object-patch. By taking advantage of the initial internal and external orientation elements of the image and the method of object variance partition as well as by adaptively calculating the initial elevation and normal vector orientation angle of the object-patch according to the change of matching point position, this algorithm ensures that accurate and reliable homonym points can be matched in the image so as to increase the degree of image connection points and expand application scenarios of the least squares image matching method based on object-patch.

Figure1. The proposed tilted image matching method
To solve the problem of matching homonymous image points in areas with deep discontinuities or large differences, this paper improves the calculation method of matching initial values and ensures image-side matching by using the correlation coefficient method by improving the accuracy of the initial value. When matching, accurate and reliable image points of the same name can be obtained. This method calculates the initial value of the object-patch elevation via space intersection with multiple images. The initial value of the normal vector direction angle of the object-patch is based on the assumptions that the local surface of the object is continuous and that the objectpatch is the tangent plane of the local surface. The object-patch information generated during matching is calculated by the object difference method. The flow chart of this matching algorithm is shown in Figure 1.

Adaptive calculation of the initial value of object-patch elevation
In this paper, the steps of the self-adapting calculation of elevation initial values of the object-patch are as follows: 1) calculate the homography matrix between the base image and the search image from the initial internal and external orientation elements of the image; 2) estimate the coordinates of the homonymous point on the base image from the relative homography matrix; 3) use space intersection with multiple images to approximate the elevation of the object-patch; 4) calculate the initial value of the direction angle of the normal vector of the object-patch by using the object difference method based on object point information; 5) obtain the object point of the object-patch center by projecting the image point to the elevation of the direction angle of the normal vector; 6) project the object-patches to each search image to get the image region and sample the gray level of the image in the image area; 7) match the image point coordinates by using the correlation coefficient method in the image area; and 8) use image points to calculate the initial values of object points via space intersection with multiple images.
In the above steps, approximating the elevation of the element by using multi slice forward intersection after estimating the coordinates of the image points with the same name from the matrix is crucial, as this step can avoid large errors that lead to the inaccuracy of the object points after projecting the image points to the object elements and back projecting the object elements to the search image. Then, matching the same name points in the image can be difficult. The process of calculating the relative homography matrix and estimating the coordinates of homonymous image points from the initial internal and external orientation elements of the image is as follows: The relationship between the interior and exterior orientation elements of the base image and the search image and the projection matrix is as follows. (1) Where , the projection matrix, the internal parameter matrix, the rotation matrix and the coordinates of the photographic center (the line elements of the external parameters of the image) of the base image and the search image are used.

Let
. Assuming that there is an object point on the object plane, the relation between image point and object point of the base image and the search matching image is as follows: (2) The base image and the search image are incorporated into the image space coordinate system of the base image as the image space auxiliary coordinate system for image matching. At this time, formula (2) can be written as follows: (3) The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) The relationship between the image points on the base image and those on the search image obtained by formulas (2) and (3) is as follows: (4) The expression of the relative homography matrix between the base image and the search image is as follows: In formula (5), the relative homography matrix between the base image and the search image can be calculated by using the initial internal and external orientation elements of the image. In image matching, the coordinates of homonymous points are estimated by the change in image point on the base image. Finally, the initial elevation of the object-patch is calculated adaptively according to steps 3 to 8.

The Initial Value of the Normal Vector Direction Angle of the Matter Aspect Element Calculated by the Object Variance Partition Method
Geometrically, an object-patch is the tangent plane of the local surface of the object. When object points corresponding to the matched image points are located on the horizontal plane, tilted image matching using the horizontal plane as the initial state of the object-patch can achieve a good matching effect, except when the object points corresponding to the matched image points are located on the elevation plane (such as buildings). When matching tilted images with horizontal plane elements as the initial state of the object-patch, there will be a large projection error, which will lead to inaccurate and unreliable matching of homonymous points. In tilted image matching, according to the change in matching points in the error equation, an accurate initial value of the normal vector direction angle of the object-patch is given. According to the local surface continuity of the object, the initial value of the normal vector direction angle of the object-patch is calculated adaptively by the object difference method to improve the image matching effect.
According to the condition that the local surface of the object is approximately continuous, the local surface is fitted by the object points generated in the matching process.
Then, the normal vectors of the tangent plane of the local surface at the point are calculated: After obtaining the normal vector of the tangent plane of the local surface of the object at the point, the normal vector of the local surface element (i.e., the object-patch) centered on the point of the object is obtained by using the difference instead of the approximate value of the partial derivative of the differential calculation: , , , represent the coordinates of adjacent points in the object square and direction, respectively.
After normalizing the normal vector n, the normal vector direction angles α and β of the object-patch are obtained.
where = the gray level of the image at the matching point; = the corrected value of the coordinates of the image points; The unknowns of the equation are the elevation of the object-patch; α, β = the normal vector orientation angles of the objectpatch and base image; h 0i , h 1i = the radiation distortion parameters between the amplitude search images.
The relationship between the corrected coordinates of image points and the unknowns of the equation is as follows: Due to space limitations, refer to reference (Jiang Wanshou, 2004) for the derivation of formula (11), as this paper does not present it.

Experimental Data
To verify the effectiveness of the proposed algorithm, two sets of tilted image data are selected for experiments. The first group contains 160 tilted images in areas with large elevation differences, and the second group contains 150 tilted images in areas with relatively flat terrains. Detailed information of the two groups of image data is shown in Table 1. Based on a Win7 64-bit operating system, this paper configures the development environment in VS 2010 and uses VC++ language to write programs to verify the algorithm. The platform is a computer with an Intel Core 3.3-GHz processor and 16GB of memory. Table1 Experimental data information

Tilted Image Matching Experiment in Regions with Large Elevation Differences
The matching experiments of the first group of tilt images are carried out by using the minimum two-image matching method (hereinafter referred to as the "initial horizontal panel method"), in which the initial state of the aspect element in document (Jiang Wanshou, 2004) is the horizontal plane element, and the method proposed in this paper. In the experiment, the size of the correlation window is changed, and the experimental results are counted. Then, the effect of the correlation window size on the image matching results is studied. In this paper, 20 pairs of images are randomly selected. The matching results of 400 pairs of identical points in each pair under different window sizes are shown in Table 2. To verify the effect of the accuracy of the initial exterior orientation elements on the matching results of this method, after matching the first group of tilt images and adjusting them by using the Bundle Adjustment, the corrections of the exterior orientation elements of two representative images and the average iteration times of matching each image point are counted (Table 3). To verify the quality of the empty three points generated by tilt image matching, the average degree of image connection points and the number of effective points are counted after the empty three points are processed (Table 4). To compare the effect of the initial value accuracy on the matching efficiency, the initial and adjusted values of partial image point matching are given in Table 5. Two matching methods are used to get the object points (empty three points) of the connection points, as shown in Figure 2. Based on the three empty points, the three-dimensional model is reconstructed by dense matching, meshing and texture mapping, as shown in Figure 3.

Tilted image matching experiment in flat terrain areas
To further verify the effect of this method on tilted image matching in flat terrain areas, the initial horizontal panel method and this method are used to match the second group of tilted images. After Bundle Adjustment, (Table 6) calculates the average residual value of image points, the average degree of image connection points, the number of effective objects and matching time. Three-dimensional reconstruction is done after image matching and Bundle Adjustment to obtain three points in space to visually compare the effects of two matching methods in three-dimensional processing ( Figure.4) and threedimensional reconstruction (Figure.5  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

Analysis of the experimental results
The experimental results are analyzed as follows.1) The influence of the size of the window on the matching result. Because of the large inclination of the tilted image, when the base image changes, the object area (i.e., the size of the aspect element) obtained by projecting the window area centered on the image point on the base image will change greatly. In this paper, the method of least squares matching based on objectpatches is used to match the image area by using the correlation coefficient method to obtain more accurate initial values of same name image points. Therefore, the size of the window centered on the image points on the base image is analyzed to indirectly study the effect of the size of the panel on the matching results. As seen from Table 2, when the window size is within a certain range (less than 31×31 pixels), it is positively correlated with the probability of matching to the same name point: with the increase in the window opened by the image side, the maximum correlation coefficient increases, that is, the accuracy of matching to the same name point is improved. After matching the exact homonym points, the coordinates of the object points and more accurate initial values can be calculated, and the number of iterations can be reduced. However, when the window is too large, the object and texture information in the window will become more complex, resulting in a smaller maximum correlation coefficient when the window slides in a given range and lower accuracy of matching image points; thus, the window size has a considerable impact on the matching results. In the experiment, we found that the matching accuracy and efficiency could be better taken into account when the window size was 31× 31 at a certain critical value.
2) The influence of the accuracy of the initial internal and external orientation elements on the matching results. The more accurate the initial internal and external azimuth elements, the more accurate the image points that are estimated by the relative homography matrix. The more accurate the elevation value of the center of the pixel obtained by using the approximate coordinates of the same name image points and the more accurate the same name image points matched by the correlation coefficient method when the object-patches are projected to the search image, the more the matching accuracy can be improved. Table 3 shows that the initial exterior orientation elements of DSC2153 are more accurate than those of DSC1521; thus, the number of iterations of the adjustment calculation is lower in image matching.
3) The influence of the accuracy of the initial value on the matching results. As seen from the influence of the accuracy of the initial values on the matching results in Table 4, this method can give more accurate initial values than the initial horizontal panel method; thus, the number of iterations required for matching is lower. In addition, when the deviation of the initial value is too large, matching the same name image points in the back-projection area of the image side fails, which leads to fewer empty three points. In the red elliptic region of Figure 2 (a), because of the inaccurate initial value, there are few matched homonym points, and the degree of the connection points is low, which leads to a scarcity of empty three points after processing. When three-dimensional reconstruction is based on empty three points, there will be holes in the model due to the lack of detail in the triangular network because there are too few points (red elliptic area in Figure 3 (a)). 4) Effect analysis of tilted image matching in flat terrain areas. The surface elevation of the second group of images is approximately 32m. In the matching experiment, the initial value of the initial horizontal panel method is 32m. The initial value algorithm of this method is automatically calculated. From the average degree of join points and the number of effective object points in Table 6, it can be seen that the matching effect of this method is similar to that of the initial horizontal panel method when matching tilted images in areas with little change in elevation difference. Because the computational complexity of this method is high, the matching time of this method is mostly due to the initial horizontal panel method. In addition, Figure 4 shows that the density of the empty three points obtained by matching and adjusting the two methods is similar.

CONCLUSIONS
In addressing problems like the low degree of connection points for images with the discontinuity of depth or the discrepancy in elevation or low availability of aerotraingulation points when applying the matching method of the least squares image based on object-patch to tilted images matching, this paper proposed an algorithm of tilted image matching based on adaptive initial object aspect. By taking advantage of the method of multiforward intersection and object variance partition in adaptively calculating the initial matching value, the algorithm manages to apply the matching method based on object-patch to tilted images matching, so as to expand the application scenario of the matching method based on object-patch.
The main idea of this algorithm is to improve the accuracy of the initial values of the elevation and direction of normal vectors to accurately and reliably match homonyms on the image side and address the problem of the initial values not being correct when matching tilted images in regions with large elevation differences. In this paper, two sets of tilted image data are matched and compared. The results show that the method as proposed by this paper can obtain better matching results when matching tilted images in areas with discontinuous depth or large differences in elevation . In the meanwhile, when matching tilted images in flat terrain, the matching effect of this method is similar to that in the literature (Jiang Wanshou, 2004).
The matching algorithm in this paper is based on the least square's algorithm, which belongs to quadratic precise matching, namely the matching accuracy depends on the accuracy of the initial value. Although the adaptive calculation of the initial value of matching is innovatively presented, the relationship between the accuracy of the initial internal and external orientation elements and the matching accuracy of the image is not quantitatively determined. The complexity of algorithm in adaptively calculating matching initial values requires further optimization. In addition, the occlusion problem in tilted image matching will be addressed in following researches.