EVALUATING TIE POINTS DISTRIBUTION, MULTIPLICITY AND NUMBER ON THE ACCURACY OF UAV PHOTOGRAMMETRY BLOCKS

Image orientation is a fundamental task in photogrammetric applications and it is performed by extracting keypoints with hand-crafted or learning-based methods, generating tie points among the images and running a bundle adjustment procedure. Nowadays, due to large number of extracted keypoints, tie point filtering approaches attempt to eliminate redundant tie points in order to increase accuracy and reduce processing time. This paper presents the results of an investigation concerning tie points impact on bundle adjustment results. Simulations and real data are processed in Australis and DBAT to evaluate different affecting factors, including tie point numbers, location accuracy, distribution and multiplicity. Achieved results show that increasing the amount of tie points improve the quality of bundle adjustment results, provided that the tie points are well-distributed on the image. Furthermore, bundle adjustment quality is improved as the multiplicity of tie points increases and their location uncertainty decrease. Based on simulation results, some suggestions for accurate tie points filtering in typical UAV photogrammetry blocks cases are derived.


INTRODUCTION
In recent years, Unmanned Aerial Vehicles (UAVs) have become a commonly used remote sensing platform (Colomina and Molina, 2014;Nex and Remondino, 2014;Hassanalian and Abdelkefi, 2017;Granshaw, 2018). Various outputs such as highdensity point cloud, 3D model of the scenes, mosaicked images, cadastral maps and high resolution orthophotos can be obtained using on UAV-based images. Within the UAV photogrammetry pipeline, image orientation parameters and sparse point clouds are simultaneously computed using a bundle adjustment procedure. This is performed using image keypoints which are automatically extracted from the set of acquired images using hand-crafted or learning-based methods (Schönberger et al., 2017;Remondino et al., 2021). Most steps of the UAV photogrammetry pipeline can be carried out in a fully automated way using Structure from Motion (SfM) and Multi-View Stereo (MVS) technique. Since some years, the use of hand-crafted feature-based matching algorithms like SIFT (Lowe, 2004) or SURF (Bay et al., 2008), which are able to detect numerous keypoints and find corresponding points between set of convergent images, has removed the efforts for the manual selection of tie points. Therefore, manual work is limited only for measuring ground control points (GCPs). Current feature-based matching algorithms extract a large number of keypoints, the number of which directly depends on the information content of the image. In addition, in large-scale UAV blocks with a large number of images, the huge number of matched keypoints lead to high number of tie points. This causes the bundle adjustment to be time-consuming and prone to errors. Many attempts have been made to improve bundle adjustment precision and accelerate its efficiency within the image orientation / Structure from Motion (SfM) step. Some studies used clustering or graph-based techniques to reduce the number of images in SfM. Snavely et al. (2008) proposed a skeleton representation of the dominant images to speed up the subsequent incremental camera additions and scene reconstruction. Li et al. (2008) clustered a set of images to find the 'iconic images' first, then computed 3D scene incrementally based on spanning trees. The studies of Cui et al. (2020) and Chen et al. (2020) can be pointed out in this field, too. The reduction of images makes large-scale adjustment more tractable, though the completeness of scene structure cannot be guaranteed. Other works provide a tie point selection\filtering schema for improving bundle adjustment performances (Nocerino et al., 2013). The tie point selection/filtering problem could be defined as finding a more robust subset of tie points to increase both accuracy and efficiency of camera calibration and image orientation. Tie point filtering in UAV photogrammetry blocks is necessary for two reasons. Firstly, in large-scale UAV blocks, tie points usually contain many wrong matches. Secondly, the number of constraints generated by all tie points is much greater than the number of constraints needed for the computation of the unknowns. Therefore, selecting a subset of tie points could increase the efficiency of calculations and extend the scalability of orientation process. Some works proposed to select tie points from high-scale feature matches based on the features scale ordering. For instance, Wu (2013) proposed a match selection method, so called pre-emptive matching algorithm, to quickly judge whether or not two images could be matched. Shah et al. (2014) proposed a similar method which begins with building a coarse 3D reconstruction using a fraction of high-scale features of given images. A related approach has been described in Mayer (2003), Lerma et al. (2013) and Liu et al. (2014). Based on the analysis of error upper bound, Liu et al. (2014) and similarly Cui et al. (2015) selected a subset of matches that has a good quality vs. quantity trade-off to enhance the accuracy of two-view SfM. In Cui et al. (2017) a fast tracks selection method to improve both efficiency and robustness of the bundle adjustment is proposed. In their method, three selection criteria of Compactness, Accurateness and Connectedness are introduced: the first two are used to calculate a selection priority for each track and the third is to guarantee the completeness of scene structure. Then, to satisfy these criteria, a more informative subset of tie points is selected by covering multiple spanning trees of epipolar geometry graph. Since tie point selection acts only an intermediate step in the whole SfM pipeline, it can be in principle embedded into any global SfM pipelines. The quality of the image orientation does not depend only on the number of the tie points, which have a limited effect on the network precision, but it is mostly affected by their correctness and distribution (Barazzetti, 2017). Poor precision and spatial distribution of the tie points prevents both accurate measurement of camera orientation and proper modelling of the systematic errors caused by camera optics. Kerner et al. (2016) studied the role of spatial distribution of tie points on aerial images and concluded that in challenging scenarios promoting spatial distribution patterns at the matching stage can be helpful in preventing degenerate configurations. Furthermore, other influential factors on the bundle adjustment results such as tie point multiplicity and location accuracy must also be considered. In an efficient tie point filtering algorithm, all aforementioned factors should be considered simultaneously. Few studies in the literature have fully investigated the impact of tie point observations and camera configuration in bundle adjustment (Voltolini et al., 2006;Nocerino et al., 2014) and the research is still lacking in clarifying the role of tie points' properties in image orientation and 3D reconstruction results, in particular for UAV datasets. Based on this, the paper aims to: (i) perform experiments with simulated and real data; (ii) analyse the impact of influential factors, such as number of tie points, multiplicity, distribution and location accuracy for filtering purposes; (iii) analyse the precision achievable with different image networks using simulations and real datasets. To this end, the proposed research involves comparison results of influential factors on the 3D reconstruction accuracy in different UAV blocks and clarifies their impact on the obtained results. The rest of the paper is organized as follows: an explanation of the evaluation methodology is presented in Section2, the obtained results are thoroughly discussed in Section3 whereas conclusions and future studies are drawn in the final Section.

EVALUATION METHODOLOGY
Influential factors related to tie points are analysed using both simulations and real datasets and statistical outcomes of a bundle adjustment results. Bundle adjustment (Triggs et al., 2000) is an indirect method able to simultaneously solve position and orientations of image blocks and ground coordinates of unknown points through collinearity equations. The task is solved using non-linear least squares methods, such as Gauss-Markov, Gauss-Newton or Levenberg-Marquardt. In evaluation phase using simulated image networks, the bundle adjustment formulation based on parametric model of collinearity equations can be written as follows (Mikhail, 1976;Kraus, 2011;Luhmann et al., 2013): (1) where v is the residuals vector, x is the unknown vector (camera parameters, 3D coordinates), l is the observations vector (image coordinates of tie points, etc.) and A is the design matrix.
! and are the covariance matrix of observations and unknowns respectively, p the weight matrix and " # is the variance factor. Given the precision of image coordinates, interior and exterior camera parameters, the " matrix in Eq.3 could be estimated easily. As this equation shows, " is affected by both A and P matrixes, indicating the fact that both the network shape i.e. tie point distributions and precision of the observations impact the network accuracy. Therefore, the " matrix can be considered as the best quantity measure to express accuracy of networks. Firstly, some simulations are performed aiming to estimate the " matrix and analyse the results based on its values. The simulation process can be summarized as follows ( Figure 1): 1. A set of 3D points, e.g. acquired with laser scanning, are used to create an approximate 3D model of the scene to be surveyed. 2. A set of simulated camera stations are defined using arbitrary positions and rotation values (exterior orientation parameters) and predefined image properties (sensor size, pixel dimension, focal length). 3. 3D points are re-projected on the image planes (Figure 2), given the camera exterior and interior parameters, in order to create image observations (tie points). 4. A bundle adjustment is run in free network (inner constraint) to solve the datum defect problem. No known coordinates in the design matrix are used, thus the solution does not have connection with the utilized coordinate system or datum and the rank deficiency is dissolved.
5. The quality of the adjustment results is measured computing the covariance matrix " (Eq. 3) and its traces to determine the precision of the estimated object point coordinates, with . N refers to the number of object (tie) points used in the bundle adjustment.  In the second phase of the experiments, the performance evaluation was checked using real image blocks. Tie points are extracted from the image sets and then the influence of tie point distribution and numbers are analysed considering the outcomes of the bundle adjustment. As no ground control / truth is available, following Remondino et al., 2017 andMousavi et al. (2021), three criteria at the end of the bundle adjustment are considered: 1. the re-projection error of all computed 3D points (the smaller, the better).

Selecting Camera Stations
Re-project 3D points on Images

Run Bundle adjustment Estimate the Ʃx
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) 2. Average number of rays per 3D point, i.e., the redundancy of computed 3D object coordinates (the higher, the better). 3. Average intersection angles per 3D points, i.e., the angle of intersection to reconstruct 3D points by triangulation (the higher, the better).

EXPERIMENTS AND DISCUSSION
Simulations results are firstly presented (Section 3.1), then real datasets are analysed (Section 3.2). Tie points numbers, distribution quality, accuracy of image observations, tie point multiplicity and, for the simulations, estimated quality of 3D object coordinates (RMSE) are considered as influential factors.

Simulation analyses and results
To figure out the effect of tie point number, distribution and multiplicity on the photogrammetric procedure, two simulated image network configurations are considered: a straight sequence of nadir UAV images ( Figure 2) and a 360° closed sequence with convergent images around an object (Figure 3). For the straight UAV sequence, three simulations containing 24, 46 and 68 images were conducted, shortening the image baseline with the increase of the image number. In all datasets we supposed a DJI FC6520 camera (5280×3956 px, 0.0043mm pixel size) with a 12mm focal length. A laser scanner 3D point cloud is used to create the necessary object points for the simulation.
For the closed sequence of images, we considered datasets with 13, 20 and 26 images acquired with a Canon EOS 7D camera (5184×3456 px, 0.005mm pixel size) mounting a 15mm focal length. A laser scanner point cloud of a statue is subsampled and used for creating the simulation input data.  In all simulations, Australis 6 is used, allowing a maximum viewing angle of 60º for each point. The precision of image keypoints observations is considered identical; thus p is the identity matrix. In all simulation experiments, distortion-free images with principal point locating in the centre of the image are used, in order to avoid possible block deformations. The focal length and sensor size are considered based on the camera type assumed in each test.

Influence of tie points number
The estimated precision of 3D points (RMSE on check points) in the simulated UAV strip is reported in Figure 4a. Increasing the number of images from 24 to 68 images leads to a better precision of the 3D points. The precision is significantly improved by increasing the number of points from 100 to about 5000, whereas a larger number of tie points (more than 10,000) does not provide significant improvement in terms of RMSE. Similarly, for a closed block (Figure 4b), increasing the number of tie points from 20 to 5000 provide more precise 3D points while a large number of tie points do not significantly improve the accuracy results. The required processing time for bundle adjustment for both straight and closed block is shown in Figure 5. As it shows, in both blocks, the processing time quadratically increase when The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) larger number of tie points are applied. The processing time is also increased as the number of applied images rises.
Comparing the required processing time as a function of the number of tie points (Figure5), we can clearly notice that large number of tie points make processing time exponential while the precision of 3D points does not improve significantly. This means that while all image observations are accurate with same precision, a huge number of 3D points is not really necessary for bundle adjustment, whereas more attention should be paid to point position to guarantee a uniform distribution in the images.
(a) (b) Figure 6. Achieved RMSE of 3D points when location uncertainty is changing. UAV straight block with 24 images (large baselines -a) and 48 images (short baselines -b).
(a) (b) Figure 7. Achieved RMSE of 3D points when location uncertainity is changing. Closed block with 13 images (large baseline -a) and 26 images (short baseline -b).
(a) (b) Figure 8. RMSE of 3D points in the distribution analysis. Closed block with 24 (a) and 48 (b) images.
(a) (b) Figure 9. RMSE of 3D points based on thetie points multiplicity analysis: straight (a) and closed (b) image block.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition)

Influence of tie points location accuracy
We run further tests changing the accuracy of tie-points' location (from 0.2 pixel to 2 pixels) and check the influence on the RMSE of the available check points. As shown in Figure 6a, the precision of the 3D points is negatively affected by increasing the location uncertainty of tie points. But increasing the number of tie points helps in improving RMSE values. The same behaviour is found when using 48 images (with short baselines), as shown in Figure 6b. Similarly, for a closed block with 13 or 26 images, illustrated in Figure 7a-b, as the location accuracy of tie points increases, the precision of 3D points also increases and adding more images help to reduce errors in the 3D space. So, increasing the number of images and observations provides for better precisions in the 3D space. It could be concluded that the accuracy of tie point observations is no more an issue because lower accuracy of the observations can be compensated by increasing the amount of observations. However, it is important to know the accuracy of the measurements in order to give them correct weights in the block adjustment.
The evaluation results of location accuracy impact on the result of bundle adjustment could be useful when automatic keypoint extraction techniques are used. Automatic keypoint extraction algorithms typically construct a Gaussian image pyramid, by convolving the original image with a sequence of Gaussian kernels with different width. Neighbouring images in the pyramid are then subtracted from each other to obtain the multiscale DoG pyramid. Strong local extrema in that scale-space are selected as keypoints and are applied in matching process. Lowresolution tie points are based on low frequency gradients and thus will in most cases have a higher location uncertainty than high-resolution ones.

Tie points distribution
As described in the previous sections, the number of tie points is not a problem but the distribution of tie points in the overlapping areas of the adjacent images is more important. In challenging photogrammetric blocks, where there are few small matchable objects in image, spatial distribution of tie points becomes important. This may result to have the majority of the tie points located either in a small part of the image or in a linear alignment.
To evaluate the distribution influence, the approximate 3D model was defined in such a way that re-projected points provide different distributions on the images. The distribution of reprojected points on the image can be evaluated using the global coverage index (α). The global coverage index is computed based on Voronoi diagrams as the follows: In Equation (5), Ai is the area of the ith Voronoi cell, n is the number of Voronoi cells and ATotal is the area of the whole image. The larger the α value, the better the spatial distribution of the matched pairs. As shown in Figure 8, using a fixed number of tie points, as the tie points' distribution increase, better results in term of RMSE of 3D points can be achieved. With an average of 400 tie points (Figure 8a), the increasing in the distribution from 50% to 80% in each image, significantly improves of the RMSE of 3D points from 1.05mm to 0.40mm. Increasing the number of tie points to 800, provides a much better situation and improves the RMSE of from 1.66mm to 0.27mm. Similar results can be obtained using larger number of tie points ( Figure 8b): with 5000 tie points, the RMSE enhances from 0.023mm to 0.012mm when the distribution of tie points increases from 50% to 80%, whereas with 8000 tie points, the RMSE are slightly better. It can be said that when the distribution is uniform, a very stable block geometry can be achieved.

Tie points multiplicity
Tie point multiplicity indicates the number of images contributing to the calculation of a 3D point, i.e., the number of images where the point has been measured. Therefore, the multiplicity value refers to the excess of image observations with respect to the number of unknown 3D object coordinates, estimated within the adjustment step. Assuming a good intersection angle, the higher the redundancy (and consequently the multiplicity), the better is the quality of the computed 3D points. Figure 9 presents the RMSE of 3D points in condition of different tie point multiplicity for both straight (a) and closed (b) block. As it shows, high multiplicity values suggest greater precision of the computed 3D tie points, considering that multiple intersecting rays contribute to the point position check. For a straight block (Figure 9a), increasing the multiplicity of tie points improves RMSE of 3D points despite the point number reduction from 2580 to 1700. Similar results are obtained in a closed block image sequence (Figure 9b): increasing the multiplicity of tie points from 2 to 18, improves the precision of 3D points from 0.18mm to 0.13mm. The tie point multiplicity evaluation results demonstrate the importance of several intersecting rays contributing to the 3D point measurement. Although the importance of a large number of multi-ray points is not as great as it is in analytical photogrammetry, ignoring this factor can result in extreme block deformations (dome effect) and inaccurate results.

Real data
We have evaluated the impact of the tie points number and distribution on the quality of image orientation and 3D point estimation also using real UAV datasets. To this end, two photogrammetry blocks including multiple strips (18 images) and a convergent block (9 images) were considered. In both cases, images were captured with a DJI FC6310 camera (5472×3648 image size) coupled with a 8mm focal length. Figure10 shows the employed datasets with sample images and camera networks. We considered four different situations, manually adjusting the average tie point distribution in the image from 75% to 45% and analysing the results of the bundle adjustment. As metrics, number of tie points per image, average re-projection error in the bundle adjustment, average number of rays per 3D point and average intersection angles per 3D points were used for comparisons and analyses. The open source toolbox DBAT (Murtiyoso et al., 2018) was used to run the photogrammetric image orientation process in MATLAB using the Levenberg-Marquardt algorithm (Moré, 1978). Results are shown in Figures  11,12,13 and details are discussed in the following sections.

Average re-projection error
The bundle adjustment results for both scenarios, including average re-projection error of computed 3D points and number of tie points are shown in Figure 11 and Figure 12. As shown in Figure 11a and Figure 12a, the average re-projection error values for each image decreased as the number of tie points and their distribution increases. The best results is achieved with an average distribution of 75% and the weakest results are obtained when tie points are not well-distributed (45%). When tie points are accumulated in a small region of the image, increasing the number of tie points does not improve good bundle adjustment results. From these results it can be concluded that particular consideration should be taken to ensure a uniform tie point distribution in the images rather than using large number of tie points.    3.2.2 Average number of rays per 3D point As the overlap between images in a dataset increases, more redundancy is expected for 3D object coordinates. As shown in Figure 13a, higher average multiplicity for the tie points is achieved in convergent image sequence with higher overlapping images. As the distribution of tie points in image increases, the higher average multiplicity is achieved. This is mainly because 3D object points are visible in more images and are calculated more accurately.

Average intersection angle per 3D point
Since 3D points are calculated by triangulation, a higher angle of intersection of similar rays provides more accurate 3D details. As shown in Figure 13b, higher intersection angles are achieved in convergent image sequence consisting of higher overlapping images. As the distribution of tie points in image increases, the higher average intersection angle is also achieved.

CONCLUSIONS
Image orientation is nowadays a fully automated procedures even without coded targets. Current hand-crafted feature extraction and matching algorithms extract a large number of keypoints depending on the image's information content. Furthermore, in large-scale UAV blocks with a large number of images, a high number of tie points is produced by the large number of matched keypoints. In this regard, tie point filtering/selection approaches are commonly used to improve bundle adjustment precision and accelerate its efficiency in SfM technique. There exist different important and influential factors in tie point filtering/selection which affects the bundle adjustment results. In this paper, the effect of influential factors for tie point filtering were analysed based on simulated and real data. To this end, different image networks were considered, and 3D points are re-projected on the image planes. The bundle adjustment is run using inner constraints and the covariance matrix of unknowns were analysed. The results showed that increase in the amount of tie points improve the quality of bundle adjustment results, however using too many tie points do not improve the results significantly but increase the required processing time of bundle adjustment. Furthermore, the bundle adjustment quality is improved as the location uncertainty of the tie points decreases. The evaluation results also showed that the distribution of tie points located in the overlapping areas of adjacent images is an important factor besides the number of tie points. The multiplicity results showed that high multiplicity values of tie points provide greater precision for the computed 3D tie points. An ideal tie point filtering algorithm should therefore consider all the influential factors for tie point selection such as tie point number, distribution, multiplicity and location accuracy simultaneously. Each of the aforementioned factors are important to select a subset of high quality tie points. To this end developing an approach that uses all important factors at the same time is suggested as future work.