TIN-BASED ROBUST MOSAICKING OF UAV IMAGES WITH CONSECUTIVE IMAGE CONNECTION

: UAV image mosaicking could be divided into processing in an absolute space and in a relative space. Among them, the latter technique generates a mosaicked image using only UAV’s EOP (Exterior Orientation Parameters) without additional spatial data. Since this technique uses only images, it is easy to use, but sometimes exhibits unstable quality. In this study, we adopted the mosaicking in a relative space by introducing the TIN (Triangulated Irregular Network) structure for consecutive image connectivity. This method refines a point cloud and extracts optimal images for mosaicking as a pre-processing. After that, this generates a TIN, and performs triangular unit mosaicking and image unit mosaicking to produce a mosaicked image. We evaluated the proposed method with datasets acquired by fixed wing and rotary wing UAVs. We compared this method’s results with Pix4Dmapper’s and the previously developed method’s results. Through the experimental results, we were able to verify that this method was less affected by external factors such as ground topology and UAV status. Moreover, we could confirm this method reduced over-transformation, and mismatch of the junction area between images on mosaicked images. It showed stable quality despite of using only images. Through the above results, our method also showed similar performance to Pix4Dmapper.


INTRODUCTION
UAV (Unmanned Aerial Vehicle) remote sensing has strengths in work convenience and utility. It is currently being actively used in various fields and regions, such as agriculture monitoring, urbans mapping, and disaster prevention . UAVs fly at extremely low altitudes compared to satellites and aircraft. Since they acquire a lot of images in this process, it is necessary to generate a mosaicked image to improve usability of the images. UAV photogrammetry uses image points, and estimates internal and external camera parameters using collinearity condition. The parameters can represent camera positions corresponding to each image and a sparse point cloud with 3D coordinates on target area. After that, depending on each purpose and method, it performs different processes such as mosaic production, point cloud densification, and 3D model generation. The UAV photogrammetry is a representative tool for UAV image processing dealing with geometry, and there is commercial software such as Agisoft's Metashape and Pix4D's Pix4Dmapper. UAV image mosaicking could be divided into processing in an absolute space and in a relative space. The former technique produces a mosaic by using spatial data such as DSM (Digital Surface Model) or GCP (Ground Control Point), and includes preparation or production of these data in a pre-processing step. The result is projected onto a ground model along with the reference data. Firstly, DSM-based mosaicking performs the processes of seamline detection, image search for grid, and ortho rectification (Maoteng et al., 2018). This method can be seen as a representative absolute method. It usually has the best quality; however, disadvantages lie in that the errors are propagated from the DSM. Additionally, it requires a lot of DSM manufacturing cost. Despite research has been conducted to improve accuracy * Corresponding author and reduce the cost Song et al., 2018;Yuan et al., 2020), there are still many limitations to its practical application. Secondly, GCP-based mosaicking utilizes GCP obtained through GNSS (Global Navigation Satellite System) surveying, etc., and conducts the process of GCP-based geometric correction and image stitching (Yi et al., 2021). This method has a lower work complexity for reference data manufacturing than DSM-based mosaicking, but still requires a lot of the cost. Moreover, its preliminary process is essential since the ground and image coordinates of GCP are required (Sanz-Ablanedo et al., 2018). The latter technique generates a mosaic using only UAV's EOP (Exterior Orientation Parameters) without additional spatial data. Although this technique may have relatively low accuracy, it is the most common because only UAV images are used. (Kim et al., 2020). This method stitches images to a relative model space using EOPs corrected through bundle adjustment or acquired with high-precision sensors. Related research mainly has been dealing with advanced bundle adjustment and seamline extraction technique in mosaicking to improve its quality (Xie et al., 2018;Zhang et al., 2018). Nevertheless, it sometimes shows unstable performance because of the absence of reference. In this study, we adopted the mosaicking in a relative space by introducing the TIN (Triangulated Irregular Network) structure for its stabilization. TIN is a model that reproduces terrain using only triangular elements, and characterized by connectivity (Park et al., 2010). Through TIN-based mosaicking development, we tried to take its advantage and increase the robustness of mosaicking. We tested the proposed method with datasets acquired by fixed wing and rotary wing UAVs. We compared this method with Pix4Dmapper, a commercial software, and the previously developed method (Kim et al., 2017) called image-based mosaicking, and tried to confirm improvements.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France In experiments, we used images taken for three target areas as shown in Figure 2. We constructed experimental cases according to target area, and the precision of mounted positioning sensors. Table 1 shows UAVs used, and Table 2 describes the experimental cases. Table 3 describes information of datasets taken according to Table 2. Figure 1 is the flowchart of the proposed method in this study. Firstly, this method generates tie points through feature matching and corrects lens distortion using them. Then, it renews it into a point cloud through block adjustment and makes an optimal image list for mosaicking and selects point cloud used for each image. Finally, it generates TINs and operates two steps of mosaicking to produce a mosaicked image. Details are described in the following subsection.   Table 3.

Descriptions of the dataset information
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France

Point Cloud Management
We extracted the tie points with SURF (Speeded Up Robust Features) method (Tareen and Saleem, 2018), and used the number of successful matches for each as a performance indicator of a point cloud. In this study, we adopted the points with an index of four or more and performed lens distortion correction, block adjustment, and point cloud generation. We first corrected lens distortion using the tie points. We set the initial values with the parameters of lens distortion provided by manufacturer. Then, we estimated the optimal parameters by applying LSE (Least Square Estimation) with collinearity condition, and reduced lens distortion with the parameters. Next, we applied block adjustment to the bundle adjustment process. This block adjustment is a photogrammetry technique that simultaneously corrects EOP and determines the ground coordinates of tie points. Even though the extracted point cloud was sparse, we could confirm that it represented the exact location in a model space as Figure 3 and 4. As shown in Equation (1), we constructed a model for ground coordinates of tie points and EOPs based collinearity conditions. Then, we corrected them with general LSE.
[̇0 0 where ̇, ̈ = coefficients of partial differential equations for EOP and ground coordinates of tie points in collinearity conditions = coefficients of identity matrix ∆̇, ∆̈ = increments for EOP and ground coordinates of tie points = differences between observed and initial values for collinearity equations ̇, ̈ = differences between observed and initial values for EOP and ground coordinates of tie points = residuals for collinearity equations ̇, ̈ = residuals for EOP, ground coordinates of tie points  The closer from nadir to oblique image, the greater the geometric distortion of the image. For this reason, recent UAVs are equipped with gimbals to acquire datasets like nadir images. Nevertheless, depending on the weather environment such as windy and the type of UAV such as fixed-wing and low-cost, the UAV sometimes may acquire dataset like oblique image.
To minimize distortions in a mosaicked image, we needed to search for image combinations that generate less distortion with minimal images (Lim et al., 2021) as Figure 5. We first calculated directions of UAV flight paths as Equation (2) and (3), and separated images for each strip.
where +1 = azimuth from i-th to (i+1)-th image , = coordinates of the starting image , = coordinates of the i-th image We then set the first image for each strip as the reference image, and selected the images so that the overlap was around 30%. Finally, we calculated the verticality of the selected images with Equation (4), and sorted them in ascending order. Through this, we tried to minimize the potential for relief displacements. We tried to keep minimization of over-transformation and mismatch of the junction area between images on the mosaic image. Figure 6 is a detailed processing diagram of mosaicking. We refined a point cloud and TIN according to their quality, and divided the inner and outer regions of the TIN structure. After that, we applied TUM (Triangle Unit Mosaicking) to the interimage junction areas and IUM (Image Unit Mosaicking) to the non-junction areas. Finally, regarding the missing areas of mosaicking, we re-arranged the point cloud and TIN, and supplemented them with the IUM.

Figure 7. Example of a TIN generated
Since a TIN model is an irregular data type, a TIN management method determines the performance of the proposed method. Before generating the TIN, we extracted the point cloud for each image and checked the reprojection error. In this process, we projected the point cloud onto the images based collinear equations, and selected their inliers with three pixels as a threshold. We set the domain of a mosaicked image by projecting the domain of point clouds onto the zero-height plane. In this domain, we generated a TIN based on Delaunay triangulation using only tie point-based point cloud, and applied segmented processing to it for the efficiency improvement. Figure 7 is an example of TIN generated in the proposed method. The results showed that small and large triangles were defined from a point cloud included in one image. They also showed that sharp acute triangles were distributed mainly in the outer region. We divided the TIN into three types and mosaiced them in separate ways as shown in Table 4. We excluded triangles in the outermost region and included triangles in the rest of region from our operation.  TUM is based the affine transformation as in Equation (5), and creates a mosaicked image by warping triangular image patches on outer region. IUM is based the homography transformation as in Equation (6), and stitched the entire patches on inner region to the mosaic image.  Figure 9 is an example of a primary mosaicking result. This result showed that there were few errors in the junction area between images while keeping the state of over-transformation minimization. However, considering the holes in the result, additional processing for supplementary mosaicking was required.
In this process, we used all images, not just the optimal images, for mosaicking. We calculated the image coverages on the ground through forward mapping and checked the overlap between it and areas required supplement. Based on its degree, we sorted the image order of the supplementary mosaicking.
According to re-optimal image list, we extracted point cloud to be used, and supplemented the mosaic image in the same way as the IUM. Figure 10. supplemented area on the result Figure 11. Result of the proposed method Figure 11 is result of the proposed method. We confirmed that the holes were resolved successfully through supplementary mosaicking as Figure 10.

RESULTS AND DISCUSSION
For the experimental cases presented in Table 1 and 2, we firstly performed mosaicking with five datasets. We summarized the TIN and mosaicked images of the proposed method, including Pix4Dmapper's and previously developed method's mosaicking results in Table 5, 6 and 7. Table 5 and 6 show TIN generation results of the proposed method. This result showed that our method was capable of evenly tie point extraction and TIN generation throughout the target areas. Table 7 and 8 show the improvement compared to the previously developed method. The previous method is an image-based mosaicking and has been improved mainly with advanced bundle adjustment. However, it showed sometimes unstable performance depending on the ground topology or flight condition. In contrast, our proposed method kept the performance regardless of these. Furthermore, we could confirm comparable performance when compared with Pix4Dmapper. Agriculture is one of the fields where UAV images are widely used. For the case 1, we checked the results within datasets 3, 4, and 5. We analysed the results with farmland datasets and confirmed our method's applicability in this field. The results showed that almost no errors occurred. It could suggest that this method can be applied sufficiently to agriculture. For the case 2, we compared the qualities according to regional difference within datasets 2 and 5. This experimental case showed better results. The results meant that the quality of proposed method was stable regardless of target area. For the case 3, we confirmed the quality difference in sensor precision within datasets 1 and 2. This experimental case also showed better results from the proposed method and indicated that the quality was not significantly affected by UAV.

CONCLUSIONS
In this study, we proposed a TIN-based mosaicking method emphasizing consecutive image connectivity. This method refines a point cloud and extracts optimal images for mosaicking as a pre-processing. After that, it generates a TIN, and performs triangular unit mosaicking and image unit mosaicking to produce a mosaicked image. We showed the results of the proposed method in three experiments, and checked the superior quality in all cases. We were able to verify that our method was less affected by external factors such as target area and UAV status. Moreover, we could interpret that our method reduced over-transformation, and mismatch of the junction area between images on the mosaic image. It showed stable quality despite of using only images. Through the above results, our method showed similar performance to Pix4Dmapper, a representative commercial software. We expected that our method could be applied to various technologies. As a future experiment, we plan to develop advanced supplementary mosaicking using pseudo-DSM. Through this, we expect to further enhance the robustness and expandability of this method.