ASSESSING THE ACCURACY OF GEOREFERENCED POINT CLOUDS FROM UAS IMAGERY

Unmanned Aircraft System (UAS) mapping methods determine the three-dimensional (3D) position of surface features. UAS mapping is often used, compared to traditional mapping techniques, such as the total station (TS), Global Navigation Satellite System (GNSS) and terrestrial laser scanning (TLS). Traditional mapping methods have become less favourable due to efficiency and cost, especially for medium to large areas. As UAS mapping increases in popularity, the need to verify its accuracy for topographic mapping is evident. In this study, an assessment of the accuracy of UAS mapping is performed. Our results suggest that there are many factors that affect the accuracy of UAS photogrammetry products. In specific, the distribution and density of ground control points (GCPs) are particularly significant for a study area of 2.861 km2 in size. The best results were obtained by strategizing the distribution and density of GCPs; where, the root mean square error (RMSE) for the X, Y, Z, and 3D was minimized to 0.012, 0.021, 0.038, and 0.045 meters, respectively, by applying a total of 15 GCPs in the aerotriangulation. Therefore, it may be concluded that UAS photogrammetric mapping can meet sub-decimeter accuracy for topographic mapping, if proper planning, data collection and processing procedures are followed.


INTRODUCTION
Topographic mapping supports civil, construction, and environmental engineering applications (Lui et al., 2014). Current topographic mapping techniques include the Global Navigation Satellite System (GNSS), Light Detection and Ranging (LiDAR), and total station (Mitasova et al., 2004). However, these methods are time consuming, labor intensive, and costly. For these reasons, new techniques are needed. In the last decade, the advancements made in topographic mapping techniques offer a unique opportunity to map and model surfaces at unprecedented scales (Kennie, 2014). In particular, the improvements made in Unmanned Aircraft Systems (UASs) coupled with vision-based systems and computer vision algorithms provide new opportunities for collecting, processing, and reconstructing the three-dimensional (3D) position of surface features (Colomina and Molina, 2014). UASs fused with vision-based systems provide unique advantages for noncontact, high temporal, and spatial resolution data Mora et al., 2019). The improvements made in both UASs, and vision-based systems have caused UASs to become an affordable and flexible option for topographic mapping. The eased restrictions on commercial drone use from the Federal Aviation Administration (FAA) has led to a surge in research and commercial services involving lightweight UAS (Love et al., 2015). As the advantages of UAS become clear, the adoption of UASs increases in parallel (Vincenzi et al., 2014). Currently, many UASs equipped with vision-based systems have been developed for topographic mapping (Lu et al., 2018;Opromolla et al., 2018). Applications include stockpile estimation, 3D modeling, planimetry extraction, and topographic mapping (Sanfourche et al., 2012;Alsalam et al., 2017;Mora et al., 2020). Ongoing developments for UASs are based on photogrammetry and computer vision algorithms (Dandois and Ellis, 2013;. Other studies focus on the impact of flying height, overlap, environmental conditions and optical sensor models (Seifert et al., 2019). Although the results from prior studies are promising, a study focused on the performance of a UAS coupled with a vision-based system for medium to large areas is not well understood. In this study, the results of an experimental evaluation to assess the accuracy of UAS photogrammetry for topographic mapping for medium to large areas is performed. Imagery is collected by using a Phantom 4 Pro v2.0 over a test site having 25 Ground Control Points (GCPs) covering an area of 2.861 km 2 . Subsequently, the imagery is processed using Bentley ContextCapture. Then, the accuracy is evaluated by using the checkpoints to compute the root mean square error (RMSE) for the X, Y, Z, and 3D. The results observed demonstrated that RMSE values up to 0.012, 0.021, 0.038, and 0.045 meters for the X, Y, Z, and 3D, respectively, can be achieved when a total of 15 GCPs are applied in the aerotriangulation. Therefore, it may be concluded that UAS photogrammetric mapping can meet subdecimeter accuracy for topographic mapping, if proper planning, data collection and processing procedures are followed.

MATERIALS AND METHODS
The procedure used to evaluate the photogrammetric mapping accuracy of UAS surveys is summarized in Figure 1.

Study Area
The study area is in Spadra Farm (Pomona), Southern California, United States of America (Latitude: 34° 02′ 21.73" N, Longitude: 117° 48′ 59.44" W). The study site has an area of 2.861 km 2 . The selection of the study site was based on its morphology, which is primarily an agricultural field. The elevation range is 47.78 meters, varying from 189.60 to 237.38 meters in the North American Vertical Datum of 1988. The study area has some vegetation from the crops in the agriculture field as shown in Figure 2.

Image Acquisition
The images were acquired from a Phantom 4 Pro v2.0. The UAS was equipped with a camera sensor of 20 megapixels (5,472 × 3,648) and a mechanical shutter. The flight altitude was 120 meters above ground level, which implies an equivalent ground sample distance of 34.2 mm/pixel. The shutter speed was adjusted based on flight altitude, UAS speed, and lighting conditions at flight time to minimize image blurring. The mission was carried out autonomously using the software DroneDeploy, where a total of 13 flight lines and 1,051 images were acquired. The flight was set to obtain a forward and side overlap of 80 and 70 %, respectively.

Ground Control Points
Before the image acquisition took place, 25 GCPs were set around the project site to assess the UAS survey's accuracy. Some of the GCPs will be used as checkpoints, while the others will be used as GCPs in the aertotriangulation. The threedimensional coordinates of the GCPs were measured with a GNSS rover in Real-Time Kinematic (RTK) mode, with the base station located within the project site. The horizontal coordinates were processed to the California State Plane Coordinate System Zone 5, while the vertical was in the North American Vertical Datum of 1988. Both base and rover systems were Trimble R10 systems. Given that the base was within the project site, both the horizontal and vertical errors were ±2 cm.

Photogrammetric Processing
The photogrammetric process was carried out using Bentley ContextCapture, update 17 -v10.17.0.39. This photogrammetric software is based on the structure-from-motion methodology. The workflow follows a four-step process. The first step is to import the imagery and GCPs/checkpoints. The second step is to perform the photo ID to identify all GCPs/checkpoints in all corresponding images. The third step is to align the images by automatic feature identification and matching. The software simultaneously estimates both the internal and external parameters, including radial and tangential distortion. The result of this step is the camera position corresponding to each image, the internal camera calibration parameters, and the 3D coordinates of a sparse point cloud of the terrain. The final step is to apply texture to the mesh. In general, the bundle adjustment can be carried out using a minimum of three GCPs; however, better results are obtained using more GCPs, which is recommended to achieve the best accuracy.

Accuracy Assessment
The accuracy of all photogrammetric projects was evaluated using the checkpoints that were not used for georeferencing, using the root mean square error (RMSE) formulation. To this end, the checkpoints were identified in the point clouds, and their corresponding surveyed GNSS coordinates were compared, resulting in RMSEX, RMSEY, RMSEZ, and RMSE3D, as defined in equations 1 -4 (Chai & Draxler, 2014).
where, n = number of checkpoints tested for each project Xi, Yi, Zi, and 3Di = coordinates estimated from the bundle adjustment for the i th checkpoint. XGNSS, YGNSS, ZGNSS, and 3DGNSS = coordinates measured with GNSS for the i th checkpoint.

Experimental Tests
To evaluate the impact of the density and distribution of the GCPs, twelve different tests were performed in the photogrammetric bundle adjustment. The twelve tests were evaluated with varying GCPs ranging from 4 -15 and checkpoints ranging from 10 -21. The location of the GCPs and checkpoints used in the experimental tests are shown in Figure 3.

RESULTS
The results are summarized in Table 1 and Figure 4 for all twelve tests. As a general remark the accuracy is significantly influenced by the number of GCPs applied in the aerotriangulation, especially for a test site of 2.861 km 2 in size. Although one would expect the accuracy performance to increase gradually by increasing the number of GCPs, an optimal performance level will be reached. At its best performance, the RMSE values will not improve significantly by adding additional GCPs in the aerotriangulation. On the contrary, the additional effort needed to add an additional GCP to the aerotriangulation may outweigh the benefits of the additional GCP (e.g., planning, data collection, and processing). For these reasons, it is critical to know the desired accuracy needed prior to project planning to determine the optimal number of GCPs needed to achieve this goal. As shown in Table 1 and Figure 4, the overall error decreases as the number of applied GCPs increases. This supports the notion that higher levels of accuracy can be reached by using more GCPs, as suggested by earlier studies (Martínez-Carricondo et al., 2018;Sanz-Ablanedo et al., 2018). There is a notable exception to this observed in our testing. The implementation of 9 GCPs produces lower RMSE values than 10, 11 or 12 GCPs in the aerotriangulation. One potential justification for this could be the visual quality of the GCPs in question (Montazeri, S. et al., 2018). The issue of visibility does not necessarily mean that a physical obstruction prevents accurate readings of the GCP but could also include light level, humidity, and air quality (Bałazy et al., 2019). Another potential explanation revolves around the distribution of the GCPs being used in higher concentrations in a particular area (James, M. R. et al., 2017;Guntel, A. et al., 2018). Shown in Table 1 and Figure 4 is a large RMSEZ value of 4.721 m when applying four GCPs in the aerotriangulation. This issue is evident for an area 2.861 km 2 in size, where the bundle adjustment may not be able to properly solve for the camera calibration parameters (Zheng et al., 2015). Preferably, the camera calibration parameters should be approximated in a laboratory environment, however, these parameters often change from flight-to-flight (Zhou et al., 2020;Lim et al., 2019). Another issue with only applying four GCPs may be due to the density and distribution of the GCPs, where there is no single GCP available to support the bundle adjustment for a span of about 1.610 km. This issue can be resolved by adding an additional GCP in the middle of the study area as shown in Figure  3. The RMSEZ value is significantly improved by 4.615 m to 0.106 m.

Table 1. Summary of Global RMSE Results
This study provides an independent evaluation of the performance of the bundle adjustment since a different number of GCPs and checkpoints were tested. The GCPs were distributed throughout the study area as shown in Figure 3. The density and distribution of the GCPs for a site with agricultural morphology is critical to improve the performance of the bundle adjustment and to minimize the RMSE values for a medium to large project area. The differences between the UAS-derived point cloud and GNSS RTK measurements were investigated. The results observed feature the quality of the accuracy of the UAS point cloud. These results are stable and provide confidence that a quality high-resolution point cloud from UAS photogrammetric surveys is possible when proper procedures are followed.

CONCLUSIONS
As a result of the above analysis, it is essential to prepare a comprehensive study of the number of the GCPs to exploit the accuracy of photogrammetric projects. To obtain the best RMSE in X, Y, Z, and 3D, the distribution and density of the GCPs must be placed strategically. As the project size increases, the number of GCPs will increase until the desired RMSE accuracy results are achieved. When planning a UAS photogrammetric survey, sufficient GCPs must be distributed and placed strategically throughout the project site. However, this may be a challenge due to access to a site or dangerous site conditions. For these reasons, it is critical that during planning, the anticipated RMSE is well understood given the density and distribution of the GCPs. In this study, the best results were obtained by minimizing the RMSE for the X, Y, Z, and 3D to 0.012, 0.021, 0.038, and 0.045 meters, respectively, by applying a total of 15 GCPs in the aerotriangulation. Therefore, the results support that UAS photogrammetric surveys can meet sub-decimeter accuracy for topographic mapping, if proper planning, data collection and processing procedures are followed.