DIGITAL SURFACE MODEL DERIVED FROM UAS IMAGERY ASSESSMENT USING HIGH-PRECISION AERIAL LIDAR AS REFERENCE SURFACE

Imagery captured from aerial unmanned systems (UAS) has found significant utility in the field of surveying and mapping as the efforts of the computer vision field combined the principles of photogrammetry. Its respectability in the remote sensing community as increased as the miniaturization of on-board survey-grade global navigation satellite system (GNSS) signal receivers has made it possible to produce high network accuracy contributing to effective aerotriangulation. UAS photogrammetry has gained much popularity because of its effectiveness, efficiency, economy, and especially its availability and ease of use. Although photogrammetry has proven to meet and exceed planimetric precision and accuracy, variables tend to cause deficiencies in the achievement of accuracy in the vertical plane. This research aims to demonstrate achievable overall accuracy of surface modelling through minimization of systematic errors at a significant level using a fixed-wing platform designed for high-accuracy surveying with the eBee Plus and X models by SenseFly equipped with survey-grade GNSS signal-receiving capabilities and 20MP integrated, fixed-focal length camera. The UAS campaign was flown over a site 320 m by 320 m with 81 surveyed 3D ground control points, where horizontal positions were surveyed to 1.0 cm horizontal accuracy and 0.5 cm vertical accuracy using static GNSS methods and digital leveling respectively. All AT accuracy was based on 75 independent checkpoints. The digital surface model (DSM) was compared to a reference DSM generated from high-precision manned aerial LiDAR using the Optech Galaxy scanner. Overall accuracy was in the sub-decimeter level vertically in both commercial software used, including Pix4Dmapper and Agisoft Metashape.


INTRODUCTION
Unmanned aerial systems equipped (UAS) equipped with cameras are able to create 3D point clouds for precise modeling for large area mapping and reconstruction using the principles of photogrammetry. In practice, aerial photogrammetry remains an established and effective method to for collecting accurate and precise planimetric and elevation data. Aerial LiDAR has become another method of effectively collecting elevation data and has some advantages over aerial photogrammetry including the ability to collect data outside of optimal sunlight conditions and multiple pulse returns allowing penetration through vegetation to capture ground points. Barring the ability to penetrate vegetation to capture ground and fly outside of optimal sunlight and sun angle, obtaining network-accurate and precise elevation data through use of UAS imagery capture is a highly, efficient, and economic method. The purpose of this study is to compare, and therefore validate, the collection of elevation data using 20MP resolution imagery captured from UAS equipped with on-board, survey-grade global navigation satellite system (GNSS) capabilities. The elevation data will be assessed in the form of digital surface model (DSM) of unobstructed, solid surfaces including paved asphalt road and rooftops generated from imagery-based point cloud computed through two popular commercial structure-from-motion software for processing UAS imagery, Pix4Dmapper and Agisoft Metashape.
There is a high saturation of studies throughout the literature demonstrating proof-of-concept of using UAS imagery for high precision mapping using surveyed ground control for network orientation (Agüera-Vega, Carvajal-Ramírez, & Martínez-Carricondo, 2017;Eisenbeiß, 2009;Eisenbeiss & Sauerbier, 2011). Consequently, point clouds and surface models have been investigated throughout the literature. However, less work has been done comparing the elevation products of point clouds and surface models produced from UAS imagery to LiDAR. There is a gap in the literature comparing DSM from UAS imagery to a DSM generated from high-precision aerial LiDAR. This is likely due to the large cost associated with planning a full-scale, manned aerial LiDAR mission. In addition, there is lack of research including a significantly large sample checkpoints as well as of flights and flight orientations using a calibration site with a highly densified control network for the assessment of digital surface models on hardscaped areas such as roofs and asphalt roads.
Various research has assessed imagery-derived point clouds and DSM's. Research has been performed to investigate the effect of flight planning, including flight orientation, on the quality of the point cloud from a sample of only 6 check points, holding 20 points as ground control (Chiabrando, Lingua, Maschio, & Losè, 2017). While the AT results were at the centimeter-level, the sample size was insufficient to ensure repeatability to achieve that level of accuracy. The quality of the point clouds were assessed from capture at different camera orientations and the point density of vertical cross sections was used as a measure of quality rather than comparison to reference surface. A study assessed imagery-derived point cloud from UAS that analyzed the effect of automatic and semi-automatic GCP measurement on the final georeferenced cloud using checkpoints surveyed by total station (Harwin & Lucieer, 2012). Anders and colleagues performed a number of flight campaigns to assess the accuracy of surface modelling finding an average vertical discrepancy in the decimeter range (Anders, Masselink, Keesstra, & Suomalainen, 2013). Although 50 GCP were used, it was unclear how they were used in the final computation of report of vertical discrepancies.
Although point clouds have been thoroughly assessed through means of checkpoint validation, elevation models derived from UAS imagery have less been compared to reference surfaces generated from LiDAR methods. A study compared UAS imagery-derived DSM to terrestrial laser scan (TLS)-derived DSM, as a reference surface, of a man-made dike and processing the imagery in Pix4Dmapper and Agisoft Photoscan (now Metashape) (Naumann, Geist, Bill, Niemeyer, & Grenzdörffer, 2013). The comparison in this case applied a simple DSM subtraction and an elevation heat map across the entire surface as well as various segment surfaces was analyzed. Although results were based on a large sample, there was no mention of checkpoint statistics nor of specific distribution of the 11 surveyed ground control point (GCP) distribution. Przybilla and colleagues also compared UAS imagery-based point cloud to TLS point cloud along cross-section vertical profiles (Przybilla, Lindstaedt, & Kersten, 2019). The author used Pix4Dmapper and Agisoft Metashape to process the imagery into point clouds. An aspect of this study was to investigate an effect of image format (TIF or JPG) on point cloud accuracy as well as an effect of image down sampling for keypoint extraction. The problem with this study is that the checkpoints for the point cloud were used as the control points in the AT process, therefore the results of this study are questionable. Kosmatin Fras and colleagues assessed the accuracy of UAS imagery-derived DSM for about 20 checkpoints without the use of a reference surface such as that derived from either terrestrial or aerial LiDAR (Fras, Kerin, Mesarič, Peterman, & Grigillo, 2016).
The study of this paper is an extension of an earlier study over the same site that compared UAS imagery-derived point clouds to a reference TLS point cloud (Peterson, Lopez, & Munjy, 2019). In this study, imagery was collected using a senseFly eBee Plus model equipped with survey-grade on-board GNSS collection capabilities. From these data a sparse point cloud was adjusted by AT and validated to be a 2-3cm horizontally accurate and 4 cm vertically accurate according to statistics based on 75 checkpoints surveyed by static-GNSS methods with elevations resolved by digital leveling. A vertical profile 1 cm-thick was extracted from both UAS imagery and TLS point clouds along 200 m of asphalt road and compared. An unresolved vertical bias of nearly 15 cm was found. Planar regression was applied to point samples on roof faces for both point clouds and compared where the vertical bias was not present such as was found along the asphalt road.
In the study of this paper, a high-precision, manned aerial LiDAR mission was performed over a site located in the foothills of the Sierra Nevada Mountain Range located in Fresno County for use as reference data for comparison. The site was densified by 81 high accuracy ground control points with horizontal positions established to 1 cm using static GNSS method and vertical positions to 0.5 cm using digital leveling. AT results were optimally established in the sparse point cloud before being densified and converted to DSM for surface comparison. Orthophoto generation was performed for the purpose of checkpoint location in the DSM for checkpoint validation of the surface.
Cross-sectional profiles were extracted from all DSM's, including the reference DSM from aerial LiDAR, along asphalt roads bisecting the site and along rooftops. Along with these comparisons, results were compiled and tabulated for flying height and flight line orientation.

Site
A calibration site 320m by 320m located about 32km north of the campus of California State University, Fresno in the foothills of the Sierra Nevada mountain range was designated for this study. The site has rolling hills topography with a control point elevation range of about 20m.

Control
Control points (80) were surveyed horizontally using static GNSS methods, observed with simultaneous setups for 8 hours on multiple occasions producing a final accuracy of 1.0cm after adjustment. The vertical positions were resolved by differential leveling producing a final accuracy of 0.3cm. A total of 80 control points were spaced approximately 40m in a 9-by-9 square grid distributed throughout the site (Figure 1). The ground targets were designed black and white, circular, 42cmdiameter ( Figure 2).  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-M-2-2020, 2020 ASPRS 2020 Annual Conference Virtual Technical Program, 22-26 June 2020

Aerial LiDAR Flight Campaign
A high precision aerial LiDAR scan (ALS) flight was performed over the site by manned-helicopter using an Optech Galaxy scanner. The ALS elevation product was used as a reference surface for comparison. Table 1 summarizes the flight parameters and network accuracy of this flight.

UAS Hardware
Two models of fixed-wing aircraft, the SenseFly eBee Plus (Plus) and eBee X (X), were used for this research. Both UAS platform models were equipped with survey-grade, dualfrequency (L1, L2) GNSS receiver antennas and are able to resolve positions by the method of post-processing-kinematic (PPK). The optical sensor integrated into both eBee models (S.O.D.A.). The optical sensor integrated into the X model has oblique-angle image capture capability, which was not used in this research. The S.O.D.A. camera has a global mechanical shutter, CMOS sensor, 20MP sensor resolution, 2.4µm pixel size, and a published focal length of 10.6mm. More specifications of the camera are given in the Table 2.   Table 2. Sensefly eBee UAS platform hardware specifications.

UAS Flight Campaigns
A total of 3 flights were performed on separate dates. Two flights were performed with the Plus model at different flying heights, and one flight with the X model. Table 3 summarizes the flight plan parameters including the number of lifts required by the platform to cover the site, the aircraft model, flying height above ground level (AGL), ground sample distance (GSD), the cardinal direction, number of photos, average overlap (O) and side overlap (S), and wind speed and direction. In addition, the flight code is used to distinguish the flights by flying height AGL and flight path direction. The Plus was flown on two separate dates (Flights A, B) and the X once on another date (Flight C). Each of the Plus flights were performed 2 times, once in an east-to-west (EW) and another time in a north-to-south (NS) flight-line orientation. A third flight line configuration was studied where both orientations were combined to form perpendicular flight-lines.

Software
Agisoft Metashape and Pix4Dmapper were used in this research. Both software were among the most popular commercial sfm software used for processing UAS imagery at the time of this study.

Software Parameters
Images were matched and aligned with image sizes at full scale resolution.
Aerotriangulation was performed using a The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-M-2-2020, 2020 ASPRS 2020 Annual Conference Virtual Technical Program, 22-26 June 2020 constrained solution with 5 GCP and accuracy was assessed using the remaining 75 control points as checkpoints (CP) prior to point cloud generation. The 5 GCP layout is given in Figure  1. Exterior orientation positions were initially assigned a priori uncertainty values of 2 to 3cm horizontally and vertically according to system accuracy. An adjustment was performed and uncertainty values were edited to maintain residual errors within two times the uncertainty. Self-calibration was performed during adjustment similarly in both software adjusting focal length (f), principal point (cx, cy), three coefficients for symmetric radial distortion (k1, k2, k3), and two coefficients of tangential distortion (p1, p2). The AT results include only ground control points, therefore the residuals were analyzed to ensure that the adjustment was not over-or underconstrained.

Analysis
An unclassified point cloud was generated from the 5 GCP AT results. All checkpoints were omitted from the software project prior to point cloud generation to eliminate the uncertainty of any proprietary software influence on checkpoint residuals. The point cloud was generated with a density of 400 pls/m 2 and converted to a 5cm digital surface model (DSM). A 2.5cm orthomosaic was created and 2D CP coordinates were measured using ESRI ArcMap. Elevations at the 2D CP locations were interpolated in both UAS imagery and ALS DSM's using ArcMap then compared to the surveyed ground control and to each other. Several elevation path profiles were extracted along paths over roofs and roads, including a driveway, from the ALS and UAS DSM's and compared (Figure 3). Flight data were separated and analyzed at checkpoints according to flight path direction including east-west (EW), north-south (NS), and perpendicular, or cross, flight paths (EW-NS). All flight-line-oriented configurations were performed two times, once in Agisoft Metashape and once in Pix4Dmapper resulting in a sample of 14 point cloud and orthomosaic results, i.e. 7 point clouds per software. The sample of results were assessed using the root-mean-square-error statistic horizontally (RMSExy) for the orthomosaic and vertically (RMSEz) for the point cloud/DSM. However, elevation profile accuracy assessment did not include investigation into the effect of flight path direction.

Pix4D
Pix4D orthomosaic accuracy did not vary substantially by flying height nor by direction (average RMSExy = 0.9 ± 0.1cm). The differences between the maximum and minimum error shown by the range statistic was least for flight B. The average vertical error range was higher in magnitude and uncertainty (8.5 ± 3.4cm) than the average horizontal error range (4.0 ± 0.8cm). An overall average negative error vertically indicated that the adjustment was biased in the negative direction. This was true for the higher altitude but generally negative for all flights (-1.2 ± 0.7cm) ( Table 3). Table 3. Results of DEM analysis at 75 checkpoints for Pix4D.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-M-2-2020, 2020 ASPRS 2020 Annual Conference Virtual Technical Program, 22-26 June 2020

Metashape
In flights A and B, the parallel flight paths (NS A1, A2, EW B1, B2) resulted in vertical accuracy that was about 2 times better than the results of the orthogonal flight paths (e.g. RMSEz A1 = 1.3cm, A3 = 3.6cm, B1 = 1.7cm, B3 = 4.7cm). Horizontal accuracy was consistently slightly above 1.0cm (Average RMSEz = 1.1 ± 0.1cm) for all flights indicating that the effect of varying flight parameters was negligible. The average horizontal error range (6.5 ± 1.2cm) was greater than in Pix4D (4.0 ± 0.8cm), which showed a more uniform adjustment throughout the control region in this software (Table 4). Table 4. Results of DEM analysis at 75 checkpoints for Metashape.

Reference Surface:
The vertical accuracy of the high precision ALS point cloud was 1.5cm (1.5 ± 1.0cm) based on a 74 checkpoint sample. The negative average vertical error (-1.1cm) indicated a negative bias. The vertical error range was 5.7cm, making it comparable to the noise exhibited by UAS point cloud (Table 5). Table 5. DEM analysis of reference surface using data collected from Optech Galaxy LiDAR mission.

Pix4D:
Average error was for all profiles (n = 140) from DSM's generated by Pix4Dmapper including various roofs and roads was -1.0cm with a standard error of the mean of 0.1cm. The standard error of the mean was computed by dividing the standard deviation by the square root of the sample.
Overall accuracy for surfaces resulted in an average RMSE of 2.0 ± 0.7cm. Table 6 below provides a summary of the profile statistics. Table 6. Summary of elevation profile differences between reference surface and surface generated by UAS imagery for Pix4D

Metashape:
The average error of all elevation profiles from DSM's generated by Agisoft Metashape was -0.7cm with a standard error of the mean of 0.2cm using formula 1 above. Average RMSEz was slightly greater than Pix4D at 2.2 ± 1.2cm with a lesser precision. Table 7 below provides a summary of statistics. Table 7. Summary of elevation profile differences for Metashape The range of error was similar to Pix4D with an average error range of approximately 6 to 8cm. Error range was large along the road (Figure 4). Overall, all flight elevation profile results agreed with the results at checkpoints with a vertical RMSE around 2cm. Figure 4. Sample of graphical display of elevation profile differences behavior along road 1 (see Figure 3 for road 1).
In Figure 4 the vertical bias between the systems was also apparent. Errors of greatest magnitude occurred at roof peaks. An example of this occurrence is given in Figure 5. Figure 5. Sample graphical display of vertical discrepancy due to resolution of roof peak feature along building 2 (see Figure 3) The graph of differences (top right graph of Figure 5) showed that the errors of greatest magnitude occurred at the peak and errors changed direction beyond that. In the larger scale profile graph (bottom) To the right of the peak a vertical separation occurred along the remainder of the roof side.

DISCUSSION
In this discussion it will be important to recall the accuracy of the surveyed ground control for ASPRS accuracy classification. This research will be classified according to product accuracy per ASPRS positional accuracy standards of 2014 (Table 8) since horizontal accuracy was determined by the orthomosaic and vertical accuracy by the point cloud-derived DSM. Product accuracy will be assessed with the ground control as 1.0cm since the vertical accuracy of the surveyed ground control less than 0.5cm and horizontal 1.0cm. Table 8. Summary of results based on ASPRS 2014 standards for positional accuracy standards.
In the checkpoint results from Pix4D the vertical accuracy may have been affected by flying height but the average RMSEz was 2.0cm ± 0.5cm. The increased altitude of flight A (114m), compared to flights B (84m) and C (105m), displayed lower accuracy. The effect of flying height was seen in accuracy from flights B (84m AGL) to flight C (104m AGL). The flight-line directions did not have an effect in the Pix4D results between flights A and B for each flight. The accuracy also may have been a function of the GSD.
Orthogonal flight paths processed in Metashape displayed a degradation of vertical accuracy in the checkpoints when compared to the results of the parallel flight lines. The effect of orthogonal flight lines were seen in the overall standard deviation of the accuracy (Average RMSEz = 2.6 ± 1.2cm) at checkpoints.
The magnitude of errors that occurred at roof peaks that was observed in all flights was a possible function of the incidence angle. Further, to the right of the peak in the profile graphs a The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-M-2-2020, 2020 ASPRS 2020 Annual Conference Virtual Technical Program, 22-26 June 2020 vertical separation was found indicating that a bias may have been caused by angle of incidence. A greater degree of spatial frequency along the road and asphalt quality may have caused the DSM to be noisier than roof profiles ( Figure YY).

DISCUSSION
This research demonstrated achievable accuracy of UAS imagery-based horizontal and elevation products using SenseFly's fixed-wing, dual-frequency GNSS-capable eBeeX and eBee Plus models (3 flights total). A total of three flight campaigns were performed on separate dates over the San Joaquin Experimental Range located in Fresno County, California in the foothills of the Sierra Nevada Mountain Range over a site with 81 surveyed control point flight targets. From the UAS-captured imagery point clouds, surface models, and orthomosaics were generated using Pix4Dmapper and Agisoft Metashape, two popular, commercial SfM software used for processing UAS imagery. A reference surface was generated from a high-precision Optech Galaxy LiDAR scan point cloud from manned-helicopter. The orthomosaic was assessed for horizontal accuracy and DSM for vertical accuracy at 75 checkpoints. Nearest neighbor interpolation was performed at vertical checkpoints using ArcMap. Various elevation profiles were extracted from UAS and ALS DSM's along roofs and roads and compared for assessment of accuracy to observe elevation accuracy between checkpoints.