EVALUATING THE ACCURACY OF DEM GENERATION ALGORITHMS FROM UAV IMAGERY

In this work we evaluated how the use of different positioning systems affects the accuracy of Digital Elevation Models (DEMs) generated from aerial imagery obtained with Unmanned Aerial Vehicles (UAVs). In this domain, state-of-the-art DEM generation algorithms suffer from typical errors obtained by GPS/INS devices in the position measurements associated with each picture obtained. The deviations from these measurements to real world positions are about meters. The experiments have been carried out using a small quadrotor in the indoor testbed at the Center for Advanced Aerospace Technologies (CATEC). This testbed houses a system that is able to track small markers mounted on the UAV and along the scenario with millimeter precision. This provides very precise position measurements, to which we can add random noise to simulate errors in different GPS receivers. The results showed that final DEM accuracy clearly depends on the positioning information.


INTRODUCTION
Unmanned Aerial Vehicles (UAVs) have seen an exponential progress in the last decades, thanks to their ability to perform complex tasks on limited environments.The scientific community has put great effort on exploiting their potential in applications such as remote sensing, disaster response, surveillance, search and rescue, atmospheric survey, among others.
The automatic generation of Digital Elevation Models (DEMs) has also gained attention in recent years.Several companies and research groups have achieved remarkable advances in this topic and they provide software packages or web services for the automatic generation of such 3D models.The common processing pipeline for the DEM generation depends on several factors such as overlapping, flight altitude, camera resolution, etc. Variations in these parameters affect the final accuracy of the model obtained, and many works (Hudzietz and Saripalli, 2011), (Kung et al., 2011), (Nagai et al., 2009) have analyzed the effects of each of them.
Standard DEM generation algorithms suffer from typical errors obtained by GPS/INS devices, especially in the position measurements associated to each acquired image.In this work we focus on the analysis of how these deviations affect the accuracy of the final 3D reconstruction.Especially in the case of UAVs, payload and cost restrictions limit the use of precise equipment, hence positioning errors are common.In order to study this effect, we performed the experiments in a controlled environment where high precision information of the UAV position is available.This is accomplished by using the indoor testbed at CATEC, which provides a highly precise and time-synchronized motion capture system, based on VICON cameras.The tracking system is able to locate small markers with millimeter precision.A small quadrotor was used in the experiments, whose position and attitude could be precisely determined by placing markers on its airframe.Scaled scenarios were configured in our testbed in order to circumvent the differences with a real outdoor scenario.Several markers were also used on the simulated scenarios as reference positions to compare them with their positions in the gen-erated 3D model.We defined these markers as Ground-Truth-Points (GTPs).
In the experimentation, we simulate different sorts of GPS by adding artificial random deviations to the accurate camera position measurements given by our testbed system.The flight planning phase, including factors like altitude and image overlapping, is also briefly discussed.Other authors (Kung et al., 2011) make use of Ground-Control-Points (GCPs) to improve the geolocation accuracy.The final experiments cover this case and compare the two georreferencing methods.
The paper is structured as follows.Section 2 contains a summary of the factors that affect the process of DEM generation using standard Structure from Motion (SFM) algorithms and GPS information.In section 3, we introduce the different GPS devices that were considered in this study, and justify the noise model used for simulating positioning errors.Section 4 presents the indoor experiments setup and the obtained results.Finally, the conclusions of this work are exposed in section 5.

DEM GENERATION PIPELINE
The processing pipeline for the generation of DEMs consists in the cooperation of several procedures and techniques in which different sensors are involved (Jiang et al., 2009).In order to understand how they relate to each other, a brief discussion about them is presented along with a classification of the common variables that affect the accuracy of the final DEM.DEM generation from UAV imagery is normally based on the framework of a sequential SFM pipelining (Irschara et al., 2012).First, state-of-the-art SFM algorithms are usually employed to obtain a sparse 3D model from triangulating correspondences between images in the scene.For this purpose, camera parameters are necessary to estimate a projection matrix for each image.Then, GPS data is used to remove the scale ambiguity of the initial model.This information allows to recover the 3D position of points matched using feature detectors.After an optimisation step (Manolis and Antonis, 2004), dense descriptors are computed and triangulated in order to increase the spatial resolution of the structure (Jiang et al., 2009).The resulting 3D point cloud is then smoothed and filtered eliminating noisy data.Finally, the dense point cloud is interpolated using a mesh-grid generator.The complete flowchart of DEM generation algorithms is depicted in Figure 1.For each of these techniques and algorithms, there exist several factors that can influence the final DEM.Nevertheless, we found that the final accuracy is more affected by the quality of input data rather than by the specific algorithms parameters.The following subsections deal with the most relevant factors that we have considered in this work.

Image quality
It is important to note that the original quality and clarity of the input images are essential in the output result of DEM generation algorithms, particularly regarding the SFM step.A low resolution or blurring effect on them will infer a low number of matched features between overlapped images.In order to obtain the best possible image quality, camera parameters must be adjusted depending on the specific scenario and lighting conditions.An balanced trade-off between shutter speed and lens aperture will avoid blurred images, hence preventing the detection of useless features.A common strategy would be based on setting an ISO sensibility as low as possible and choose a certain shutter speed according to flight parameters and lighting.
Environmental factors such as climatic conditions or the structural peculiarities of the scene must be also considered for obtaining useful image data sets (Bosak, 2012).Shadows, flashing lights or water areas might create occlusions in the final 3D model due to the absence of texture in the images.These occlusions are interpolated in the mesh-grid generation step, accumulating errors during this process.

Flight planning
Once the camera is properly configured according to the scene, a flight plan must be designed and carried out.Firstly, a study of the terrain elements must be performed in order to assert that the images will contain enough features.The overlap between consecutive images is crucial for achieving a good estimation of the real 3D positions.A factor of 60-70 % is normally employed (Xing et al., 2010).Overlapping directly affects to the number of matched features and consequently, the calibration step and the final result might vary.
A general method for designing flight plans is presented in (Hudzietz and Saripalli, 2011), where altitude is defined using the camera parameters and the desirable 3D model resolution, expressed as Ground Sample Distance (GSD).Figure 2

Geotagging information
Geometric information from matched features allow to obtain projection matrices up to a projective ambiguity.In order to solve such uncertainty, external GPS/INS data is used to locate each image in a reference coordinate system.

xij = PiXj
(1) Figure 3 depicts the general approach for estimating the 3D positions from image features.
Assuming a pinhole camera model, the projection matrix is estimated with information from inertial measurements and intrinsic parameters as follows: where K is the intrinsic camera matrix, R the rotation matrix and t the translation vector.Since inertial data provides an initial estimate for rotation and translation parameters, errors in GPS/INS devices induce deviations in the 3D locations.For a more detailed description on how to obtain these matrices, we refer the reader to (Hartley and Zisserman, 2003).
Furthermore, a good synchronization between the image acquisition time and the corresponding GPS/INS data sample is required.Even with a poor synchronization, projection matrices are optimized during the bundle adjustment step using geometric information from images.However, inaccuracies and global shift in positioning sensors directly affect the final result.

GPS SIMULATION
The main contribution of this work is to study how typical GPS receivers mounted on-board UAV platforms affect the DEM generation process.The testbed at CATEC provides very precise position measurements which are taken as ground truth.Random noise is then added to simulate common errors in the receivers under test.Previous section discussed three main factors that affect DEM generation algorithms.In order to isolate the influence of GPS effects, we will assume that the image acquisition and flight planning issues are correctly configured.Firstly we introduce a brief overview of existing errors in GPS receivers.Afterwards, several receivers are presented with their associated accuracies.
Finally, the noise model adopted for experimentation will be discussed.

Errors in GPS
Positioning errors in GPS are distributed along three defined segments: the space segment, the control segment and the user segment (Solimeno, 2007).Errors in these segments are caused by many different sources such as propagation delays, orbit synchronization or receiver noise.
In order to evaluate the combined effects, they are converted into an equivalent range error experienced by a user, called User Equivalent Range Error (UERE).Commonly, errors from different sources have different statistical properties.Nonetheless, if sufficiently long time periods are considered, all errors can be assumed as independent zero-mean random processes that can be combined to form a single UERE.
Final accuracy in GPS devices is a function of the UERE and the constellation geometry, and is usually split into horizontal and vertical accuracy.

GPS receivers under test
After selective availability deactivation (ESA, 2011), a standard GPS receiver for civil use could be located within 15-20 meters, depending on the number and position of available satellites.

Noise model
According to (Park and Gao, 2008), GPS errors can be modeled as time correlated low-order Gauss-Markov processes, which have an exponentially decaying correlation.For this purpose, three zero-mean Gauss-Markov processes were simulated and sampled over the image acquisition timeline.
As illustrated in Figure 4, positioning information from CATEC testbed system is distorted by adding a noise process with a specific standard deviation, according to each simulated GPS receiver.The distorted signal is then used to generate the final DEM, whose accuracy is evaluated through several GTPs placed along the scenario.

Indoor aerial testbed
CATEC facilities count with an indoor multi-vehicle aerial testbed that can be used to develop and test different algorithms applied to multiple aerial platforms.The tests can be conducted in a 15m × 15m × 5m volume.The testbed has an indoor localization system based on 20 VICON cameras (see Figure 5) that only needs an installation of passive markers on the aerial vehicle and along the scenario.This system is able to provide, in real-time, the position and attitude of the aerial vehicle with millimeter precision.The sample rate of data acquisition is 100 Hz.

Experiment setup
The position measurements typically obtained by GPS/INS devices are replaced by the information reported by the testbed system.Noise signals are added to these measurements according to the specific GPS receiver under test.Then, the input images and positional data are introduced in a DEM generation pipeline similar to Figure 1.The experiments have been performed using the software Pix4UAV Desktop (Kung et al., 2011) that offers an The scale factor between our testbed dimensions and a typical flight scenario for UAVs is about (1 : 100).Artificial objects were placed in the scenario to simulate buildings and other structures at a small scale.For these experiments, a Pelican quadrotor has been used as aerial platform (see Figure 6), from Ascending Technologies Gmbh.
The platform has about 600gr payload and up to 15 minutes of autonomy.The camera used for taking photos was a Canon Pow-erShot G10.

Results
Using the VICON positioning information from a single flight, four different data sets were artificially generated by adding noise signals.Each of the data sets represents simulated information provided by different types of GPS receivers, according to Table 1.In order to measure and report the DEM accuracy, the root mean square error statistic was employed (Minnesota-Planning, 1999).Planimetric (horizontal) and elevation (vertical) accuracies were tested separately, and computed by comparing position deviations using 18 GTPs evenly distributed along the scaled scenario.Both metrics are defined as where {∆Xi, ∆Yi, ∆Zi} is the Euclidean distance between the i-th GTP and its corresponding matched point in the generated DEM.
Figure 7 shows the results obtained for horizontal and vertical accuracies of the generated DEM, as if we had used different GPS receivers for obtaining positioning information.Note that our scenario has a relationship with common UAV flights of (1 : 100).Hence, the accuracies expressed in millimeters correspond to decimeters in a real scenario.Both horizontal and vertical accuracies tend to two exponential distributions since they rise heavily for GPS deviations greater than 100 × GSD.However, smaller deviation values present a lower growth in accuracy.
Figure 7: Accuracy of GTPs for different simulated GPS receivers After this experiment, 5 GCPs were used in the scaled scenario in order to assess their influence in the final DEM accuracy.For this purpose, 5 of the previously used GTPs were selected to serve as control points.The accuracies were calculated using the observed deviation in the remaining 13 GTPs.Final results are presented in Figure 8, where errors are considerably reduced.It can be seen that the accuracies tend to similar distributions for all the GPS receivers, without considering geotagging accuracy from inertial measurements.The results reveal a correlation between the position errors and the final DEM accuracy, which rapidly grows with standard GPS receivers.The indoor testbed at CATEC allowed to acquire very precise information as ground-truth, in order to test the different approaches.Alternatively, using GCPs as a georreferencing sensor is the best option to improve spatial accuracy.However, placing GCPs might not always be possible, depending on the terrain type or accessibility.
Finally, to achieve a more thorough analysis on DEM accuracy, more experiments modifying GSD and flight planning parameters are necessary.
represents a typical UAV flight path in a grid form.

Figure 2 :
Figure 2: Example of UAV flight planning.Using a desired overlapping and a GSD value, a GIA is defined.Grid path is formed linking check points optimally.

Figure 5 :
Figure 5: VICON camera (left) and CATEC indoor testbed (right) optimized implementation of the algorithms described above.Finally, accuracy is calculated as the distance between the real point and the matched point in the 3D model.Complete flowchart is shown in Figure 4.

Figure 6 :
Figure 6: Pelican UAV used in the experiments

Figure 8 :
Figure 8: Accuracy of different GPS receivers using GCPs

Figure 9 :
Figure 9: Scaled scenario and DEM generated during experiments

Table 1 :
Typical accuracy in different GPS receivers Table1summarizes the typical GPS receivers accuracies (horizontal and vertical) using the aforementioned techniques.

Table 2 :
Table 2 compares several parameters used in the scaled scenario with typical parameters in a mapping flight.Typical parameters used in the experiments.Note the (1 : 100) scale factor.